an order-theoretic approach toward entropy
DESCRIPTION
An Order-Theoretic Approach toward Entropy. Kevin H. Knuth Department of Physics University at Albany. In the Beginning…. Lifting Rocks. This caveman finds it easy to order these rocks in terms of how heavy they are to lift. Diagram Representing the Ordering. Heavier. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/1.jpg)
Kevin H. KnuthDepartment of Physics
University at Albany
An Order-Theoretic Approach toward
Entropy
![Page 2: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/2.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
In the Beginning…
![Page 3: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/3.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Lifting Rocks
This caveman finds it easy to order these rocks in terms of how heavy they are to lift.
![Page 4: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/4.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Diagram Representing the OrderingH
eavi
er
He uses a binary weight comparison to order his rocks
This configuration is called a chain
![Page 5: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/5.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
IsomorphismsIs
Les
s R
ipe
than
1
2
3
4
5
Is L
ess
than
or
Equ
al t
o
1
2
4
8
16
Div
ides
![Page 6: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/6.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Antichains
These elements are incomparable under our binary ordering relation
They form a poset called an ANTICHAIN
![Page 7: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/7.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Partitioning
A set of elements together with a binary ordering relation based onthe notion of containing
![Page 8: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/8.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Two Posets with Integers
1
2
3
4
Is L
ess
than
or
Equ
al t
o
1
2 3
4D
ivid
es
5
6
7
8
9
![Page 9: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/9.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Powerset of {a, b, c}
}{b
}{a }{c
},{ ca},{ ba },{ cb
},,{ cba
,},,{},,{},,{},,{},{},{},{, cbacbcabacbaP
Is a subset of
![Page 10: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/10.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Posets
A partially ordered set (poset) is a set of elements together with a binary ordering relation .
}{},,{ acba }{},{ aba
1312
includes
covers
![Page 11: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/11.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Lattices
A lattice is a poset P where every pair of elements x and y has
a least upper bound called the join
a greatest lower bound called the meet
yx
}{b
}{a }{c
},{ ca},{ ba },{ cb
},,{ cba },{}{}{ cbcb
}{},{},{ bcbba Similarly
yx
The greenelements areupper boundsof the blue circledpair. The green circled element is theirleast upper bound ortheir join.
![Page 12: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/12.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Lattice Identities
L1. Idempotent
L2. Commutative
L3. Associative
L4. Absorption
If the meet and join follow the Consistency Relations
C1. (x is the greatest lower bound of x and y)
C2 . (y is the least upper bound of x and y)
Lattice Identities
xxxxxx ,
xyyxxyyx ,
zyxzyxzyxzyx )()(,)()(
xyxxyxx )()(
xyx
yyx
yx
![Page 13: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/13.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Lattices are Algebras
aba
bbaba
StructuralViewpoint
OperationalViewpoint
![Page 14: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/14.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Lattices are Algebras
aba
bbaba
StructuralViewpoint
OperationalViewpoint
aba
bbaba
Assertions, Implies
aba
bbaba
Sets, Is a subset of
aba
bbaba
),gcd(
),lcm(|
Positive Integers, Divides
aba
bbaba
),min(
),max(Integers, Is less than or equal to
![Page 15: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/15.jpg)
Hypothesis Space(Our States of Knowledge)
![Page 16: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/16.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Describing a State of Knowledge
These statements describe the fruit:
a = ‘It is an Apple!’
b = ‘It is a Banana!’
c = ‘It is an Citrus Fruit!’
They can also describe what one knows about the fruit.
![Page 17: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/17.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Hypothesis Space
a b c
The atoms are mutually exclusive and exhaustive logical statements
a = ‘It is an apple!’
b = ‘It is a banana!’
c = ‘It is a citrus fruit!’
![Page 18: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/18.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Boolean Hypothesis Space
a b c
The meet of any two atoms is the absurdity: a b =
We do not allow our state of knowledge to include:
‘The fruit is an apple AND a banana!’
![Page 19: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/19.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Boolean Hypothesis Space
a b c
![Page 20: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/20.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Boolean Hypothesis Space
a b c
a b
The join of any two elements represents a logical OR: a b
![Page 21: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/21.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Boolean Hypothesis Space
a b c
a b a c b c
The join of any two elements represents a logical OR: a b
![Page 22: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/22.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Boolean Hypothesis Space
a b c
a b a c b c
The final join gives us the TOP element, often called the TRUISM
= a b c“It is an Apple or a Banana or an Orange!”
![Page 23: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/23.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Hypothesis Space
This is a HYPOTHESIS SPACE!!!
It consists of all the statements that can be constructed from a set of mutually exclusive exhaustive statements.
The space is ordered by the ordering relation “implies”
We allow concepts like:‘The fruit is an apple OR a banana!’
while we disallow concepts like:‘The fruit is an apple AND a banana!’
a b c
a b a c b c
![Page 24: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/24.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Superpositions of States???
This was your initial state of knowledge.
The fruit was never in this state!
This was the state of the fruit.
![Page 25: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/25.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Two Spaces
Space of statements
describing the fruitSpace of statements describing
a state of knowledge about the fruit
![Page 26: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/26.jpg)
Generalizing Partial Orders to Measures
![Page 27: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/27.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion and the Zeta Function
yxif
yxifyx
0
1),(
The Zeta function encodes inclusion on the lattice.
a b c
a b a c b c
![Page 28: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/28.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion and the Zeta Function
yxif
yxifyx
0
1),(
a b c
a b a c b c
1),( baa
0),( abasince
since
aba baa
The Zeta function encodes inclusion on the lattice.
![Page 29: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/29.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Zeta Function
yxif
yxifyx
0
1),(
a b c
a b a c b c
a b c avb avc bvc T
1 1 1 1 1 1 1 1
a 0 1 0 0 1 1 0 1
b 0 0 1 0 1 0 1 1
c 0 0 0 1 0 1 1 1
avb 0 0 0 0 1 0 0 1
avc 0 0 0 0 0 1 0 1
bvc 0 0 0 0 0 0 1 1
T 0 0 0 0 0 0 0 1
![Page 30: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/30.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion and the Zeta Function
yxif
yxifyx
0
1),(
The Zeta function encodes inclusion on the lattice.
We can define its dual by flipping around the ordering relation
yxif
yxifyx
0
1),(
![Page 31: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/31.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Degrees of Inclusion and Z
yxif
yxifyx
0
1),(
We generalize the dual of the Zeta function
yxif
yxifz
yxif
yxz
0
1
),(
to the function z
![Page 32: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/32.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Z
yxif
yxifz
yxif
yxz
0
1
),(
The function z
Continues to encode inclusion, but has generalized the concept to degrees of inclusion.
In the lattice of logical statements ordered by implies, this function describes degrees of implication.
![Page 33: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/33.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
How do we Assign Values to z?
yxif
yxifz
yxif
yxz
0
1
),(
Are all of the values of the function z arbitrary?
Or are there constraints?
Here there be monsters…
a b c avb avc bvc T
1 0 0 0 0 0 0 0
a 1 1 0 0 ? ? 0 ?
b 1 0 1 0 ? 0 ? ?
c 1 0 0 1 0 ? ? ?
avb 1 1 1 0 1 ? ? ?
avc 1 1 0 1 ? 1 ? ?
bvc 1 0 1 1 ? ? 1 ?
T 1 1 1 1 1 1 1 1
![Page 34: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/34.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Lattice Structure Imposes Constraints
I showed that in “general”:
Associativity leads to a Sum Rule…
Distributivity leads to a Product Rule…
Commutivity leads to Bayes Theorem…
zyxzyx )()(
)()()( zxyxzyx
xyyx
),(),(),(),( wyxzwyzwxzwyxz
),(),(),( wxyzwxzwyxz
),(
),(),(),(
wyz
wxyzwxzwyxz
![Page 35: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/35.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion-Exclusion (The Sum Rule)
),(),(),(),( wyxzwyzwxzwyxz The Sum Rule for Lattices
![Page 36: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/36.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion-Exclusion (The Sum Rule)
)|()|()|()|( iyxpiypixpiyxp
The Sum Rule for Probability
),(),(),(),( wyxzwyzwxzwyxz
![Page 37: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/37.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion-Exclusion (The Sum Rule)
),()()();( YXHYHXHYXI
Definition of Mutual Information
),(),(),(),( wyxzwyzwxzwyxz
)|()|()|()|( iyxpiypixpiyxp
![Page 38: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/38.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion-Exclusion (The Sum Rule)
),min(),max( yxyxyx
Polya’s Min-Max Rule for Integers
),(),(),(),( wyxzwyzwxzwyxz
)|()|()|()|( iyxpiypixpiyxp
),()()();( YXHYHXHYXI
![Page 39: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/39.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inclusion-Exclusion (The Sum Rule)
This is intimately related to the Möbius function for the lattice, which is related to the Zeta function.
),(),(),(),( wyxzwyzwxzwyxz
)|()|()|()|( iyxpiypixpiyxp
),()()();( YXHYHXHYXI
),min(),max( yxyxyx
![Page 40: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/40.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Probability
yxif
xyifp
xyif
yxp
0
1
)|(
Changing notation
The MEANING of p(x|y) is made explicit via the Zeta function.
These are degrees of implication!
NOT plausibility!NOT degrees of belief!NOT frequencies of occurrences!
yxif
yxifz
yxif
yxz
0
1
),(
![Page 41: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/41.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Statement – QuestionDuality
![Page 42: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/42.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Richard T. Cox (1979) defined a question as the set of all possible assertions that answer it. I recast his definition to obtain new insights.
In lattice theory, such a set is called adown-set
In the figure the colored elementsbelong to
I call the set of top elements the irreducible set.
Defining a Question
cba
ba ca cb
ba c
xX
},{
)()(
cbba
cbbaBCAB
![Page 43: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/43.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
When are Questions Equal?
Two questions are equivalent when they are defined by the same set of assertions.
Consider the questions
“Is it raining?”
“Is it not raining?”
They are both answered by the set of assertions generatedby the irreducible set { “It is raining!”, “It is not raining!”} They are therefore equivalent.
![Page 44: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/44.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Animal, Vegetable, Mineral
Space of statementsdescribing the object
Space of Statementsdescribing a state of knowledge about
the object
a v m
![Page 45: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/45.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Central Issue
I = “Is it an Animal, a Vegetable, or a Mineral?”
This question is answered by the following set of statements:
I = { a = “It is an animal!”, v = “It is a vegetable!”, m = “It is a mineral!” }
},,,{ mvaI
![Page 46: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/46.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Some Questions Answer Others
Now consider the binary question
B = “Is it an animal?”
B = {a = “It is an animal!”, ~a = “It is not an animal!”}
As the defining set of I is exhaustive, mva ~
},,,,{ mvmvaB
![Page 47: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/47.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Ordering Questions
B = “Is it an animal?”
I = “Is it an Animal, a Vegetable, or a Mineral?”
BI I answers B
B includes I
},,,{ mvaI
},,,,{ mvmvaB
![Page 48: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/48.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Meets and Joins of Questions
With “is a subset of” as the ordering relation among questions,one can show that:
The meet of two questions is the set intersection of the set of assertions answering the question.
The join of two questions is the set union of the set of assertions answering the question.
YXYX
YXYX
![Page 49: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/49.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Ideals
An ideal is a nonvoid subset J of a lattice A with the properties (Birkhoff 1967)I1. , where thenI2. , then
I1 is the condition for the set J to be a down-set, or equivalently a question.
I2 assures that there is a unique maximum
Therefore, as ideals are questions, I call them Ideal Questions.
JxJx Jz
Ay xy Jzx
Jy
![Page 50: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/50.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Ideals and Ideal Questions
![Page 51: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/51.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
There are More Questions!
Questions can be formed from set unions of ideal questions.
which can be written
These questions along with the ideal questions form alattice ordered by set inclusion.
)()( mvaMAV
},,,,{},{},,,{ mvavamvava
![Page 52: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/52.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
The Lattice of Questions
The lattice of questions is the Free Distributive Lattice, Q(N) = FD(N)
No complements!
![Page 53: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/53.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Real Questions
Real Questions are questions thatcan always be answered by a true assertion.
Real questions must include the exhaustive set of mutually exclusive assertions.
The Real Question lattice R(N)looks almost Boolean for N=3,but this is not true in general.
![Page 54: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/54.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Real Questions
In Q(3) these are the questions R where
The bottom question is called the Central Issue and as such it results in an unambiguous answer.
RMVA
R(3)
![Page 55: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/55.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Partition Questions
A partition question P neatly partitions the possible answers.These are the join-irreducible elements of the lattice R(N).
partition latticeR(3)
![Page 56: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/56.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Valuations on the Question Lattice
![Page 57: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/57.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Inquiry Calculus
Generalizing the zeta function for the question lattice gives a valuation that describes the degree to which the question Q answers the issue I
I call this quantity the relevance
The relevance follows the Sum Rule
otherwised
QIif
QIif
QId
10
0
1
)|(
),()|( QIzQId
![Page 58: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/58.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Consistency Principle
Probability assignments in A must be consistent with relevance assignments in Q
![Page 59: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/59.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Ideal Questions again
Only one assumption:
))|(,),|(),|(()|( 21 TTT nn xpxpxpHPId
The degree to which a partition question resolves the central issue is a function of the probabilities of the statements comprising the partition question’s irreducible set.
The goal is to determine the form of the functions Hn().
Again, the lattice properties constrain the solution!
![Page 60: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/60.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Consequences of Our Assumption
YXiffYIdXIdYXId )|()|()|(
1. Additivity
(sum rule)
otherwiseYIdXIdYXId )|()|()|(
2. Subadditivity
(sum rule)
)|()|( )()2()1(21 nn XXXIdXXXId
3. Commutativity
(commute)
4. Expansibility
(identifying with bottom)
)|()|( 2121 nn XXXIdFXXXId
![Page 61: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/61.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Entropy
),,,(),,,(),,,( 212121 nonnn pppHbpppHapppH
n
iiin pppppH
1221 log),,,(
NpppH no 221 log),,,(
Shannon
Hartley
Aczél (Aczél et al., 1974): The function Hm satisfying additivity, subadditivity, commutativity and expansibility is a linear combination of Shannon (1949) and Hartley (1928) entropies
![Page 62: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/62.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Normalization Conditions
)(
log1 2
IH
Nba
Relevance is maximized when
1)|( IId
This implies that
Relevance is minimized when which gives0)|( TId
I
T
)()(log
)()1()(
12
1
IHNPHN
IHNPH
bP
ii
P
ii
![Page 63: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/63.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Results
a
a
a a
aa
v v v
v v v
m m m
m m m
AVM VAM MAV
AVAM AVVM AMVM
![Page 64: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/64.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Relevanceand
Information Theory
![Page 65: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/65.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Relevance and Entropy
)|( QId
mmvvaamva pppppppppHIH 222 logloglog),,()(
),( mva ppH
![Page 66: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/66.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Higher-Order Informations
))()(|()|()|()|( VMAAMVIdVMAIdAMVIdVMAMId
);(~)|( VMAAMVIVMAMId
This relevance is related to the mutual information.
In this way one can obtain higher-order informations.
![Page 67: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/67.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Questions and Geometry
![Page 68: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/68.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Join-Irreducible Elements
a
a
v
a
v
m
The join-irreducible elements are the simplexes.With the bottom, they form a Boolean Lattice.
![Page 69: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/69.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Questions ~ Simplicial Complexes
Is it or Is it Not an Animal?
a m
v
mv
![Page 70: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/70.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Lattice of Simplicial Complices
The lattice of questions is isomorphic to the lattice ofsimplicial complices.
These are alsoisomorphic to the lattice of hypergraphs.
![Page 71: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/71.jpg)
26 Oct 2007 Kevin H KnuthFacets of Entropy
Summary Inclusion can be generalized to degrees of inclusion
The lattice structure places constraints on the values of these degrees of inclusion.
On a Distributive Lattice, these constraints take the form of a Sum Rule, a Product Rule and a Bayes Theorem.
The logical statements ordered by implication gives rise to degrees of implication that follow the rules of probability theory
The lattice of questions is dual to the lattice of assertions in the sense of Birkhoff’s Representation Theorem.
The relevance of questions depends on the Shannon and Hartley entropies.
![Page 72: An Order-Theoretic Approach toward Entropy](https://reader036.vdocument.in/reader036/viewer/2022062720/56813358550346895d9a6c7c/html5/thumbnails/72.jpg)
Special Thanks to:
John SkillingAriel CatichaJanos AczélPhilip GoyalSteve GullJeffrey JewellCarlos Rodriguez
for many insightful discussions and comments
Supported in part by:NASA AIST-QRS-07-0001NASA 05-AISR05-0143