an order-theoretic approach toward entropy

Kevin H. KnuthDepartment of Physics

University at Albany

An Order-Theoretic Approach toward

Entropy

26 Oct 2007 Kevin H KnuthFacets of Entropy

In the Beginning…

26 Oct 2007 Kevin H KnuthFacets of Entropy

Lifting Rocks

This caveman finds it easy to order these rocks in terms of how heavy they are to lift.

26 Oct 2007 Kevin H KnuthFacets of Entropy

Diagram Representing the OrderingH

eavi

er

He uses a binary weight comparison to order his rocks

This configuration is called a chain

26 Oct 2007 Kevin H KnuthFacets of Entropy

IsomorphismsIs

Les

s R

ipe

than

1

2

3

4

5

Is L

ess

than

or

Equ

al t

o

1

2

4

8

16

Div

ides

26 Oct 2007 Kevin H KnuthFacets of Entropy

Antichains

These elements are incomparable under our binary ordering relation

They form a poset called an ANTICHAIN

26 Oct 2007 Kevin H KnuthFacets of Entropy

Partitioning

A set of elements together with a binary ordering relation based onthe notion of containing

26 Oct 2007 Kevin H KnuthFacets of Entropy

Two Posets with Integers

1

2

3

4

Is L

ess

than

or

Equ

al t

o

1

2 3

4D

ivid

es

5

6

7

8

9

26 Oct 2007 Kevin H KnuthFacets of Entropy

The Powerset of {a, b, c}

}{b

}{a }{c

},{ ca},{ ba },{ cb

},,{ cba

,},,{},,{},,{},,{},{},{},{, cbacbcabacbaP

Is a subset of

26 Oct 2007 Kevin H KnuthFacets of Entropy

Posets

A partially ordered set (poset) is a set of elements together with a binary ordering relation .

}{},,{ acba }{},{ aba

1312

includes

covers

26 Oct 2007 Kevin H KnuthFacets of Entropy

Lattices

A lattice is a poset P where every pair of elements x and y has

a least upper bound called the join

a greatest lower bound called the meet

yx

}{b

}{a }{c

},{ ca},{ ba },{ cb

},,{ cba },{}{}{ cbcb

}{},{},{ bcbba Similarly

yx

The greenelements areupper boundsof the blue circledpair. The green circled element is theirleast upper bound ortheir join.

26 Oct 2007 Kevin H KnuthFacets of Entropy

The Lattice Identities

L1. Idempotent

L2. Commutative

L3. Associative

L4. Absorption

If the meet and join follow the Consistency Relations

C1. (x is the greatest lower bound of x and y)

C2 . (y is the least upper bound of x and y)

Lattice Identities

xxxxxx ,

xyyxxyyx ,

zyxzyxzyxzyx )()(,)()(

xyxxyxx )()(

xyx

yyx

yx

26 Oct 2007 Kevin H KnuthFacets of Entropy

Lattices are Algebras

aba

bbaba

StructuralViewpoint

OperationalViewpoint

26 Oct 2007 Kevin H KnuthFacets of Entropy

Lattices are Algebras

aba

bbaba

StructuralViewpoint

OperationalViewpoint

aba

bbaba

Assertions, Implies

aba

bbaba

Sets, Is a subset of

aba

bbaba

),gcd(

),lcm(|

Positive Integers, Divides

aba

bbaba

),min(

),max(Integers, Is less than or equal to

Hypothesis Space(Our States of Knowledge)

26 Oct 2007 Kevin H KnuthFacets of Entropy

Describing a State of Knowledge

These statements describe the fruit:

a = ‘It is an Apple!’

b = ‘It is a Banana!’

c = ‘It is an Citrus Fruit!’

They can also describe what one knows about the fruit.

26 Oct 2007 Kevin H KnuthFacets of Entropy

The Hypothesis Space

a b c

The atoms are mutually exclusive and exhaustive logical statements

a = ‘It is an apple!’

b = ‘It is a banana!’

c = ‘It is a citrus fruit!’

26 Oct 2007 Kevin H KnuthFacets of Entropy

The Boolean Hypothesis Space

a b c

The meet of any two atoms is the absurdity: a b =

We do not allow our state of knowledge to include:

‘The fruit is an apple AND a banana!’

26 Oct 2007 Kevin H KnuthFacets of Entropy

The Boolean Hypothesis Space

a b c

26 Oct 2007 Kevin H KnuthFacets of Entropy

The Boolean Hypothesis Space

a b c

a b

The join of any two elements represents a logical OR: a b

26 Oct 2007 Kevin H KnuthFacets of Entropy

The Boolean Hypothesis Space

a b c

a b a c b c

The join of any two elements represents a logical OR: a b

26 Oct 2007 Kevin H KnuthFacets of Entropy

The Boolean Hypothesis Space

a b c

a b a c b c

The final join gives us the TOP element, often called the TRUISM

= a b c“It is an Apple or a Banana or an Orange!”

26 Oct 2007 Kevin H KnuthFacets of Entropy

The Hypothesis Space

This is a HYPOTHESIS SPACE!!!

It consists of all the statements that can be constructed from a set of mutually exclusive exhaustive statements.

The space is ordered by the ordering relation “implies”

We allow concepts like:‘The fruit is an apple OR a banana!’

while we disallow concepts like:‘The fruit is an apple AND a banana!’

a b c

a b a c b c

26 Oct 2007 Kevin H KnuthFacets of Entropy

Superpositions of States???

This was your initial state of knowledge.

The fruit was never in this state!

This was the state of the fruit.

26 Oct 2007 Kevin H KnuthFacets of Entropy

Two Spaces

Space of statements

describing the fruitSpace of statements describing

a state of knowledge about the fruit

Generalizing Partial Orders to Measures

26 Oct 2007 Kevin H KnuthFacets of Entropy

Inclusion and the Zeta Function

yxif

yxifyx

0

1),(

The Zeta function encodes inclusion on the lattice.

a b c

a b a c b c

26 Oct 2007 Kevin H KnuthFacets of Entropy

Inclusion and the Zeta Function

yxif

yxifyx

0

1),(

a b c

a b a c b c

1),( baa

0),( abasince

since

aba baa

The Zeta function encodes inclusion on the lattice.

26 Oct 2007 Kevin H KnuthFacets of Entropy

The Zeta Function

yxif

yxifyx

0

1),(

a b c

a b a c b c

a b c avb avc bvc T

1 1 1 1 1 1 1 1

a 0 1 0 0 1 1 0 1

b 0 0 1 0 1 0 1 1

c 0 0 0 1 0 1 1 1

avb 0 0 0 0 1 0 0 1

avc 0 0 0 0 0 1 0 1

bvc 0 0 0 0 0 0 1 1

T 0 0 0 0 0 0 0 1

26 Oct 2007 Kevin H KnuthFacets of Entropy

Inclusion and the Zeta Function

yxif

yxifyx

0

1),(

The Zeta function encodes inclusion on the lattice.

We can define its dual by flipping around the ordering relation

yxif

yxifyx

0

1),(

26 Oct 2007 Kevin H KnuthFacets of Entropy

Degrees of Inclusion and Z

yxif

yxifyx

0

1),(

We generalize the dual of the Zeta function

yxif

yxifz

yxif

yxz

0

1

),(

to the function z

26 Oct 2007 Kevin H KnuthFacets of Entropy

Z

yxif

yxifz

yxif

yxz

0

1

),(

The function z

Continues to encode inclusion, but has generalized the concept to degrees of inclusion.

In the lattice of logical statements ordered by implies, this function describes degrees of implication.

26 Oct 2007 Kevin H KnuthFacets of Entropy

How do we Assign Values to z?

yxif

yxifz

yxif

yxz

0

1

),(

Are all of the values of the function z arbitrary?

Or are there constraints?

Here there be monsters…

a b c avb avc bvc T

1 0 0 0 0 0 0 0

a 1 1 0 0 ? ? 0 ?

b 1 0 1 0 ? 0 ? ?

c 1 0 0 1 0 ? ? ?

avb 1 1 1 0 1 ? ? ?

avc 1 1 0 1 ? 1 ? ?

bvc 1 0 1 1 ? ? 1 ?

T 1 1 1 1 1 1 1 1

26 Oct 2007 Kevin H KnuthFacets of Entropy

Lattice Structure Imposes Constraints

I showed that in “general”:

Associativity leads to a Sum Rule…

Distributivity leads to a Product Rule…

Commutivity leads to Bayes Theorem…

zyxzyx )()(

)()()( zxyxzyx

xyyx

),(),(),(),( wyxzwyzwxzwyxz

),(),(),( wxyzwxzwyxz

),(

),(),(),(

wyz

wxyzwxzwyxz

26 Oct 2007 Kevin H KnuthFacets of Entropy

Inclusion-Exclusion (The Sum Rule)

),(),(),(),( wyxzwyzwxzwyxz The Sum Rule for Lattices

26 Oct 2007 Kevin H KnuthFacets of Entropy

Inclusion-Exclusion (The Sum Rule)

)|()|()|()|( iyxpiypixpiyxp

The Sum Rule for Probability

),(),(),(),( wyxzwyzwxzwyxz

26 Oct 2007 Kevin H KnuthFacets of Entropy

Inclusion-Exclusion (The Sum Rule)

),()()();( YXHYHXHYXI

Definition of Mutual Information

),(),(),(),( wyxzwyzwxzwyxz

)|()|()|()|( iyxpiypixpiyxp

26 Oct 2007 Kevin H KnuthFacets of Entropy

Inclusion-Exclusion (The Sum Rule)

),min(),max( yxyxyx

Polya’s Min-Max Rule for Integers

),(),(),(),( wyxzwyzwxzwyxz

)|()|()|()|( iyxpiypixpiyxp

),()()();( YXHYHXHYXI

26 Oct 2007 Kevin H KnuthFacets of Entropy

Inclusion-Exclusion (The Sum Rule)

This is intimately related to the Möbius function for the lattice, which is related to the Zeta function.

),(),(),(),( wyxzwyzwxzwyxz

)|()|()|()|( iyxpiypixpiyxp

),()()();( YXHYHXHYXI

),min(),max( yxyxyx

26 Oct 2007 Kevin H KnuthFacets of Entropy

Probability

yxif

xyifp

xyif

yxp

0

1

)|(

Changing notation

The MEANING of p(x|y) is made explicit via the Zeta function.

These are degrees of implication!

NOT plausibility!NOT degrees of belief!NOT frequencies of occurrences!

yxif

yxifz

yxif

yxz

0

1

),(

26 Oct 2007 Kevin H KnuthFacets of Entropy

Statement – QuestionDuality

26 Oct 2007 Kevin H KnuthFacets of Entropy

Richard T. Cox (1979) defined a question as the set of all possible assertions that answer it. I recast his definition to obtain new insights.

In lattice theory, such a set is called adown-set

In the figure the colored elementsbelong to

I call the set of top elements the irreducible set.

Defining a Question

cba

ba ca cb

ba c

xX

},{

)()(

cbba

cbbaBCAB

26 Oct 2007 Kevin H KnuthFacets of Entropy

When are Questions Equal?

Two questions are equivalent when they are defined by the same set of assertions.

Consider the questions

“Is it raining?”

“Is it not raining?”

They are both answered by the set of assertions generatedby the irreducible set { “It is raining!”, “It is not raining!”} They are therefore equivalent.

26 Oct 2007 Kevin H KnuthFacets of Entropy

Animal, Vegetable, Mineral

Space of statementsdescribing the object

Space of Statementsdescribing a state of knowledge about

the object

a v m

26 Oct 2007 Kevin H KnuthFacets of Entropy

The Central Issue

I = “Is it an Animal, a Vegetable, or a Mineral?”

This question is answered by the following set of statements:

I = { a = “It is an animal!”, v = “It is a vegetable!”, m = “It is a mineral!” }

},,,{ mvaI

26 Oct 2007 Kevin H KnuthFacets of Entropy

Some Questions Answer Others

Now consider the binary question

B = “Is it an animal?”

B = {a = “It is an animal!”, ~a = “It is not an animal!”}

As the defining set of I is exhaustive, mva ~

},,,,{ mvmvaB

26 Oct 2007 Kevin H KnuthFacets of Entropy

Ordering Questions

B = “Is it an animal?”

I = “Is it an Animal, a Vegetable, or a Mineral?”

BI I answers B

B includes I

},,,{ mvaI

},,,,{ mvmvaB

26 Oct 2007 Kevin H KnuthFacets of Entropy

Meets and Joins of Questions

With “is a subset of” as the ordering relation among questions,one can show that:

The meet of two questions is the set intersection of the set of assertions answering the question.

The join of two questions is the set union of the set of assertions answering the question.

YXYX

YXYX

26 Oct 2007 Kevin H KnuthFacets of Entropy

Ideals

An ideal is a nonvoid subset J of a lattice A with the properties (Birkhoff 1967)I1. , where thenI2. , then

I1 is the condition for the set J to be a down-set, or equivalently a question.

I2 assures that there is a unique maximum

Therefore, as ideals are questions, I call them Ideal Questions.

JxJx Jz

Ay xy Jzx

Jy

26 Oct 2007 Kevin H KnuthFacets of Entropy

Ideals and Ideal Questions

26 Oct 2007 Kevin H KnuthFacets of Entropy

There are More Questions!

Questions can be formed from set unions of ideal questions.

which can be written

These questions along with the ideal questions form alattice ordered by set inclusion.

)()( mvaMAV

},,,,{},{},,,{ mvavamvava

26 Oct 2007 Kevin H KnuthFacets of Entropy

The Lattice of Questions

The lattice of questions is the Free Distributive Lattice, Q(N) = FD(N)

No complements!

26 Oct 2007 Kevin H KnuthFacets of Entropy

Real Questions

Real Questions are questions thatcan always be answered by a true assertion.

Real questions must include the exhaustive set of mutually exclusive assertions.

The Real Question lattice R(N)looks almost Boolean for N=3,but this is not true in general.

26 Oct 2007 Kevin H KnuthFacets of Entropy

Real Questions

In Q(3) these are the questions R where

The bottom question is called the Central Issue and as such it results in an unambiguous answer.

RMVA

R(3)

26 Oct 2007 Kevin H KnuthFacets of Entropy

Partition Questions

A partition question P neatly partitions the possible answers.These are the join-irreducible elements of the lattice R(N).

partition latticeR(3)

26 Oct 2007 Kevin H KnuthFacets of Entropy

Valuations on the Question Lattice

26 Oct 2007 Kevin H KnuthFacets of Entropy

Inquiry Calculus

Generalizing the zeta function for the question lattice gives a valuation that describes the degree to which the question Q answers the issue I

I call this quantity the relevance

The relevance follows the Sum Rule

otherwised

QIif

QIif

QId

10

0

1

)|(

),()|( QIzQId

26 Oct 2007 Kevin H KnuthFacets of Entropy

Consistency Principle

Probability assignments in A must be consistent with relevance assignments in Q

26 Oct 2007 Kevin H KnuthFacets of Entropy

Ideal Questions again

Only one assumption:

))|(,),|(),|(()|( 21 TTT nn xpxpxpHPId

The degree to which a partition question resolves the central issue is a function of the probabilities of the statements comprising the partition question’s irreducible set.

The goal is to determine the form of the functions Hn().

Again, the lattice properties constrain the solution!

26 Oct 2007 Kevin H KnuthFacets of Entropy

Consequences of Our Assumption

YXiffYIdXIdYXId )|()|()|(

1. Additivity

(sum rule)

otherwiseYIdXIdYXId )|()|()|(

2. Subadditivity

(sum rule)

)|()|( )()2()1(21 nn XXXIdXXXId

3. Commutativity

(commute)

4. Expansibility

(identifying with bottom)

)|()|( 2121 nn XXXIdFXXXId

26 Oct 2007 Kevin H KnuthFacets of Entropy

Entropy

),,,(),,,(),,,( 212121 nonnn pppHbpppHapppH

n

iiin pppppH

1221 log),,,(

NpppH no 221 log),,,(

Shannon

Hartley

Aczél (Aczél et al., 1974): The function Hm satisfying additivity, subadditivity, commutativity and expansibility is a linear combination of Shannon (1949) and Hartley (1928) entropies

26 Oct 2007 Kevin H KnuthFacets of Entropy

Normalization Conditions

)(

log1 2

IH

Nba

Relevance is maximized when

1)|( IId

This implies that

Relevance is minimized when which gives0)|( TId

I

T

)()(log

)()1()(

12

1

IHNPHN

IHNPH

bP

ii

P

ii

26 Oct 2007 Kevin H KnuthFacets of Entropy

Results

a

a

a a

aa

v v v

v v v

m m m

m m m

AVM VAM MAV

AVAM AVVM AMVM

26 Oct 2007 Kevin H KnuthFacets of Entropy

Relevanceand

Information Theory

26 Oct 2007 Kevin H KnuthFacets of Entropy

Relevance and Entropy

)|( QId

mmvvaamva pppppppppHIH 222 logloglog),,()(

),( mva ppH

26 Oct 2007 Kevin H KnuthFacets of Entropy

Higher-Order Informations

))()(|()|()|()|( VMAAMVIdVMAIdAMVIdVMAMId

);(~)|( VMAAMVIVMAMId

This relevance is related to the mutual information.

In this way one can obtain higher-order informations.

26 Oct 2007 Kevin H KnuthFacets of Entropy

Questions and Geometry

26 Oct 2007 Kevin H KnuthFacets of Entropy

Join-Irreducible Elements

a

a

v

a

v

m

The join-irreducible elements are the simplexes.With the bottom, they form a Boolean Lattice.

26 Oct 2007 Kevin H KnuthFacets of Entropy

Questions ~ Simplicial Complexes

Is it or Is it Not an Animal?

a m

v

mv

26 Oct 2007 Kevin H KnuthFacets of Entropy

Lattice of Simplicial Complices

The lattice of questions is isomorphic to the lattice ofsimplicial complices.

These are alsoisomorphic to the lattice of hypergraphs.

26 Oct 2007 Kevin H KnuthFacets of Entropy

Summary Inclusion can be generalized to degrees of inclusion

The lattice structure places constraints on the values of these degrees of inclusion.

On a Distributive Lattice, these constraints take the form of a Sum Rule, a Product Rule and a Bayes Theorem.

The logical statements ordered by implication gives rise to degrees of implication that follow the rules of probability theory

The lattice of questions is dual to the lattice of assertions in the sense of Birkhoff’s Representation Theorem.

The relevance of questions depends on the Shannon and Hartley entropies.

Special Thanks to:

John SkillingAriel CatichaJanos AczélPhilip GoyalSteve GullJeffrey JewellCarlos Rodriguez

for many insightful discussions and comments

Supported in part by:NASA AIST-QRS-07-0001NASA 05-AISR05-0143