a modern history of probability theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · a modern...
TRANSCRIPT
![Page 1: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/1.jpg)
4/29/2016 Knuth - Bayes Forum 1
A Modern History of Probability Theory
Kevin H. Knuth Depts. of Physics and Informatics University at Albany (SUNY) Albany NY USA
![Page 2: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/2.jpg)
A Modern History of Probability Theory
Kevin H. Knuth Depts. of Physics and Informatics University at Albany (SUNY) Albany NY USA
4/29/2016
A Modern History of Probability Theory
Knuth - Bayes Forum 2
![Page 3: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/3.jpg)
A Long History
4/29/2016
The History of Probability Theory, Anthony J.M. Garrett
MaxEnt 1997, pp. 223-238.
Hájek, Alan, "Interpretations of Probability", The Stanford Encyclopedia of Philosophy (Winter 2012 Edition), Edward N. Zalta (ed.), URL = <http://plato.stanford.edu/archives/win2012/entries/probability-interpret/>.
Knuth - Bayes Forum 3
![Page 4: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/4.jpg)
4/29/2016
Pierre Simon de Laplace Théorie Analytique des Probabilités
… the theory of probabilities is basically just common sense reduced to calculation …
… la théorie des probabilités n'est, au fond, que le bon sens réduit au calcul …
Knuth - Bayes Forum 4
![Page 5: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/5.jpg)
4/29/2016
Taken from Harold Jeffreys “Theory of Probability”
Knuth - Bayes Forum 5
![Page 6: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/6.jpg)
4/29/2016
The terms certain and probable describe the various degrees of rational belief about a proposition which different amounts of knowledge authorise us to entertain. All propositions are true or false, but the knowledge we have of them depends on our circumstances; and while it is often convenient to speak of propositions as certain or probable, this expresses strictly a relationship in which they stand to a corpus of knowledge, actual or hypothetical, and not a characteristic of the propositions in themselves. A proposition is capable at the same time of varying degrees of this relationship, depending upon the knowledge to which it is related, so that it is without significance to call a proposition probable unless we specify the knowledge to which we are relating it.
To this extent, therefore, probability may be called subjective. But in the sense important to logic, probability is not subjective. It is not, that is to say, subject to human caprice. A proposition is not probable because we think it so. When once the facts are given which determine our knowledge, what is probable or improbable in these circumstances has been fixed objectively, and is independent of our opinion. The Theory of Probability is logical, therefore, because it is concerned with the degree of belief which it is rational to entertain in given conditions, and not merely with the actual beliefs of particular individuals, which may or may not be rational.
John Maynard Keynes
Knuth - Bayes Forum 6
![Page 7: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/7.jpg)
4/29/2016
“In deriving the laws of probability from more fundamental ideas, one has to engage with what ‘probability’ means.
- Anthony J.M. Garrett, “Whence the Laws of Probability”, MaxEnt 1997
This is a notoriously contentious issue; fortunately, if you disagree with the definition that is proposed, there will be a get-out that allows other definitions to be preserved.”
Meaning of Probability
Knuth - Bayes Forum 7
![Page 8: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/8.jpg)
4/29/2016
The function 𝑝(𝑥|𝑦) is often read as ‘the probability of 𝑥 given 𝑦’
Meaning of Probability
This is most commonly interpreted as the probability that the proposition 𝑥 is true given that the proposition 𝑦 is true.
This concept can be summarized as a degree of truth
Concepts of Probability:
- degree of truth - degree of rational belief - degree of implication
Knuth - Bayes Forum 8
![Page 9: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/9.jpg)
4/29/2016
Laplace, Maxwell, Keynes, Jeffreys and Cox all presented a concept of probability based on a degree of rational belief.
As Keynes points out, this is not to be thought of as subject to human capriciousness, but rather what an ideally rational agent ought to believe.
Meaning of Probability
Concepts of Probability:
- degree of truth - degree of rational belief - degree of implication
Knuth - Bayes Forum 9
![Page 10: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/10.jpg)
4/29/2016
Anton Garrett discusses Keynes as conceiving of probability as a degree of implication. I don’t get that impression reading Keynes. Instead, it seems to me that this is the concept that Garrett had (at the time) adopted.
Garrett uses the word implicability.
Meaning of Probability
Concepts of Probability:
- degree of truth - degree of rational belief - degree of implication
Knuth - Bayes Forum 10
![Page 11: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/11.jpg)
Concepts of Probability:
- degree of truth - degree of rational belief - degree of implication
4/29/2016
Meaning of Probability
John Skilling argued against relying on the concept of truth thusly:
“You wouldn’t know the truth if I told it to you!”
Knuth - Bayes Forum 11
![Page 12: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/12.jpg)
Concepts of Probability:
- degree of truth - degree of rational belief - degree of implication
4/29/2016
Meaning of Probability
Jeffrey Scargle once pointed out that if probability quantifies truth or degrees of belief, one cannot assign a non-zero probability to a model that is known to be an approximation.
One cannot claim to be making inferences with any honesty or consistency while entertaining a concept of probability based on a degree of truth or a degree of rational belief.
Knuth - Bayes Forum 12
![Page 13: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/13.jpg)
Concepts of Probability:
- degree of truth - degree of rational belief - degree of implication
4/29/2016
Meaning of Probability
Jeffrey Scargle once pointed out that if probability quantifies truth or degrees of belief, one cannot assign a non-zero probability to a model that is known to be an approximation.
One cannot claim to be making inferences with any honesty or consistency while entertaining a concept of probability based on a degree of truth or a degree of rational belief.
Knuth - Bayes Forum 13
Can I give you a “Get-Out” like Anton did?
![Page 14: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/14.jpg)
4/29/2016
Bruno de Finetti - 1931 Andrey Kolmogorov - 1933 Richard Threlkeld Cox - 1946
Knuth - Bayes Forum 14
Three Foundations of Probability Theory
Foundation Based on Consistent Betting
Unfortunately, the most commonly presented foundation of probability theory in modern quantum foundations
Foundation Based on Measures on Sets
of Events
Perhaps the most widely accepted foundation by modern Bayesians
Foundation Based on Generalizing Boolean
Implication to Degrees
The foundation which has inspired the most
investigation and development
![Page 15: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/15.jpg)
4/29/2016 Knuth - Bayes Forum 15
Three Foundations of Probability Theory
Bruno de Finetti - 1931
Foundation Based on Consistent Betting
Unfortunately, the most commonly presented foundation of probability theory in modern quantum foundations
Subjective Bayesianism and the Dutch Book Argument De Finetti conceived of probabilities as a degree of belief which could be quantified by considering how much one would be willing to bet on a proposition. Consistency in betting is central to the foundation. A Dutch Book is a series of bets which guarantees that one person will profit over another regardless of the outcome. One can show that if one’s subjective degree of belief does not obey the probability calculus, then one is susceptible to a Dutch Book. Moreover, one can avoid a Dutch Book by ensuring that one’s subjective degree of belief is in agreement with the probability calculus. Important due to its reliance on consistency.
![Page 16: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/16.jpg)
4/29/2016 Knuth - Bayes Forum 16
Three Foundations of Probability Theory
Andrey Kolmogorov - 1933
Foundation Based on Measures on Sets
of Events
Perhaps the most widely accepted foundation by modern Bayesians
Kolmogorov’s Probability Calculus Axiom I (Non-Negativity) Probability is quantified by a non-negative real number. Axiom II (Normalization) Probability has a maximum value Pr 𝑒 ≤ 1 such that the probability that an event in the set E will occur is unity. Axiom III (Finite Additivity) Probability is σ-additive, such that the probability of any countable union of disjoint events 𝑒1, 𝑒2, ⋯ 𝜖 𝐸 is given by Pr 𝑒1 ∪ 𝑒2 ∪ ⋯ = Pr (𝑒𝑖)
∞𝑖 .
It is perhaps the both the conventional nature of his approach and the simplicity of the axioms that has led to such wide acceptance of his foundation.
![Page 17: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/17.jpg)
4/29/2016 Knuth - Bayes Forum 17
Three Foundations of Probability Theory
Richard Threlkeld Cox - 1946
Foundation Based on Generalizing Boolean
Implication to Degrees
The foundation which has inspired the most
investigation and development
Generalizing Boolean Logic to Degrees of Belief Axiom 0 Probability quantifies the reasonable credibility of a proposition when another proposition is known to be true Axiom I The likelihood 𝑐 ∙ 𝑏 | 𝑎 is a function of 𝑏|𝑎 and 𝑐| 𝑏 ∙ 𝑎 𝑐 ∙ 𝑏 | 𝑎 = F(𝑏|𝑎, 𝑐 | 𝑏 ∙ 𝑎) Axiom II There is a relation between the likelihood of a proposition and its contradictory ~𝑏|𝑎 = 𝑆 𝑏 𝑎)
![Page 18: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/18.jpg)
4/29/2016 Knuth - Bayes Forum 18
In Physics we have a saying, “The greatness of a scientist is measured by how long he/she retards progress in the field.” Both de Finetti and Kolmogorov considered a well-defined domain, left few loose ends, and no noticeable conceptual glitches to give their disciples sufficient reason or concern to keep investigating. Cox, on the other hand, proposed a radical approach that raised concerns about how belief could be quantified as well as whether one could improve upon his axioms despite justification by common-sense.
His work was just the right balance between - Pushing it far enough to be interesting - Getting it right enough to be compelling - Leaving it rough enough for there to be remaining work to be done
![Page 19: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/19.jpg)
4/29/2016 Knuth - Bayes Forum 19
And Work Was Done! (Knuth-centric partial illustration) Richard T. Cox
Ed Jaynes Gary Erickson C. Ray Smith Myron Tribus Ariel Caticha
Kevin Van Horn Investigate Alternate Axioms
Anthony Garrett Efficiently Employs NAND
Steve Gull & Yoel Tikochinsky Work to derive Feynman
Rules for Quantum Mechanics
Ariel Caticha Feynman Rules for QM Setups Associativity and Distributivity
R. T. Cox Inquiry
Robert Fry Inquiry
Kevin Knuth Logic of Questions
Associativity and Distributivity
Kevin Knuth Order-theory and Probability
Associativity and Distributivity
Kevin Knuth & John Skilling Order-theory and Probability
Associativity, Associativity, Associativity
Philip Goyal, Kevin Knuth, John Skilling Feynman Rules for QM Kevin Knuth
Inquiry Calculus
Philip Goyal Identical Particles in QM
Jos Uffink Imre Czisar
![Page 20: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/20.jpg)
4/29/2016
John Maynard Keynes - 1921
Bruno de Finetti - 1931 Andrey Kolmogorov - 1933
Sir Harold Jeffreys - 1939
Richard Threlkeld Cox - 1946
Edwin Thompson Jaynes - 1957
Claude Shannon - 1948
Probability Theory Timeline
1920
1930
1940
1950
1960
Knuth - Bayes Forum 20
![Page 21: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/21.jpg)
4/29/2016
John Maynard Keynes - 1921
Bruno de Finetti - 1931 Andrey Kolmogorov - 1933
Sir Harold Jeffreys - 1939
Richard Threlkeld Cox - 1946
Edwin Thompson Jaynes - 1957
Claude Shannon - 1948
Probability Theory Timeline
1920
1930
1940
1950
1960
Knuth - Bayes Forum 21
![Page 22: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/22.jpg)
4/29/2016
John Maynard Keynes - 1921
Bruno de Finetti - 1931 Andrey Kolmogorov - 1933
Sir Harold Jeffreys - 1939
Richard Threlkeld Cox - 1946
Edwin Thompson Jaynes - 1957
Claude Shannon - 1948
Probability Theory Timeline
1920
1930
1940
1950
1960
Knuth - Bayes Forum 22
![Page 23: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/23.jpg)
4/29/2016
John Maynard Keynes - 1921
Bruno de Finetti - 1931 Andrey Kolmogorov - 1933
Sir Harold Jeffreys - 1939
Richard Threlkeld Cox - 1946
Edwin Thompson Jaynes - 1957
Claude Shannon - 1948
Probability Theory Timeline
1920
1930
1940
1950
1960
Knuth - Bayes Forum 23
![Page 24: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/24.jpg)
4/29/2016
John Maynard Keynes - 1921
Bruno de Finetti - 1931 Andrey Kolmogorov - 1933
Sir Harold Jeffreys - 1939
Richard Threlkeld Cox - 1946
Edwin Thompson Jaynes - 1957
Claude Shannon - 1948
Probability Theory Timeline
1920
1930
1940
1950
1960
Knuth - Bayes Forum 24
![Page 25: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/25.jpg)
4/29/2016
John Maynard Keynes - 1921
Bruno de Finetti - 1931 Andrey Kolmogorov - 1933
Sir Harold Jeffreys - 1939
Richard Threlkeld Cox - 1946
Edwin Thompson Jaynes - 1957
Claude Shannon - 1948
Probability Theory Timeline
1920
1930
1940
1950
1960
Knuth - Bayes Forum 25
![Page 26: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/26.jpg)
4/29/2016
John Maynard Keynes - 1921
Bruno de Finetti - 1931 Andrey Kolmogorov - 1933
Sir Harold Jeffreys - 1939
Richard Threlkeld Cox - 1946
Edwin Thompson Jaynes - 1957
Claude Shannon - 1948
Probability Theory Timeline
1920
1930
1940
1950
1960
Knuth - Bayes Forum 26
![Page 27: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/27.jpg)
4/29/2016
John Maynard Keynes - 1921
Bruno de Finetti - 1931 Andrey Kolmogorov - 1933
Sir Harold Jeffreys - 1939
Richard Threlkeld Cox - 1946
Edwin Thompson Jaynes - 1957
Claude Shannon - 1948
1920
1930
1940
1950
1960
Knuth - Bayes Forum 27
Probability Theory Timeline
![Page 28: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/28.jpg)
4/29/2016
John Maynard Keynes - 1921
Bruno de Finetti - 1931 Andrey Kolmogorov - 1933
Sir Harold Jeffreys - 1939
Richard Threlkeld Cox - 1946
Edwin Thompson Jaynes - 1957
Claude Shannon - 1948
Probability Theory Timeline
1920
1930
1940
1950
1960
Quantum Mechanics Timeline
Erwin Schrödinger - 1926
Werner Heisenberg – 1932 (NP)
John Von Neumann - 1936
Richard Feynman - 1948
1920
1930
1940
1950
1960
Niels Bohr – 1922 (NP)
Knuth - Bayes Forum 28
![Page 29: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/29.jpg)
4/29/2016 Knuth - Bayes Forum 29
Familiarity breeds the illusion of understanding
Anonymous
![Page 30: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/30.jpg)
In graduate school I asked:
why
results in
1 + 2 = 3 30 Knuth - Bayes Forum 4/29/2016
= ⋁
![Page 31: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/31.jpg)
=
1 + 2 = 3 31 Knuth - Bayes Forum 4/29/2016
⋁
![Page 32: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/32.jpg)
=
1 + 2 = 3 32 Knuth - Bayes Forum 4/29/2016
⋁
![Page 33: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/33.jpg)
A B = A B
𝑣 𝐴 ∪ 𝐵 = 𝑣 𝐴 + 𝑣(𝐵)
33 Knuth - Bayes Forum 4/29/2016
⋁
![Page 34: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/34.jpg)
=
𝑣 𝐴 ∪ 𝐵 = 𝑣 𝐴 + 𝑣 𝐵 − 𝑣(𝐴 ∩ 𝐵)
volume
34 Knuth - Bayes Forum 4/29/2016
A B A B
Knuth, MaxEnt 2003
⋁
![Page 35: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/35.jpg)
𝑠 𝐴 ∪ 𝐵 = 𝑠 𝐴 + 𝑠 𝐵 − 𝑠(𝐴 ∩ 𝐵)
= A B
surface area
35 Knuth - Bayes Forum 4/29/2016
A B
Knuth, MaxEnt 2003
⋁
![Page 36: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/36.jpg)
sum rule of probability
I)|BPr(AI)|Pr(BI)|Pr(A)|Pr(A IB
36 Knuth - Bayes Forum 4/29/2016
Knuth, MaxEnt 2003
![Page 37: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/37.jpg)
𝐼(𝐴; 𝐵) = 𝐻(𝐴) + 𝐻(𝐵) − 𝐻(𝐴, 𝐵)
mutual information
37 Knuth - Bayes Forum 4/29/2016
Knuth, MaxEnt 2003
![Page 38: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/38.jpg)
𝑚𝑎𝑥 𝑎, 𝑏 = 𝑎 + 𝑏 −𝑚𝑖𝑛(𝑎, 𝑏)
polya’s min-max rule
38 Knuth - Bayes Forum 4/29/2016
Knuth, MaxEnt 2003
![Page 39: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/39.jpg)
log 𝐿𝐶𝑀(𝑎, 𝑏)= log 𝑎 + log 𝑏 − log(𝐺𝐶𝐷(𝑎, 𝑏))
number theory identity
39 Knuth - Bayes Forum 4/29/2016
Knuth, MaxEnt 2009
![Page 40: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/40.jpg)
Clearly, my original question:
why
results in
1 + 2 = 3 40 Knuth - Bayes Forum 4/29/2016
= ⋁
![Page 41: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/41.jpg)
Is related to:
why the disjunction of A and B results in
I)|BPr(AI)|Pr(BI)|Pr(A)|Pr(A IB
41 Knuth - Bayes Forum 4/29/2016
![Page 42: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/42.jpg)
the essential content of both statistical mechanics and communication theory, of course, does not lie in the equations; it lies in the ideas that lead to those equations E. T. Jaynes
42 Knuth - Bayes Forum 4/29/2016
![Page 43: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/43.jpg)
the essential content of both statistical mechanics and communication theory, of course, does not lie in the equations; it lies in the ideas that lead to those equations E. T. Jaynes
43 Knuth - Bayes Forum 4/29/2016
![Page 44: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/44.jpg)
the essential content of both statistical mechanics and communication theory, of course, does not lie in the equations; it lies in the ideas that lead to those equations E. T. Jaynes
44 Knuth - Bayes Forum 4/29/2016
![Page 45: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/45.jpg)
4/29/2016 Knuth - Bayes Forum 45
A MODERN PERSPECTIVE
![Page 46: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/46.jpg)
Measure what is measurable,
and make measurable that which is not so.
Galileo Galilei
46 Knuth - Bayes Forum 4/29/2016
![Page 47: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/47.jpg)
Laws of the Universe Reflect
An Underlying Order
Underlying Order Constrains
Quantification
Laws are fundamental and are dictated by
God or Mother Nature
Order and symmetries are fundamental
Laws are constraints on quantification
47 Knuth - Bayes Forum 4/29/2016
ideas that lead to equations
Paradigm Shift
![Page 48: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/48.jpg)
When hypothesizing Laws, one can be right or wrong
Applying consistent quantification can only be useful or not useful
whereas
48 Knuth - Bayes Forum 4/29/2016
Methodology
![Page 49: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/49.jpg)
4/29/2016 Knuth - Bayes Forum 49
Lattices
Lattices are partially ordered sets where each pair of elements has a least upper bound and a greatest lower bound
![Page 50: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/50.jpg)
4/29/2016 Knuth - Bayes Forum
aba
bbaba
Structural Viewpoint
Operational Viewpoint
Lattices are Algebras
Lattices
50
![Page 51: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/51.jpg)
4/29/2016 Knuth - Bayes Forum
aba
bbaba
Structural Viewpoint
Operational Viewpoint
aba
bbaba
Assertions, Implies
aba
bbaba
Sets, Is a subset of
aba
bbaba
),gcd(
),lcm(|
Positive Integers, Divides
aba
bbaba
),min(
),max(
Integers, Is less than or equal to
Lattices and Ubiquity
51
![Page 52: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/52.jpg)
quantify the partial order ≡ assign real numbers to the elements
Require that quantification be consistent with the structure. Otherwise, information about the partial order is lost.
{ a, b } { a, c } { b, c }
{ a } { b } { c }
{ a, b, c }
52 Knuth - Bayes Forum 4/29/2016
𝑓: 𝑥 ∈ 𝐿 → ℝ
Quantification
![Page 53: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/53.jpg)
Enforce local consistency
𝑥 𝑦
Any general rule must hold for special cases Look at special cases to constrain general rule
where ⊕ is an unknown operator to be determined.
53 Knuth - Bayes Forum 4/29/2016
𝑥 ∨ 𝑦
𝑓: 𝑥 ∈ 𝐿 → ℝ
𝑓 𝑥 ∨ 𝑦 = 𝑓 𝑥 ⊕ 𝑓 𝑦
Local Consistency
![Page 54: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/54.jpg)
Write the same element two different ways
which implies
Note that the unknown operator ⨁ is nested in two distinct ways, which reflects associativity
54 Knuth - Bayes Forum 4/29/2016
Associativity of Join
𝑥 ∨ 𝑦 ∨ 𝑧 = 𝑥 ∨ 𝑦 ∨ 𝑧
𝑓 𝑥 ⊕ 𝑓 𝑦 ⊕ 𝑓 𝑧 = 𝑓 𝑥 ⊕ 𝑓 𝑦 ⊕ 𝑓 𝑧
![Page 55: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/55.jpg)
55 Knuth - Bayes Forum 4/29/2016
Associativity Equation
where the aim is to find all the possible operators ⊕ that satisfy the equation above.
We require that the join operations are closed, That the valuations respect ranking, i.e. 𝑥 ≥ 𝑦 ⇒ 𝑓 𝑥 ≥ 𝑓 𝑦 And that ⊕ is commutative and associative.
𝑓 𝑥 ⊕ 𝑓 𝑦 ⊕ 𝑓 𝑧 = 𝑓 𝑥 ⊕ 𝑓 𝑦 ⊕ 𝑓 𝑧
This is a functional equation known as the Associativity Equation
![Page 56: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/56.jpg)
The general solution to the Associativity Equation
where 𝐹 is an arbitrary invertible function.
56 Knuth - Bayes Forum 4/29/2016
𝑓 𝑥 ⊕ 𝑓 𝑦 ⊕ 𝑓 𝑧 = 𝑓 𝑥 ⊕ 𝑓 𝑦 ⊕ 𝑓 𝑧
Associativity Equation
is (Aczel 1966; Craigen and Pales 1989; Knuth and Skilling 2012):
𝐹 𝑓 𝑥 ⊕ 𝑓 𝑦 = 𝐹 𝑓 𝑥 + 𝐹 𝑓 𝑦
![Page 57: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/57.jpg)
57 Knuth - Bayes Forum 4/29/2016
Regraduation
Since the function 𝐹 is arbitrary and invertible, we can define a
new quantification 𝑣 𝑥 = 𝐹 𝑓 𝑥 so that the combination is
always additive. Thus we can always write
𝐹 𝑓 𝑥 ⊕ 𝑓 𝑦 = 𝐹 𝑓 𝑥 + 𝐹 𝑓 𝑦
𝑣 𝑥 ∨ 𝑦 = 𝑣 𝑥 + 𝑣 𝑦
In essence, we have derived measure theory from algebraic symmetries.
![Page 58: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/58.jpg)
Additivity
𝑥 𝑦
58 Knuth - Bayes Forum 4/29/2016
Knuth, MaxEnt 2009
𝑣 𝑥 ∨ 𝑦 = 𝑣 𝑥 + 𝑣 𝑦
𝑥 ∨ 𝑦
Additivity
![Page 59: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/59.jpg)
Epiphany! 59 Knuth - Bayes Forum 4/29/2016
![Page 60: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/60.jpg)
1 + 2 = 3
60 Knuth - Bayes Forum 4/29/2016
+ = always results in
because combining crayons in this way is closed, commutative, associative,
and I can order sets of crayons.
Why We Sum
![Page 61: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/61.jpg)
𝑥 𝑦
𝑥 ˅ 𝑦
𝑥 ˄ 𝑦 𝑧
More General Cases
General Case
4/29/2016 Knuth - Bayes Forum 61
![Page 62: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/62.jpg)
General Case
4/29/2016 Knuth - Bayes Forum 62
More General Cases
𝑥 𝑦
𝑥 ˅ 𝑦
𝑥 ˄ 𝑦 𝑧
𝑣 𝑦 = 𝑣 𝑥 ∧ 𝑦 + 𝑣 𝑧
![Page 63: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/63.jpg)
General Case
4/29/2016 Knuth - Bayes Forum 63
More General Cases
𝑥 𝑦
𝑥 ˅ 𝑦
𝑥 ˄ 𝑦 𝑧
𝑣 𝑦 = 𝑣 𝑥 ∧ 𝑦 + 𝑣 𝑧 𝑣 𝑥 ∨ 𝑦 = 𝑣 𝑥 + 𝑣 𝑧
![Page 64: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/64.jpg)
General Case
4/29/2016 Knuth - Bayes Forum 64
More General Cases
𝑥 𝑦
𝑥 ˅ 𝑦
𝑥 ˄ 𝑦 𝑧
𝑣 𝑦 = 𝑣 𝑥 ∧ 𝑦 + 𝑣 𝑧 𝑣 𝑥 ∨ 𝑦 = 𝑣 𝑥 + 𝑣 𝑧
𝑣 𝑥 ∨ 𝑦 = 𝑣 𝑥 + 𝑣 𝑦 − 𝑣 𝑥 ∧ 𝑦
![Page 65: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/65.jpg)
symmetric form (self-dual)
Sum Rule
4/29/2016 Knuth - Bayes Forum 65
The Sum Rule
𝑣 𝑥 ∨ 𝑦 = 𝑣 𝑥 + 𝑣 𝑦 − 𝑣 𝑥 ∧ 𝑦
𝑣 𝑥 ∨ 𝑦 + 𝑣 𝑥 ∧ 𝑦 = 𝑣 𝑥 + 𝑣 𝑦
![Page 66: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/66.jpg)
4/29/2016
A Curious Observation
Knuth - Bayes Forum 66
Fundamental symmetries are why the Sum Rule is ubiquitous
Pr 𝐴 ∨ 𝐵 𝐶) = Pr 𝐴 𝐶) + Pr 𝐵 𝐶) − Pr 𝐴 ∧ 𝐵 𝐶) Probability
𝐼 𝐴; 𝐵 = 𝐻 𝐴 + 𝐻 𝐵 − 𝐻 𝐴, 𝐵 Mutual Information
𝐴𝑟𝑒𝑎 𝐴 ∪ 𝐵 = 𝐴𝑟𝑒𝑎 𝐴 + 𝐴𝑟𝑒𝑎 𝐵 − 𝐴𝑟𝑒𝑎 𝐴 ∩ 𝐵 Areas of Sets
max 𝐴, 𝐵 = 𝐴 + 𝐵 −min 𝐴, 𝐵 Polya’s Min-Max Rule
log 𝐿𝐶𝑀 𝐴, 𝐵 = log𝐴 + log𝐵 − log𝐺𝐶𝐷 𝐴, 𝐵 Integral Divisors
Ubiquity (inclusion-exclusion)
𝐼3 𝐴, 𝐵, 𝐶 = 𝐴⨆𝐵⨆𝐶 − 𝐴⨆𝐵 − 𝐴⨆𝐶 − 𝐵⨆𝐶 + 𝐴 + 𝐵 + |𝐶| Amplitudes from three-slits (Sorkin arXiv:\\gr-qc/9401003)
The relations above are constraint equations ensuring consistent quantification in the face of certain symmetries
Knuth, 2003. Deriving Laws, arXiv:physics/0403031 [physics.data-an] Knuth, 2009. Measuring on Lattices, arXiv:0909.3684 [math.GM]
Knuth, 2015. The Deeper Roles of Mathematics in Physical Laws, arXiv:1504.06686 [math.HO]
![Page 67: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/67.jpg)
4/29/2016 Knuth - Bayes Forum 67
INFERENCE
![Page 68: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/68.jpg)
apple banana cherry
states of the contents of my grocery basket
What can be said about a system?
states
4/29/2016 Knuth - Bayes Forum 68
![Page 69: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/69.jpg)
crudely describe knowledge by listing a set of potential states
powerset
states of the contents of my grocery basket
statements about the contents of
my grocery basket
subset inclusion
a b c
{ a, b } { a, c } { b, c }
{ a } { b } { c }
{ a, b, c }
What can be said about a system?
4/29/2016 Knuth - Bayes Forum 69
![Page 70: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/70.jpg)
ordering encodes implication DEDUCTION
statements about the contents of
my grocery basket
implies
{ a, b } { a, c } { b, c }
{ a } { b } { c }
{ a, b, c }
What can be said about a system?
4/29/2016 Knuth - Bayes Forum 70
![Page 71: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/71.jpg)
statements about the contents of
my grocery basket
inference works backwards
Quantify to what degree the statement that the system is in
one of three states {a, b, c} implies knowing that it is in some other set of states
{ a, b } { a, c } { b, c }
{ a } { b } { c }
{ a, b, c }
What can be said about a system?
4/29/2016 Knuth - Bayes Forum 71
![Page 72: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/72.jpg)
4/29/2016 Knuth - Bayes Forum
yxif
yxifyx
0
1),(
The Zeta function encodes inclusion on the lattice.
{ a, b } { a, c } { b, c }
{ a } { b } { c }
{ a, b, c }
Inclusion and the Zeta Function
72
![Page 73: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/73.jpg)
Valuation Bi-Valuation
v(x)i)|p(x (x)vi
Measure of x with respect to
Context i
Context i is implicit
Context i is explicit
Bi-valuations generalize lattice inclusion to degrees of inclusion
BI-VALUATION RLix,:p
Quantifying Lattices
Context and Bi-Valuations
4/29/2016 Knuth - Bayes Forum 73
![Page 74: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/74.jpg)
i)|yp(xi)|yp(xi)|p(yi)|p(x
Sum Rule
Context is Explicit
Quantifying Lattices
4/29/2016 Knuth - Bayes Forum 74
![Page 75: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/75.jpg)
=
Associativity of Context
Quantifying Lattices
4/29/2016 Knuth - Bayes Forum 75
![Page 76: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/76.jpg)
c)|p(bb)|p(ac)|p(a
a
c
b
Chain Rule
Quantifying Lattices
4/29/2016 Knuth - Bayes Forum 76
![Page 77: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/77.jpg)
x)|yp(xx)|yp(xx)|p(yx)|p(x
Since x ≤ x and x ≤ x˅y, p(x | x) = 1 and p(x˅y | x) = 1
x)|yp(xx)|p(y x y
x ˄ y
x ˅ y
Lemma
Quantifying Lattices
4/29/2016 Knuth - Bayes Forum 77
![Page 78: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/78.jpg)
y)x|zyp(xx)|yp(xx)|zyp(x
y x z
x ˄ y y ˄ z
x ˄ y ˄ z
Extending the Chain Rule
Quantifying Lattices
4/29/2016 Knuth - Bayes Forum 78
![Page 79: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/79.jpg)
y)x|p(zx)|p(yx)|zp(y
y x z
x ˄ y y ˄ z
x ˄ y ˄ z
Extending the Chain Rule
Quantifying Lattices
4/29/2016 Knuth - Bayes Forum 79
y)x|zyp(xx)|yp(xx)|zyp(x
![Page 80: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/80.jpg)
y x z
x ˄ y y ˄ z
x ˄ y ˄ z
Extending the Chain Rule
Quantifying Lattices
4/29/2016 Knuth - Bayes Forum 80
y)x|p(zx)|p(yx)|zp(y
y)x|zyp(xx)|yp(xx)|zyp(x
![Page 81: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/81.jpg)
y x z
x ˄ y y ˄ z
x ˄ y ˄ z
Extending the Chain Rule
Quantifying Lattices
4/29/2016 Knuth - Bayes Forum 81
y)x|zyp(xx)|yp(xx)|zyp(x
y)x|p(zx)|p(yx)|zp(y
![Page 82: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/82.jpg)
y x z
x ˄ y y ˄ z
x ˄ y ˄ z
Extending the Chain Rule
Quantifying Lattices
4/29/2016 Knuth - Bayes Forum 82
y)x|zyp(xx)|yp(xx)|zyp(x
y)x|p(zx)|p(yx)|zp(y
![Page 83: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/83.jpg)
Commutativity of the product leads to Bayes Theorem…
Bayes Theorem involves a change of context.
i)|p(y
i)|p(xi)x|p(yi)y|p(x
i)|p(y
i)|p(xx)|p(yy)|p(x
Quantifying Lattices
4/29/2016 Knuth - Bayes Forum 83
![Page 84: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/84.jpg)
x =
Direct (Cartesian) product of two spaces
Lattice Products
Quantifying Lattices
4/29/2016 Knuth - Bayes Forum 84
![Page 85: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/85.jpg)
The lattice product is also associative
CB)(AC)(BA
After the sum rule, the only freedom left is rescaling
which is again summation (after taking the logarithm)
Direct Product Rule
Quantifying Lattices
4/29/2016 Knuth - Bayes Forum 85
j)|p(bi)|p(aj)i,|bp(a,
![Page 86: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/86.jpg)
i)|yp(xi)|p(yi)|p(xi)|yp(x
Sum Rule
y)x|p(zx)|p(yx)|zp(y
Product Rule
i)|p(y
i)|p(xx)|p(yy)|p(x
Bayes Theorem
j)|p(bi)|p(aj)i,|bp(a,
Direct Product Rule
Bayesian Probability Theory = Constraint Equations
4/29/2016 Knuth - Bayes Forum 86
![Page 87: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/87.jpg)
statements
Given a quantification of the join-irreducible elements,
one uses the constraint equations to consistently
assign any desired bi-valuations (probability)
Inference
{ a, b } { a, c } { b, c }
{ a } { b } { c }
{ a, b, c }
4/29/2016 Knuth - Bayes Forum 87
![Page 88: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/88.jpg)
How far can we take these ideas?
88 Knuth - Bayes Forum 4/29/2016
Quantum Mechanics!
![Page 89: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/89.jpg)
89 Knuth - Bayes Forum 4/29/2016
Quantum Measurements in Series
Quantum measurements can be performed in series. Series combinations of measurement sequences are associative.
![Page 90: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/90.jpg)
90 Knuth - Bayes Forum 4/29/2016
Quantum Measurements in Parallel
A B C
Quantum measurements can be performed in parallel (coarse graining). Parallel combinations of measurement sequences are commutative and associative.
![Page 91: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/91.jpg)
91 Knuth - Bayes Forum 4/29/2016
Consistent Quantification of Quantum Measurement Sequences
By quantifying a measurement sequence with a pair of numbers, 𝑎 =𝑎1𝑎2
,
Associativity and Commutativity of Parallel combinations of measurements results in component-wise additivity of the pairs:
A
B C
𝑐 =𝑐1𝑐2
=𝑎1 + 𝑏1𝑎2 + 𝑏2
Distributivity of Series over Parallel combinations of measurements results in a bilinear multiplicative form for combining the pairs:
= ∨
C B
A
𝑐 =𝑐1𝑐2
=𝛾1𝑎1𝑏1 + 𝛾2𝑎1𝑏2 + 𝛾3𝑎2𝑏1 + 𝛾4𝑎2𝑏2𝛾5𝑎1𝑏1 + 𝛾6𝑎1𝑏2 + 𝛾7𝑎2𝑏1 + 𝛾8𝑎2𝑏2
![Page 92: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/92.jpg)
92 Knuth - Bayes Forum 4/29/2016
Quantum Measurement Sequences
One can then show that the probabilities of measurement sequences
are given by the Born Rule, where for 𝑎 =𝑎1𝑎2
, 𝐏 𝐀 = 𝐩 𝐚 = 𝑎1𝟐 + 𝑎2
𝟐
![Page 93: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/93.jpg)
93 Knuth - Bayes Forum 4/29/2016
Quantum Mechanics and Inference
![Page 94: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/94.jpg)
4/29/2016
Foundations are Important. A solid foundation acts as a broad base on which theories can be constructed to unify seemingly disparate phenomena.
Knuth - Bayes Forum 94
Cox’s Approach
(degrees of rational belief)
Boolean Algebra
Distributive Algebra
Associativity & Order
![Page 95: A Modern History of Probability Theoryaws/knuth-mpi-talk---final.pdf · 2016. 5. 2. · A Modern History of Probability Theory Kevin H. Knuth Depts. of Physics and Informatics University](https://reader035.vdocument.in/reader035/viewer/2022070215/6118c757734f135611341d0e/html5/thumbnails/95.jpg)
4/29/2016
THANK YOU
Knuth - Bayes Forum 95
Knuth K.H., Skilling J. 2012. Foundations of Inference. Axioms 1:38-73. arXiv:1008.4831 [math.PR] Goyal P., Knuth K.H., Skilling J. 2010. Origin of complex quantum amplitudes and Feynman's rules, Physical Review A 81, 022109. arXiv:0907.0909v3 [quant-ph]