![Page 1: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/1.jpg)
1
Basics of information theory and information complexity
June 1, 2013
Mark Braverman
Princeton University
a tutorial
![Page 2: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/2.jpg)
Part I: Information theory
β’ Information theory, in its modern format was introduced in the 1940s to study the problem of transmitting data over physical channels.
2
communication channel
Alice Bob
![Page 3: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/3.jpg)
Quantifying βinformationβ
β’ Information is measured in bits.
β’ The basic notion is Shannonβs entropy.
β’ The entropy of a random variable is the (typical) number of bits needed to remove the uncertainty of the variable.
β’ For a discrete variable: π» π β βPr π = π₯ log 1/Pr[π = π₯]
3
![Page 4: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/4.jpg)
Shannonβs entropy
β’ Important examples and properties:
β If π = π₯ is a constant, then π» π = 0.
β If π is uniform on a finite set π of possible values, then π» π = log π.
β If π is supported on at most π values, then π» X β€ log π.
β If π is a random variable determined by π, then π» π β€ π»(π).
4
![Page 5: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/5.jpg)
Conditional entropy
β’ For two (potentially correlated) variables π, π, the conditional entropy of π given π is the amount of uncertainty left in π given π:
π» π π β πΈπ¦~πH X Y = y .
β’ One can show π» ππ = π» π + π»(π|π).
β’ This important fact is knows as the chain rule.
β’ If π β₯ π, then π» ππ = π» π + π» π π = π» π + π» π .
5
![Page 6: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/6.jpg)
Example
β’ π = π΅1, π΅2, π΅3
β’ π = π΅1 βπ΅2 , π΅2 βπ΅4 , π΅3 βπ΅4 , π΅5
β’ Where π΅1, π΅2, π΅3, π΅4, π΅5 βπ {0,1}.
β’ Then
β π» π = 3; π» π = 4; π» ππ = 5;
β π» π π = 1 = π» ππ β π» π ;
β π» π π = 2 = π» ππ β π» π .
6
![Page 7: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/7.jpg)
Mutual information
β’ π = π΅1, π΅2, π΅3
β’ π = π΅1 βπ΅2 , π΅2 βπ΅4 , π΅3 βπ΅4 , π΅5
7
π»(π) π»(π) π»(π|π) π»(π|π)
π΅1 π΅1 βπ΅2
π΅2 βπ΅3
π΅4
π΅5
πΌ(π; π)
![Page 8: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/8.jpg)
Mutual information
β’ The mutual information is defined as πΌ π; π = π» π β π» π π = π» π β π»(π|π)
β’ βBy how much does knowing π reduce the entropy of π?β
β’ Always non-negative πΌ π; π β₯ 0.
β’ Conditional mutual information: πΌ π; π π β π» π π β π»(π|ππ)
β’ Chain rule for mutual information: πΌ ππ; π = πΌ π; π + πΌ(π; π|π)
β’ Simple intuitive interpretation.
8
![Page 9: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/9.jpg)
Example β a biased coin
β’ A coin with π-Heads or Tails bias is tossed several times.
β’ Let π΅ β {π», π} be the bias, and suppose that a-priori both options are equally likely: π» π΅ = 1.
β’ How many tosses needed to find π΅?
β’ Let π1, β¦ , ππ be a sequence of tosses.
β’ Start with π = 2. 9
![Page 10: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/10.jpg)
What do we learn about π΅? β’ πΌ π΅; π1π2 = πΌ π΅; π1 + πΌ π΅; π2 π1 =
πΌ π΅; π1 + πΌ π΅π1; π2 β πΌ π1; π2
β€ πΌ π΅; π1 + πΌ π΅π1; π2 = πΌ π΅; π1 + πΌ π΅; π2 + πΌ π1; π2 π΅
= πΌ π΅; π1 + πΌ π΅; π2 = 2 β πΌ π΅; π1 .
β’ Similarly, πΌ π΅; π1β¦ππ β€ π β πΌ π΅; π1 .
β’ To determine π΅ with constant accuracy, need 0 < c < πΌ π΅; π1β¦ππ β€ π β πΌ π΅; π1 .
β’ π = Ξ©(1/πΌ(π΅; π1)). 10
![Page 11: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/11.jpg)
KullbackβLeibler (KL)-Divergence
β’ A distance metric between distributions on the same space.
β’ Plays a key role in information theory.
π·(π β₯ π) β π π₯ logπ[π₯]
π[π₯].
π₯
β’ π·(π β₯ π) β₯ 0, with equality when π = π.
β’ Caution: π·(π β₯ π) β π·(π β₯ π)!
11
![Page 12: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/12.jpg)
Properties of KL-divergence
β’ Connection to mutual information: πΌ π; π = πΈπ¦βΌππ·(ππ=π¦ β₯ π).
β’ If π β₯ π, then ππ=π¦ = π, and both sides are 0.
β’ Pinskerβs inequality:
π β π 1 = π( π·(π β₯ π)).
β’ Tight!
π·(π΅1/2+π β₯ π΅1/2) = Ξ(π2).
12
![Page 13: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/13.jpg)
Back to the coin example
β’ πΌ π΅; π1 = πΈπβΌπ΅π·(π1,π΅=π β₯ π1) = π· π΅1
2Β±π
β₯ π΅12
= Ξ π2 .
β’ π = Ξ©1
πΌ π΅;π1= Ξ©(
1
π2).
β’ βFollow the information learned from the coin tossesβ
β’ Can be done using combinatorics, but the information-theoretic language is more natural for expressing whatβs going on.
![Page 14: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/14.jpg)
Back to communication
β’ The reason Information Theory is so important for communication is because information-theoretic quantities readily operationalize.
β’ Can attach operational meaning to Shannonβs entropy: π» π β βthe cost of transmitting πβ.
β’ Let πΆ π be the (expected) cost of transmitting a sample of π.
14
![Page 15: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/15.jpg)
π» π = πΆ(π)?
β’ Not quite.
β’ Let trit π βπ 1,2,3 .
β’ πΆ π =5
3β 1.67.
β’ π» π = log 3 β 1.58.
β’ It is always the case that πΆ π β₯ π»(π).
15
1 0
2 10
3 11
![Page 16: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/16.jpg)
But π» π and πΆ(π) are close
β’ Huffmanβs coding: πΆ π β€ π» π + 1.
β’ This is a compression result: βan uninformative message turned into a short oneβ.
β’ Therefore: π» π β€ πΆ π β€ π» π + 1.
16
![Page 17: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/17.jpg)
Shannonβs noiseless coding β’ The cost of communicating many copies of π
scales as π»(π).
β’ Shannonβs source coding theorem:
β Let πΆ ππ be the cost of transmitting π independent copies of π. Then the amortized transmission cost
limπββ
πΆ(ππ)/π = π» π .
β’ This equation gives π»(π) operational meaning.
17
![Page 18: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/18.jpg)
communication channel
π» π ππππππ‘πππππππ§ππ
18
π1, β¦ , ππ, β¦ π»(π) per copy to transmit πβs
![Page 19: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/19.jpg)
π»(π) is nicer than πΆ(π)
β’ π» π is additive for independent variables.
β’ Let π1, π2 βπ {1,2,3} be independent trits.
β’ π» π1π2 = log 9 = 2 log 3.
β’ πΆ π1π2 =29
9< πΆ π1 + πΆ π2 = 2 Γ
5
3=
30
9.
β’ Works well with concepts such as channel capacity.
19
![Page 20: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/20.jpg)
βProofβ of Shannonβs noiseless coding
β’ π β π» π = π» ππ β€ πΆ ππ β€ π» ππ + 1.
β’ Therefore limπββ
πΆ(ππ)/π = π» π .
20
Additivity of entropy
Compression (Huffman)
![Page 21: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/21.jpg)
Operationalizing other quantities
β’ Conditional entropy π» π π :
β’ (cf. Slepian-Wolf Theorem).
communication channel
π1, β¦ , ππ, β¦ π»(π|π) per copy to transmit πβs
π1, β¦ , ππ, β¦
![Page 22: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/22.jpg)
communication channel
Operationalizing other quantities
β’ Mutual information πΌ π; π :
π1, β¦ , ππ, β¦ πΌ π; π per copy to sample πβs
π1, β¦ , ππ, β¦
![Page 23: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/23.jpg)
Information theory and entropy
β’ Allows us to formalize intuitive notions.
β’ Operationalized in the context of one-way transmission and related problems.
β’ Has nice properties (additivity, chain ruleβ¦)
β’ Next, we discuss extensions to more interesting communication scenarios.
23
![Page 24: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/24.jpg)
Communication complexity
β’ Focus on the two party randomized setting.
24
A B
X Y A & B implement a functionality πΉ(π, π).
F(X,Y)
e.g. πΉ π, π = βπ = π? β
Shared randomness R
![Page 25: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/25.jpg)
Communication complexity
A B
X Y
Goal: implement a functionality πΉ(π, π). A protocol π(π, π) computing πΉ(π, π):
F(X,Y)
m1(X,R)
m2(Y,m1,R)
m3(X,m1,m2,R)
Communication cost = #of bits exchanged.
Shared randomness R
![Page 26: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/26.jpg)
Communication complexity
β’ Numerous applications/potential applications (some will be discussed later today).
β’ Considerably more difficult to obtain lower bounds than transmission (still much easier than other models of computation!).
26
![Page 27: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/27.jpg)
Communication complexity
β’ (Distributional) communication complexity with input distribution π and error π: πΆπΆ πΉ, π, π . Error β€ π w.r.t. π.
β’ (Randomized/worst-case) communication complexity: πΆπΆ(πΉ, π). Error β€ π on all inputs.
β’ Yaoβs minimax:
πΆπΆ πΉ, π = maxπ
πΆπΆ(πΉ, π, π).
27
![Page 28: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/28.jpg)
Examples
β’ π, π β 0,1 π.
β’ Equality πΈπ π, π β 1π=π.
β’ πΆπΆ πΈπ, π β log1
π.
β’ πΆπΆ πΈπ, 0 β π.
28
![Page 29: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/29.jpg)
Equality β’πΉis βπ = π? β. β’πis a distribution where w.p. Β½π = π and w.p. Β½ (π, π) are random.
A B
X Y
β’ Shows that πΆπΆ πΈπ, π, 2β129 β€ 129.
MD5(X) [128 bits]
X=Y? [1 bit]
Error?
![Page 30: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/30.jpg)
Examples
β’ π, π β 0,1 π.
β’ InnerproductπΌπ π, π β β ππ β πππ (πππ2).
β’ πΆπΆ πΌπ, 0 = π β π(π).
In fact, using information complexity:
β’ πΆπΆ πΌπ, π = π β ππ(π).
30
![Page 31: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/31.jpg)
Information complexity
β’ Information complexity πΌπΆ(πΉ, π)::
communication complexity πΆπΆ πΉ, π
as
β’ Shannonβs entropy π»(π) ::
transmission cost πΆ(π)
31
![Page 32: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/32.jpg)
Information complexity
β’ The smallest amount of information Alice and Bob need to exchange to solve πΉ.
β’ How is information measured?
β’ Communication cost of a protocol?
β Number of bits exchanged.
β’ Information cost of a protocol?
β Amount of information revealed.
32
![Page 33: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/33.jpg)
Basic definition 1: The information cost of a protocol
β’ Prior distribution: π, π βΌ π.
A B
X Y
Protocol Ο Protocol
transcript Ξ
πΌπΆ(π, π) = πΌ(Ξ ; π|π) + πΌ(Ξ ; π|π)
what Alice learns about Y + what Bob learns about X
![Page 34: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/34.jpg)
Example β’πΉis βπ = π? β. β’πis a distribution where w.p. Β½π = π and w.p. Β½ (π, π) are random.
A B
X Y
πΌπΆ(π, π) = πΌ(Ξ ; π|π) + πΌ(Ξ ; π|π) β1 + 64.5 = 65.5 bits
what Alice learns about Y + what Bob learns about X
MD5(X) [128 bits]
X=Y? [1 bit]
![Page 35: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/35.jpg)
Prior π matters a lot for information cost!
β’ If π = 1 π₯,π¦ a singleton,
πΌπΆ π, π = 0.
35
![Page 36: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/36.jpg)
Example β’πΉis βπ = π? β. β’πis a distribution where (π, π) are just uniformly random.
A B
X Y
πΌπΆ(π, π) = πΌ(Ξ ; π|π) + πΌ(Ξ ; π|π) β0 + 128 = 128 bits
what Alice learns about Y + what Bob learns about X
MD5(X) [128 bits]
X=Y? [1 bit]
![Page 37: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/37.jpg)
Basic definition 2: Information complexity
β’ Communication complexity:
πΆπΆ πΉ, π, π β minππππππ’π‘ππ
πΉπ€ππ‘βπππππβ€π
π .
β’ Analogously:
πΌπΆ πΉ, π, π β infππππππ’π‘ππ
πΉπ€ππ‘βπππππβ€π
πΌπΆ(π, π).
37
Needed!
![Page 38: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/38.jpg)
Prior-free information complexity
β’ Using minimax can get rid of the prior.
β’ For communication, we had:
πΆπΆ πΉ, π = maxπ
πΆπΆ(πΉ, π, π).
β’ For information
πΌπΆ πΉ, π β infππππππ’π‘ππ
πΉπ€ππ‘βπππππβ€π
maxπ
πΌπΆ(π, π).
38
![Page 39: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/39.jpg)
Ex: The information complexity of Equality
β’ What is πΌπΆ(πΈπ, 0)?
β’ Consider the following protocol.
39 A B
X in {0,1}n Y in {0,1}n
A non-singular in ππΓπ
A1Β·X
A1Β·Y
A2Β·X
A2Β·Y
Continue for n steps, or until a disagreement is
discovered.
![Page 40: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/40.jpg)
Analysis (sketch)
β’ If Xβ Y, the protocol will terminate in O(1) rounds on average, and thus reveal O(1) information.
β’ If X=Yβ¦ the players only learn the fact that X=Y (β€1 bit of information).
β’ Thus the protocol has O(1) information complexity for any prior π.
40
![Page 41: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/41.jpg)
Operationalizing IC: Information equals amortized communication
β’ Recall [Shannon]: limπββ
πΆ(ππ)/π = π» π .
β’ Turns out: limπββ
πΆπΆ(πΉπ, ππ, π)/π = πΌπΆ πΉ, π, π ,
for π > 0. [Error π allowed on each copy]
β’ For π = 0: limπββ
πΆπΆ(πΉπ, ππ, 0+)/π = πΌπΆ πΉ, π, 0 .
β’ [ limπββ
πΆπΆ(πΉπ, ππ, 0)/π an interesting open
problem.] 41
![Page 42: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/42.jpg)
Information = amortized communication
β’ limπββ
πΆπΆ(πΉπ, ππ, π)/π = πΌπΆ πΉ, π, π .
β’ Two directions: ββ€β and ββ₯β.
β’ π β π» π = π» ππ β€ πΆ ππ β€ π» ππ + 1.
Additivity of entropy
Compression (Huffman)
![Page 43: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/43.jpg)
The ββ€β direction
β’ limπββ
πΆπΆ(πΉπ, ππ, π)/π β€ πΌπΆ πΉ, π, π .
β’ Start with a protocol π solving πΉ, whose πΌπΆ π, π is close to πΌπΆ πΉ, π, π .
β’ Show how to compress many copies of π into a protocol whose communication cost is close to its information cost.
β’ More on compression later.
43
![Page 44: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/44.jpg)
The ββ₯β direction
β’ limπββ
πΆπΆ(πΉπ, ππ, π)/π β₯ πΌπΆ πΉ, π, π .
β’ Use the fact that πΆπΆ πΉπ,ππ,π
πβ₯
πΌπΆ πΉπ,ππ,π
π.
β’ Additivity of information complexity: πΌπΆ πΉπ, ππ, π
π= πΌπΆ πΉ, π, π .
44
![Page 45: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/45.jpg)
Proof: Additivity of information complexity
β’ Let π1(π1, π1) and π2(π2, π2) be two two-party tasks.
β’ E.g. βSolve πΉ(π, π) with error β€ π w.r.t. πβ
β’ Then
πΌπΆ π1 Γ π2, π1 Γ π2 = πΌπΆ π1, π1 + πΌπΆ π2, π2
β’ ββ€β is easy.
β’ ββ₯β is the interesting direction.
![Page 46: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/46.jpg)
πΌπΆ π1, π1 + πΌπΆ π2, π2 β€πΌπΆ π1 Γ π2, π1 Γ π2
β’ Start from a protocol π for π1 Γ π2 with prior π1 Γ π2, whose information cost is πΌ.
β’ Show how to construct two protocols π1 for π1 with prior π1 and π2 for π2 with prior π2, with information costs πΌ1 and πΌ2, respectively, such that πΌ1 + πΌ2 = πΌ.
46
![Page 47: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/47.jpg)
π( π1, π2 , π1, π2 )
π1 π1, π1 β’ Publicly sample π2 βΌ π2 β’ Bob privately samples
π2 βΌ π2|X2
β’ Run π( π1, π2 , π1, π2 )
π2 π2, π2 β’ Publicly sample π1 βΌ π1 β’ Alice privately samples
π1 βΌ π1|Y1
β’ Run π( π1, π2 , π1, π2 )
![Page 48: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/48.jpg)
π1 π1, π1 β’ Publicly sample π2 βΌ π2 β’ Bob privately samples
π2 βΌ π2|X2
β’ Run π( π1, π2 , π1, π2 )
Analysis - π1
β’ Alice learns about π1: πΌ Ξ ; π1 X1X2)
β’ Bob learns about π1: πΌ Ξ ; π1 π1π2π2).
β’ πΌ1 = πΌ Ξ ; π1 X1X2) + πΌ Ξ ; π1 π1π2π2).
48
![Page 49: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/49.jpg)
π2 π2, π2 β’ Publicly sample π1 βΌ π1 β’ Alice privately samples
π1 βΌ π1|Y1
β’ Run π( π1, π2 , π1, π2 )
Analysis - π2
β’ Alice learns about π2: πΌ Ξ ; π2 X1X2Y1)
β’ Bob learns about π2: πΌ Ξ ; π2 π1π2).
β’ πΌ2 = πΌ Ξ ; π2 X1X2Y1) + πΌ Ξ ; π2 π1π2).
49
![Page 50: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/50.jpg)
Adding πΌ1 and πΌ2
πΌ1 + πΌ2 = πΌ Ξ ; π1 X1X2) + πΌ Ξ ; π1 π1π2π2)
+ πΌ Ξ ; π2 X1X2Y1) + πΌ Ξ ; π2 π1π2)
= πΌ Ξ ; π1 X1X2) + πΌ Ξ ; π2 X1X2Y1) +πΌ Ξ ; π2 π1π2) + πΌ Ξ ; π1 π1π2π2) =
πΌ Ξ ; π1π2 π1π2 + πΌ Ξ ; π2π1 π1π2 = πΌ.
50
![Page 51: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/51.jpg)
Summary
β’ Information complexity is additive.
β’ Operationalized via βInformation = amortized communicationβ.
β’ limπββ
πΆπΆ(πΉπ, ππ, π)/π = πΌπΆ πΉ, π, π .
β’ Seems to be the βrightβ analogue of entropy for interactive computation.
51
![Page 52: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/52.jpg)
Entropy vs. Information Complexity
Entropy IC
Additive? Yes Yes
Operationalized limπββ
πΆ(ππ)/π limπββ
πΆπΆ πΉπ, ππ, π
π
Compression? Huffman:
πΆ π β€ π» π + 1 ???!
![Page 53: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/53.jpg)
Can interactive communication be compressed?
β’ Is it true that πΆπΆ πΉ, π, π β€ πΌπΆ πΉ, π, π + π(1)?
β’ Less ambitiously:
πΆπΆ πΉ, π, π(π) = π πΌπΆ πΉ, π, π ?
β’ (Almost) equivalently: Given a protocol π with πΌπΆ π, π = πΌ, can Alice and Bob simulate π using π πΌ communication?
β’ Not known in generalβ¦
53
![Page 54: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/54.jpg)
Direct sum theorems
β’ Let πΉ be any functionality.
β’ Let πΆ(πΉ) be the cost of implementing πΉ.
β’ Let πΉπ be the functionality of implementing π independent copies of πΉ.
β’ The direct sum problem:
βDoes πΆ(πΉπ) β π β πΆ(πΉ)?β
β’ In most cases it is obvious that πΆ(πΉπ) β€ π β πΆ(πΉ).
54
![Page 55: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/55.jpg)
Direct sum β randomized communication complexity
β’ Is it true that
πΆπΆ(πΉπ, ππ, π) = Ξ©(π β πΆπΆ(πΉ, π, π))?
β’ Is it true that πΆπΆ(πΉπ, π) = Ξ©(π β πΆπΆ(πΉ, π))?
55
![Page 56: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/56.jpg)
Direct product β randomized communication complexity
β’ Direct sum
πΆπΆ(πΉπ, ππ, π) = Ξ©(π β πΆπΆ(πΉ, π, π))?
β’ Direct product
πΆπΆ(πΉπ, ππ, (1 β π)π) = Ξ©(π β πΆπΆ(πΉ, π, π))?
56
![Page 57: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/57.jpg)
Direct sum for randomized CC and interactive compression
Direct sum:
β’ πΆπΆ(πΉπ, ππ, π) = Ξ©(π β πΆπΆ(πΉ, π, π))?
In the limit:
β’ π β πΌπΆ(πΉ, π, π) = Ξ©(π β πΆπΆ(πΉ, π, π))?
Interactive compression:
β’ πΆπΆ πΉ, π, π = π(πΌπΆ πΉ, π, π) ?
Same question!
57
![Page 58: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/58.jpg)
The big picture
πΆπΆ(πΉ, π, π) πΆπΆ(πΉπ, ππ, π)/π
πΌπΆ(πΉ, π, π) πΌπΆ(πΉπ, ππ, π)/π
additivity (=direct sum)
for information
information = amortized
communication direct sum for
communication?
interactive compression?
![Page 59: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/59.jpg)
Current results for compression A protocol πthat has πΆ bits of communication, conveys πΌ bits of information over prior π, and works in π rounds can be simulated:
β’ Using π (πΌ + π) bits of communication.
β’ Using π πΌ β πΆ bits of communication.
β’ Using 2π(πΌ) bits of communication.
β’ If π = ππ Γ ππ, then using π(πΌ polylog πΆ) bits of communication. 59
![Page 60: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/60.jpg)
Their direct sum counterparts
β’ πΆπΆ(πΉπ, ππ, π) = Ξ© (π1/2 β πΆπΆ(πΉ, π, π)).
β’ πΆπΆ(πΉπ, π) = Ξ© (π1/2 β πΆπΆ(πΉ, π)).
For product distributions π = ππ Γ ππ,
β’ πΆπΆ(πΉπ, ππ, π) = Ξ© (π β πΆπΆ(πΉ, π, π)).
When the number of rounds is bounded by π βͺ π, a direct sum theorem holds.
60
![Page 61: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/61.jpg)
Direct product
β’ The best one can hope for is a statement of the type:
πΆπΆ(πΉπ, ππ, 1 β 2βπ π ) = Ξ©(π β πΌπΆ(πΉ, π, 1/3)).
β’ Can prove:
πΆπΆ πΉπ, ππ, 1 β 2βπ π = Ξ© (π1/2 β πΆπΆ(πΉ, π, 1/3)).
61
![Page 62: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/62.jpg)
Proof 2: Compressing a one-round protocol
β’ Say Alice speaks: πΌπΆ π, π = πΌ π;π π .
β’ Recall KL-divergence: πΌ π; π π = πΈππ· πππ β₯ ππ = πΈππ· ππ β₯ ππ
β’ Bottom line:
β Alice has ππ; Bob has ππ;
β Goal: sample from ππ using βΌ π·(ππ β₯ ππ) communication.
62
![Page 63: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/63.jpg)
The dart board
q1 q2 q3 q4 q5 q6 q7 β¦. u1 u2 u3 u4 u5 u6 u7
1
0
β’ Interpret the public randomness as random points in π Γ [0,1], where π is the universe of all possible messages.
β’ First message under the histogram of π is distributed βΌ π.
π
u1
u2
u3 u4
u5
![Page 64: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/64.jpg)
64
Proof Idea β’ Sample using π(log1/π + π·(ππ β₯ ππ))
communication with statistical error Ξ΅.
MX MY
u1 u1
u2 u2
u3 u3
u4 u4
u4
~|U| samples Public randomness:
q1 q2 q3 q4 q5 q6 q7 β¦. u1 u2 u3 u4 u5 u6 u7
MX MY 1 1
0 0
![Page 65: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/65.jpg)
Proof Idea β’ Sample using π(log1/π + π·(ππ β₯ ππ))
communication with statistical error Ξ΅.
u4 u2
h1(u4) h2(u4)
65 MX MY u4
1 1
0 0 u2
MX MY
![Page 66: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/66.jpg)
66 66
Proof Idea β’ Sample using π(log1/π + π·(ππ β₯ ππ))
communication with statistical error Ξ΅.
u4 u2
h4(u4)β¦ hlog 1/ Ξ΅(u4)
u4
h3(u4)
MX 2MY
MX MY u4
u4
h1(u4), h2(u4)
1 1
0 0
![Page 67: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/67.jpg)
67
Analysis
β’ If ππ(π’4) β 2πππ(π’4), then the protocol
will reach round π of doubling.
β’ There will be β 2π candidates.
β’ About π + log1/π hashes to narrow to one.
β’ The contribution of π’4 to cost:
β ππ(π’4)(logππ(π’4)/ππ(
π’4) + log1/π).
Done! π·(ππ β₯ ππ) β ππ(π’) logππ(π’)
ππ(π’).
π’
![Page 68: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/68.jpg)
External information cost
β’ (π, π)~π.
A B
X Y
Protocol Ο Protocol
transcript Ο
πΌπΆππ₯π‘(π, π) = πΌ(Ξ ; ππ)
what Charlie learns about (X,Y)
C
![Page 69: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/69.jpg)
Example
69
β’F is βX=Y?β. β’ΞΌ is a distribution where w.p. Β½ X=Y and w.p. Β½ (X,Y) are random.
MD5(X)
X=Y?
A B
X Y
πΌπΆππ₯π‘(π, π) = πΌ(Ξ ; ππ) = 129πππ‘π
what Charlie learns about (X,Y)
![Page 70: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/70.jpg)
External information cost
β’ It is always the case that πΌπΆππ₯π‘ π, π β₯ πΌπΆ π, π .
β’ If π = ππ Γ ππ is a product distribution, then
πΌπΆππ₯π‘ π, π = πΌπΆ π, π .
70
![Page 71: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/71.jpg)
External information complexity
β’ πΌπΆππ₯π‘ πΉ, π, π β infππππππ’π‘ππ
πΉπ€ππ‘βπππππβ€π
πΌπΆππ₯π‘(π, π).
β’ Can it be operationalized?
71
![Page 72: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/72.jpg)
Operational meaning of πΌπΆππ₯π‘?
β’ Conjecture: Zero-error communication scales like external information:
limπββ
πΆπΆ πΉπ, ππ, 0
π= πΌπΆππ₯π‘ πΉ, π, 0 ?
β’ Recall:
limπββ
πΆπΆ πΉπ, ππ, 0+
π= πΌπΆ πΉ, π, 0 .
72
![Page 73: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/73.jpg)
Example β transmission with a strong prior
β’ π, π β 0,1
β’ π is such that π βπ 0,1 , and π = π with a very high probability (say 1 β 1/ π).
β’ πΉ π, π = π is just the βtransmit πβ function.
β’ Clearly, π should just have Alice send π to Bob.
β’ πΌπΆ πΉ, π, 0 = πΌπΆ π, π = π»1
π= o(1).
β’ πΌπΆππ₯π‘ πΉ, π, 0 = πΌπΆππ₯π‘ π, π = 1. 73
![Page 74: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/74.jpg)
Example β transmission with a strong prior
β’ πΌπΆ πΉ, π, 0 = πΌπΆ π, π = π»1
π= o(1).
β’ πΌπΆππ₯π‘ πΉ, π, 0 = πΌπΆππ₯π‘ π, π = 1.
β’ πΆπΆ πΉπ, ππ, 0+ = π π .
β’ πΆπΆ πΉπ, ππ, 0 = Ξ©(π).
Other examples, e.g. the two-bit AND function fit into this picture.
74
![Page 75: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/75.jpg)
Additional directions
75
Information complexity
Interactive coding
Information theory in TCS
![Page 76: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/76.jpg)
Interactive coding theory
β’ So far focused the discussion on noiseless coding.
β’ What if the channel has noise?
β’ [What kind of noise?]
β’ In the non-interactive case, each channel has a capacity πΆ.
76
![Page 77: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/77.jpg)
Channel capacity
β’ The amortized number of channel uses needed to send π over a noisy channel of capacity πΆ is
π» π
πΆ
β’ Decouples the task from the channel!
77
![Page 78: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/78.jpg)
Example: Binary Symmetric Channel
β’ Each bit gets independently flipped with probability π < 1/2.
β’ One way capacity 1 β π» π .
78
0
1
0
1 1 β π
1 β π
π π
![Page 79: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/79.jpg)
Interactive channel capacity
β’ Not clear one can decouple channel from task in such a clean way.
β’ Capacity much harder to calculate/reason about.
β’ Example: Binary symmetric channel.
β’ One way capacity 1 β π» π .
β’ Interactive (for simple pointer jumping,
[Kol-Razβ13]):
1 β Ξ π» π .
0
1
0
1 1 β π
1 β π
π
![Page 80: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/80.jpg)
Information theory in communication complexity and beyond
β’ A natural extension would be to multi-party communication complexity.
β’ Some success in the number-in-hand case.
β’ What about the number-on-forehead?
β’ Explicit bounds for β₯ log π players would imply explicit π΄πΆπΆ0 circuit lower bounds.
80
![Page 81: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/81.jpg)
NaΓ―ve multi-party information cost
πΌπΆ π, π = πΌ(Ξ ; π|ππ)+ πΌ(Ξ ; π|ππ)+ πΌ(Ξ ; π|ππ)
81
A B C
YZ XZ XY
![Page 82: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/82.jpg)
NaΓ―ve multi-party information cost
πΌπΆ π, π = πΌ(Ξ ; π|ππ)+ πΌ(Ξ ; π|ππ)+ πΌ(Ξ ; π|ππ)
β’ Doesnβt seem to work.
β’ Secure multi-party computation [Ben-Or,Goldwasser, Wigderson], means that anything can be computed at near-zero information cost.
β’ Although, these construction require the players to share private channels/randomness.
82
![Page 83: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/83.jpg)
Communication and beyondβ¦
β’ The rest of today:
β Data structures;
β Streaming;
β Distributed computing;
β Privacy.
β’ Exact communication complexity bounds.
β’ Extended formulations lower bounds.
β’ Parallel repetition?
β’ β¦ 83
![Page 84: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/84.jpg)
84
Thank You!
![Page 85: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/85.jpg)
Open problem: Computability of IC
β’ Given the truth table of πΉ π, π , π and π, compute πΌπΆ πΉ, π, π .
β’ Via πΌπΆ πΉ, π, π = limπββ
πΆπΆ(πΉπ, ππ, π)/π can
compute a sequence of upper bounds.
β’ But the rate of convergence as a function of π is unknown.
85
![Page 86: Basics of information theory and information complexitymbraverm/pmwiki/uploads/BravermanSlidesS...Β Β· 1ΰ΅ 2,π1ΰ΅π2 β’Start from a protocol π for 1ΰ΅ 2 with prior π1ΰ΅π2,](https://reader033.vdocument.in/reader033/viewer/2022041800/5e50f07a1b3d1f2a4f57b809/html5/thumbnails/86.jpg)
Open problem: Computability of IC
β’ Can compute the π-roundπΌπΆπ πΉ, π, π information complexity of πΉ.
β’ But the rate of convergence as a function of π is unknown.
β’ Conjecture:
πΌπΆπ πΉ, π, π β πΌπΆ πΉ, π, π = ππΉ,π,π1
π2.
β’ This is the relationship for the two-bit AND.
86