1
Basics of information theory and information complexity
June 1, 2013
Mark Braverman
Princeton University
a tutorial
Part I: Information theory
β’ Information theory, in its modern format was introduced in the 1940s to study the problem of transmitting data over physical channels.
2
communication channel
Alice Bob
Quantifying βinformationβ
β’ Information is measured in bits.
β’ The basic notion is Shannonβs entropy.
β’ The entropy of a random variable is the (typical) number of bits needed to remove the uncertainty of the variable.
β’ For a discrete variable: π» π β βPr π = π₯ log 1/Pr[π = π₯]
3
Shannonβs entropy
β’ Important examples and properties:
β If π = π₯ is a constant, then π» π = 0.
β If π is uniform on a finite set π of possible values, then π» π = log π.
β If π is supported on at most π values, then π» X β€ log π.
β If π is a random variable determined by π, then π» π β€ π»(π).
4
Conditional entropy
β’ For two (potentially correlated) variables π, π, the conditional entropy of π given π is the amount of uncertainty left in π given π:
π» π π β πΈπ¦~πH X Y = y .
β’ One can show π» ππ = π» π + π»(π|π).
β’ This important fact is knows as the chain rule.
β’ If π β₯ π, then π» ππ = π» π + π» π π = π» π + π» π .
5
Example
β’ π = π΅1, π΅2, π΅3
β’ π = π΅1 βπ΅2 , π΅2 βπ΅4 , π΅3 βπ΅4 , π΅5
β’ Where π΅1, π΅2, π΅3, π΅4, π΅5 βπ {0,1}.
β’ Then
β π» π = 3; π» π = 4; π» ππ = 5;
β π» π π = 1 = π» ππ β π» π ;
β π» π π = 2 = π» ππ β π» π .
6
Mutual information
β’ π = π΅1, π΅2, π΅3
β’ π = π΅1 βπ΅2 , π΅2 βπ΅4 , π΅3 βπ΅4 , π΅5
7
π»(π) π»(π) π»(π|π) π»(π|π)
π΅1 π΅1 βπ΅2
π΅2 βπ΅3
π΅4
π΅5
πΌ(π; π)
Mutual information
β’ The mutual information is defined as πΌ π; π = π» π β π» π π = π» π β π»(π|π)
β’ βBy how much does knowing π reduce the entropy of π?β
β’ Always non-negative πΌ π; π β₯ 0.
β’ Conditional mutual information: πΌ π; π π β π» π π β π»(π|ππ)
β’ Chain rule for mutual information: πΌ ππ; π = πΌ π; π + πΌ(π; π|π)
β’ Simple intuitive interpretation.
8
Example β a biased coin
β’ A coin with π-Heads or Tails bias is tossed several times.
β’ Let π΅ β {π», π} be the bias, and suppose that a-priori both options are equally likely: π» π΅ = 1.
β’ How many tosses needed to find π΅?
β’ Let π1, β¦ , ππ be a sequence of tosses.
β’ Start with π = 2. 9
What do we learn about π΅? β’ πΌ π΅; π1π2 = πΌ π΅; π1 + πΌ π΅; π2 π1 =
πΌ π΅; π1 + πΌ π΅π1; π2 β πΌ π1; π2
β€ πΌ π΅; π1 + πΌ π΅π1; π2 = πΌ π΅; π1 + πΌ π΅; π2 + πΌ π1; π2 π΅
= πΌ π΅; π1 + πΌ π΅; π2 = 2 β πΌ π΅; π1 .
β’ Similarly, πΌ π΅; π1β¦ππ β€ π β πΌ π΅; π1 .
β’ To determine π΅ with constant accuracy, need 0 < c < πΌ π΅; π1β¦ππ β€ π β πΌ π΅; π1 .
β’ π = Ξ©(1/πΌ(π΅; π1)). 10
KullbackβLeibler (KL)-Divergence
β’ A distance metric between distributions on the same space.
β’ Plays a key role in information theory.
π·(π β₯ π) β π π₯ logπ[π₯]
π[π₯].
π₯
β’ π·(π β₯ π) β₯ 0, with equality when π = π.
β’ Caution: π·(π β₯ π) β π·(π β₯ π)!
11
Properties of KL-divergence
β’ Connection to mutual information: πΌ π; π = πΈπ¦βΌππ·(ππ=π¦ β₯ π).
β’ If π β₯ π, then ππ=π¦ = π, and both sides are 0.
β’ Pinskerβs inequality:
π β π 1 = π( π·(π β₯ π)).
β’ Tight!
π·(π΅1/2+π β₯ π΅1/2) = Ξ(π2).
12
Back to the coin example
β’ πΌ π΅; π1 = πΈπβΌπ΅π·(π1,π΅=π β₯ π1) = π· π΅1
2Β±π
β₯ π΅12
= Ξ π2 .
β’ π = Ξ©1
πΌ π΅;π1= Ξ©(
1
π2).
β’ βFollow the information learned from the coin tossesβ
β’ Can be done using combinatorics, but the information-theoretic language is more natural for expressing whatβs going on.
Back to communication
β’ The reason Information Theory is so important for communication is because information-theoretic quantities readily operationalize.
β’ Can attach operational meaning to Shannonβs entropy: π» π β βthe cost of transmitting πβ.
β’ Let πΆ π be the (expected) cost of transmitting a sample of π.
14
π» π = πΆ(π)?
β’ Not quite.
β’ Let trit π βπ 1,2,3 .
β’ πΆ π =5
3β 1.67.
β’ π» π = log 3 β 1.58.
β’ It is always the case that πΆ π β₯ π»(π).
15
1 0
2 10
3 11
But π» π and πΆ(π) are close
β’ Huffmanβs coding: πΆ π β€ π» π + 1.
β’ This is a compression result: βan uninformative message turned into a short oneβ.
β’ Therefore: π» π β€ πΆ π β€ π» π + 1.
16
Shannonβs noiseless coding β’ The cost of communicating many copies of π
scales as π»(π).
β’ Shannonβs source coding theorem:
β Let πΆ ππ be the cost of transmitting π independent copies of π. Then the amortized transmission cost
limπββ
πΆ(ππ)/π = π» π .
β’ This equation gives π»(π) operational meaning.
17
communication channel
π» π ππππππ‘πππππππ§ππ
18
π1, β¦ , ππ, β¦ π»(π) per copy to transmit πβs
π»(π) is nicer than πΆ(π)
β’ π» π is additive for independent variables.
β’ Let π1, π2 βπ {1,2,3} be independent trits.
β’ π» π1π2 = log 9 = 2 log 3.
β’ πΆ π1π2 =29
9< πΆ π1 + πΆ π2 = 2 Γ
5
3=
30
9.
β’ Works well with concepts such as channel capacity.
19
βProofβ of Shannonβs noiseless coding
β’ π β π» π = π» ππ β€ πΆ ππ β€ π» ππ + 1.
β’ Therefore limπββ
πΆ(ππ)/π = π» π .
20
Additivity of entropy
Compression (Huffman)
Operationalizing other quantities
β’ Conditional entropy π» π π :
β’ (cf. Slepian-Wolf Theorem).
communication channel
π1, β¦ , ππ, β¦ π»(π|π) per copy to transmit πβs
π1, β¦ , ππ, β¦
communication channel
Operationalizing other quantities
β’ Mutual information πΌ π; π :
π1, β¦ , ππ, β¦ πΌ π; π per copy to sample πβs
π1, β¦ , ππ, β¦
Information theory and entropy
β’ Allows us to formalize intuitive notions.
β’ Operationalized in the context of one-way transmission and related problems.
β’ Has nice properties (additivity, chain ruleβ¦)
β’ Next, we discuss extensions to more interesting communication scenarios.
23
Communication complexity
β’ Focus on the two party randomized setting.
24
A B
X Y A & B implement a functionality πΉ(π, π).
F(X,Y)
e.g. πΉ π, π = βπ = π? β
Shared randomness R
Communication complexity
A B
X Y
Goal: implement a functionality πΉ(π, π). A protocol π(π, π) computing πΉ(π, π):
F(X,Y)
m1(X,R)
m2(Y,m1,R)
m3(X,m1,m2,R)
Communication cost = #of bits exchanged.
Shared randomness R
Communication complexity
β’ Numerous applications/potential applications (some will be discussed later today).
β’ Considerably more difficult to obtain lower bounds than transmission (still much easier than other models of computation!).
26
Communication complexity
β’ (Distributional) communication complexity with input distribution π and error π: πΆπΆ πΉ, π, π . Error β€ π w.r.t. π.
β’ (Randomized/worst-case) communication complexity: πΆπΆ(πΉ, π). Error β€ π on all inputs.
β’ Yaoβs minimax:
πΆπΆ πΉ, π = maxπ
πΆπΆ(πΉ, π, π).
27
Examples
β’ π, π β 0,1 π.
β’ Equality πΈπ π, π β 1π=π.
β’ πΆπΆ πΈπ, π β log1
π.
β’ πΆπΆ πΈπ, 0 β π.
28
Equality β’πΉis βπ = π? β. β’πis a distribution where w.p. Β½π = π and w.p. Β½ (π, π) are random.
A B
X Y
β’ Shows that πΆπΆ πΈπ, π, 2β129 β€ 129.
MD5(X) [128 bits]
X=Y? [1 bit]
Error?
Examples
β’ π, π β 0,1 π.
β’ InnerproductπΌπ π, π β β ππ β πππ (πππ2).
β’ πΆπΆ πΌπ, 0 = π β π(π).
In fact, using information complexity:
β’ πΆπΆ πΌπ, π = π β ππ(π).
30
Information complexity
β’ Information complexity πΌπΆ(πΉ, π)::
communication complexity πΆπΆ πΉ, π
as
β’ Shannonβs entropy π»(π) ::
transmission cost πΆ(π)
31
Information complexity
β’ The smallest amount of information Alice and Bob need to exchange to solve πΉ.
β’ How is information measured?
β’ Communication cost of a protocol?
β Number of bits exchanged.
β’ Information cost of a protocol?
β Amount of information revealed.
32
Basic definition 1: The information cost of a protocol
β’ Prior distribution: π, π βΌ π.
A B
X Y
Protocol Ο Protocol
transcript Ξ
πΌπΆ(π, π) = πΌ(Ξ ; π|π) + πΌ(Ξ ; π|π)
what Alice learns about Y + what Bob learns about X
Example β’πΉis βπ = π? β. β’πis a distribution where w.p. Β½π = π and w.p. Β½ (π, π) are random.
A B
X Y
πΌπΆ(π, π) = πΌ(Ξ ; π|π) + πΌ(Ξ ; π|π) β1 + 64.5 = 65.5 bits
what Alice learns about Y + what Bob learns about X
MD5(X) [128 bits]
X=Y? [1 bit]
Prior π matters a lot for information cost!
β’ If π = 1 π₯,π¦ a singleton,
πΌπΆ π, π = 0.
35
Example β’πΉis βπ = π? β. β’πis a distribution where (π, π) are just uniformly random.
A B
X Y
πΌπΆ(π, π) = πΌ(Ξ ; π|π) + πΌ(Ξ ; π|π) β0 + 128 = 128 bits
what Alice learns about Y + what Bob learns about X
MD5(X) [128 bits]
X=Y? [1 bit]
Basic definition 2: Information complexity
β’ Communication complexity:
πΆπΆ πΉ, π, π β minππππππ’π‘ππ
πΉπ€ππ‘βπππππβ€π
π .
β’ Analogously:
πΌπΆ πΉ, π, π β infππππππ’π‘ππ
πΉπ€ππ‘βπππππβ€π
πΌπΆ(π, π).
37
Needed!
Prior-free information complexity
β’ Using minimax can get rid of the prior.
β’ For communication, we had:
πΆπΆ πΉ, π = maxπ
πΆπΆ(πΉ, π, π).
β’ For information
πΌπΆ πΉ, π β infππππππ’π‘ππ
πΉπ€ππ‘βπππππβ€π
maxπ
πΌπΆ(π, π).
38
Ex: The information complexity of Equality
β’ What is πΌπΆ(πΈπ, 0)?
β’ Consider the following protocol.
39 A B
X in {0,1}n Y in {0,1}n
A non-singular in ππΓπ
A1Β·X
A1Β·Y
A2Β·X
A2Β·Y
Continue for n steps, or until a disagreement is
discovered.
Analysis (sketch)
β’ If Xβ Y, the protocol will terminate in O(1) rounds on average, and thus reveal O(1) information.
β’ If X=Yβ¦ the players only learn the fact that X=Y (β€1 bit of information).
β’ Thus the protocol has O(1) information complexity for any prior π.
40
Operationalizing IC: Information equals amortized communication
β’ Recall [Shannon]: limπββ
πΆ(ππ)/π = π» π .
β’ Turns out: limπββ
πΆπΆ(πΉπ, ππ, π)/π = πΌπΆ πΉ, π, π ,
for π > 0. [Error π allowed on each copy]
β’ For π = 0: limπββ
πΆπΆ(πΉπ, ππ, 0+)/π = πΌπΆ πΉ, π, 0 .
β’ [ limπββ
πΆπΆ(πΉπ, ππ, 0)/π an interesting open
problem.] 41
Information = amortized communication
β’ limπββ
πΆπΆ(πΉπ, ππ, π)/π = πΌπΆ πΉ, π, π .
β’ Two directions: ββ€β and ββ₯β.
β’ π β π» π = π» ππ β€ πΆ ππ β€ π» ππ + 1.
Additivity of entropy
Compression (Huffman)
The ββ€β direction
β’ limπββ
πΆπΆ(πΉπ, ππ, π)/π β€ πΌπΆ πΉ, π, π .
β’ Start with a protocol π solving πΉ, whose πΌπΆ π, π is close to πΌπΆ πΉ, π, π .
β’ Show how to compress many copies of π into a protocol whose communication cost is close to its information cost.
β’ More on compression later.
43
The ββ₯β direction
β’ limπββ
πΆπΆ(πΉπ, ππ, π)/π β₯ πΌπΆ πΉ, π, π .
β’ Use the fact that πΆπΆ πΉπ,ππ,π
πβ₯
πΌπΆ πΉπ,ππ,π
π.
β’ Additivity of information complexity: πΌπΆ πΉπ, ππ, π
π= πΌπΆ πΉ, π, π .
44
Proof: Additivity of information complexity
β’ Let π1(π1, π1) and π2(π2, π2) be two two-party tasks.
β’ E.g. βSolve πΉ(π, π) with error β€ π w.r.t. πβ
β’ Then
πΌπΆ π1 Γ π2, π1 Γ π2 = πΌπΆ π1, π1 + πΌπΆ π2, π2
β’ ββ€β is easy.
β’ ββ₯β is the interesting direction.
πΌπΆ π1, π1 + πΌπΆ π2, π2 β€πΌπΆ π1 Γ π2, π1 Γ π2
β’ Start from a protocol π for π1 Γ π2 with prior π1 Γ π2, whose information cost is πΌ.
β’ Show how to construct two protocols π1 for π1 with prior π1 and π2 for π2 with prior π2, with information costs πΌ1 and πΌ2, respectively, such that πΌ1 + πΌ2 = πΌ.
46
π( π1, π2 , π1, π2 )
π1 π1, π1 β’ Publicly sample π2 βΌ π2 β’ Bob privately samples
π2 βΌ π2|X2
β’ Run π( π1, π2 , π1, π2 )
π2 π2, π2 β’ Publicly sample π1 βΌ π1 β’ Alice privately samples
π1 βΌ π1|Y1
β’ Run π( π1, π2 , π1, π2 )
π1 π1, π1 β’ Publicly sample π2 βΌ π2 β’ Bob privately samples
π2 βΌ π2|X2
β’ Run π( π1, π2 , π1, π2 )
Analysis - π1
β’ Alice learns about π1: πΌ Ξ ; π1 X1X2)
β’ Bob learns about π1: πΌ Ξ ; π1 π1π2π2).
β’ πΌ1 = πΌ Ξ ; π1 X1X2) + πΌ Ξ ; π1 π1π2π2).
48
π2 π2, π2 β’ Publicly sample π1 βΌ π1 β’ Alice privately samples
π1 βΌ π1|Y1
β’ Run π( π1, π2 , π1, π2 )
Analysis - π2
β’ Alice learns about π2: πΌ Ξ ; π2 X1X2Y1)
β’ Bob learns about π2: πΌ Ξ ; π2 π1π2).
β’ πΌ2 = πΌ Ξ ; π2 X1X2Y1) + πΌ Ξ ; π2 π1π2).
49
Adding πΌ1 and πΌ2
πΌ1 + πΌ2 = πΌ Ξ ; π1 X1X2) + πΌ Ξ ; π1 π1π2π2)
+ πΌ Ξ ; π2 X1X2Y1) + πΌ Ξ ; π2 π1π2)
= πΌ Ξ ; π1 X1X2) + πΌ Ξ ; π2 X1X2Y1) +πΌ Ξ ; π2 π1π2) + πΌ Ξ ; π1 π1π2π2) =
πΌ Ξ ; π1π2 π1π2 + πΌ Ξ ; π2π1 π1π2 = πΌ.
50
Summary
β’ Information complexity is additive.
β’ Operationalized via βInformation = amortized communicationβ.
β’ limπββ
πΆπΆ(πΉπ, ππ, π)/π = πΌπΆ πΉ, π, π .
β’ Seems to be the βrightβ analogue of entropy for interactive computation.
51
Entropy vs. Information Complexity
Entropy IC
Additive? Yes Yes
Operationalized limπββ
πΆ(ππ)/π limπββ
πΆπΆ πΉπ, ππ, π
π
Compression? Huffman:
πΆ π β€ π» π + 1 ???!
Can interactive communication be compressed?
β’ Is it true that πΆπΆ πΉ, π, π β€ πΌπΆ πΉ, π, π + π(1)?
β’ Less ambitiously:
πΆπΆ πΉ, π, π(π) = π πΌπΆ πΉ, π, π ?
β’ (Almost) equivalently: Given a protocol π with πΌπΆ π, π = πΌ, can Alice and Bob simulate π using π πΌ communication?
β’ Not known in generalβ¦
53
Direct sum theorems
β’ Let πΉ be any functionality.
β’ Let πΆ(πΉ) be the cost of implementing πΉ.
β’ Let πΉπ be the functionality of implementing π independent copies of πΉ.
β’ The direct sum problem:
βDoes πΆ(πΉπ) β π β πΆ(πΉ)?β
β’ In most cases it is obvious that πΆ(πΉπ) β€ π β πΆ(πΉ).
54
Direct sum β randomized communication complexity
β’ Is it true that
πΆπΆ(πΉπ, ππ, π) = Ξ©(π β πΆπΆ(πΉ, π, π))?
β’ Is it true that πΆπΆ(πΉπ, π) = Ξ©(π β πΆπΆ(πΉ, π))?
55
Direct product β randomized communication complexity
β’ Direct sum
πΆπΆ(πΉπ, ππ, π) = Ξ©(π β πΆπΆ(πΉ, π, π))?
β’ Direct product
πΆπΆ(πΉπ, ππ, (1 β π)π) = Ξ©(π β πΆπΆ(πΉ, π, π))?
56
Direct sum for randomized CC and interactive compression
Direct sum:
β’ πΆπΆ(πΉπ, ππ, π) = Ξ©(π β πΆπΆ(πΉ, π, π))?
In the limit:
β’ π β πΌπΆ(πΉ, π, π) = Ξ©(π β πΆπΆ(πΉ, π, π))?
Interactive compression:
β’ πΆπΆ πΉ, π, π = π(πΌπΆ πΉ, π, π) ?
Same question!
57
The big picture
πΆπΆ(πΉ, π, π) πΆπΆ(πΉπ, ππ, π)/π
πΌπΆ(πΉ, π, π) πΌπΆ(πΉπ, ππ, π)/π
additivity (=direct sum)
for information
information = amortized
communication direct sum for
communication?
interactive compression?
Current results for compression A protocol πthat has πΆ bits of communication, conveys πΌ bits of information over prior π, and works in π rounds can be simulated:
β’ Using π (πΌ + π) bits of communication.
β’ Using π πΌ β πΆ bits of communication.
β’ Using 2π(πΌ) bits of communication.
β’ If π = ππ Γ ππ, then using π(πΌ polylog πΆ) bits of communication. 59
Their direct sum counterparts
β’ πΆπΆ(πΉπ, ππ, π) = Ξ© (π1/2 β πΆπΆ(πΉ, π, π)).
β’ πΆπΆ(πΉπ, π) = Ξ© (π1/2 β πΆπΆ(πΉ, π)).
For product distributions π = ππ Γ ππ,
β’ πΆπΆ(πΉπ, ππ, π) = Ξ© (π β πΆπΆ(πΉ, π, π)).
When the number of rounds is bounded by π βͺ π, a direct sum theorem holds.
60
Direct product
β’ The best one can hope for is a statement of the type:
πΆπΆ(πΉπ, ππ, 1 β 2βπ π ) = Ξ©(π β πΌπΆ(πΉ, π, 1/3)).
β’ Can prove:
πΆπΆ πΉπ, ππ, 1 β 2βπ π = Ξ© (π1/2 β πΆπΆ(πΉ, π, 1/3)).
61
Proof 2: Compressing a one-round protocol
β’ Say Alice speaks: πΌπΆ π, π = πΌ π;π π .
β’ Recall KL-divergence: πΌ π; π π = πΈππ· πππ β₯ ππ = πΈππ· ππ β₯ ππ
β’ Bottom line:
β Alice has ππ; Bob has ππ;
β Goal: sample from ππ using βΌ π·(ππ β₯ ππ) communication.
62
The dart board
q1 q2 q3 q4 q5 q6 q7 β¦. u1 u2 u3 u4 u5 u6 u7
1
0
β’ Interpret the public randomness as random points in π Γ [0,1], where π is the universe of all possible messages.
β’ First message under the histogram of π is distributed βΌ π.
π
u1
u2
u3 u4
u5
64
Proof Idea β’ Sample using π(log1/π + π·(ππ β₯ ππ))
communication with statistical error Ξ΅.
MX MY
u1 u1
u2 u2
u3 u3
u4 u4
u4
~|U| samples Public randomness:
q1 q2 q3 q4 q5 q6 q7 β¦. u1 u2 u3 u4 u5 u6 u7
MX MY 1 1
0 0
Proof Idea β’ Sample using π(log1/π + π·(ππ β₯ ππ))
communication with statistical error Ξ΅.
u4 u2
h1(u4) h2(u4)
65 MX MY u4
1 1
0 0 u2
MX MY
66 66
Proof Idea β’ Sample using π(log1/π + π·(ππ β₯ ππ))
communication with statistical error Ξ΅.
u4 u2
h4(u4)β¦ hlog 1/ Ξ΅(u4)
u4
h3(u4)
MX 2MY
MX MY u4
u4
h1(u4), h2(u4)
1 1
0 0
67
Analysis
β’ If ππ(π’4) β 2πππ(π’4), then the protocol
will reach round π of doubling.
β’ There will be β 2π candidates.
β’ About π + log1/π hashes to narrow to one.
β’ The contribution of π’4 to cost:
β ππ(π’4)(logππ(π’4)/ππ(
π’4) + log1/π).
Done! π·(ππ β₯ ππ) β ππ(π’) logππ(π’)
ππ(π’).
π’
External information cost
β’ (π, π)~π.
A B
X Y
Protocol Ο Protocol
transcript Ο
πΌπΆππ₯π‘(π, π) = πΌ(Ξ ; ππ)
what Charlie learns about (X,Y)
C
Example
69
β’F is βX=Y?β. β’ΞΌ is a distribution where w.p. Β½ X=Y and w.p. Β½ (X,Y) are random.
MD5(X)
X=Y?
A B
X Y
πΌπΆππ₯π‘(π, π) = πΌ(Ξ ; ππ) = 129πππ‘π
what Charlie learns about (X,Y)
External information cost
β’ It is always the case that πΌπΆππ₯π‘ π, π β₯ πΌπΆ π, π .
β’ If π = ππ Γ ππ is a product distribution, then
πΌπΆππ₯π‘ π, π = πΌπΆ π, π .
70
External information complexity
β’ πΌπΆππ₯π‘ πΉ, π, π β infππππππ’π‘ππ
πΉπ€ππ‘βπππππβ€π
πΌπΆππ₯π‘(π, π).
β’ Can it be operationalized?
71
Operational meaning of πΌπΆππ₯π‘?
β’ Conjecture: Zero-error communication scales like external information:
limπββ
πΆπΆ πΉπ, ππ, 0
π= πΌπΆππ₯π‘ πΉ, π, 0 ?
β’ Recall:
limπββ
πΆπΆ πΉπ, ππ, 0+
π= πΌπΆ πΉ, π, 0 .
72
Example β transmission with a strong prior
β’ π, π β 0,1
β’ π is such that π βπ 0,1 , and π = π with a very high probability (say 1 β 1/ π).
β’ πΉ π, π = π is just the βtransmit πβ function.
β’ Clearly, π should just have Alice send π to Bob.
β’ πΌπΆ πΉ, π, 0 = πΌπΆ π, π = π»1
π= o(1).
β’ πΌπΆππ₯π‘ πΉ, π, 0 = πΌπΆππ₯π‘ π, π = 1. 73
Example β transmission with a strong prior
β’ πΌπΆ πΉ, π, 0 = πΌπΆ π, π = π»1
π= o(1).
β’ πΌπΆππ₯π‘ πΉ, π, 0 = πΌπΆππ₯π‘ π, π = 1.
β’ πΆπΆ πΉπ, ππ, 0+ = π π .
β’ πΆπΆ πΉπ, ππ, 0 = Ξ©(π).
Other examples, e.g. the two-bit AND function fit into this picture.
74
Additional directions
75
Information complexity
Interactive coding
Information theory in TCS
Interactive coding theory
β’ So far focused the discussion on noiseless coding.
β’ What if the channel has noise?
β’ [What kind of noise?]
β’ In the non-interactive case, each channel has a capacity πΆ.
76
Channel capacity
β’ The amortized number of channel uses needed to send π over a noisy channel of capacity πΆ is
π» π
πΆ
β’ Decouples the task from the channel!
77
Example: Binary Symmetric Channel
β’ Each bit gets independently flipped with probability π < 1/2.
β’ One way capacity 1 β π» π .
78
0
1
0
1 1 β π
1 β π
π π
Interactive channel capacity
β’ Not clear one can decouple channel from task in such a clean way.
β’ Capacity much harder to calculate/reason about.
β’ Example: Binary symmetric channel.
β’ One way capacity 1 β π» π .
β’ Interactive (for simple pointer jumping,
[Kol-Razβ13]):
1 β Ξ π» π .
0
1
0
1 1 β π
1 β π
π
Information theory in communication complexity and beyond
β’ A natural extension would be to multi-party communication complexity.
β’ Some success in the number-in-hand case.
β’ What about the number-on-forehead?
β’ Explicit bounds for β₯ log π players would imply explicit π΄πΆπΆ0 circuit lower bounds.
80
NaΓ―ve multi-party information cost
πΌπΆ π, π = πΌ(Ξ ; π|ππ)+ πΌ(Ξ ; π|ππ)+ πΌ(Ξ ; π|ππ)
81
A B C
YZ XZ XY
NaΓ―ve multi-party information cost
πΌπΆ π, π = πΌ(Ξ ; π|ππ)+ πΌ(Ξ ; π|ππ)+ πΌ(Ξ ; π|ππ)
β’ Doesnβt seem to work.
β’ Secure multi-party computation [Ben-Or,Goldwasser, Wigderson], means that anything can be computed at near-zero information cost.
β’ Although, these construction require the players to share private channels/randomness.
82
Communication and beyondβ¦
β’ The rest of today:
β Data structures;
β Streaming;
β Distributed computing;
β Privacy.
β’ Exact communication complexity bounds.
β’ Extended formulations lower bounds.
β’ Parallel repetition?
β’ β¦ 83
84
Thank You!
Open problem: Computability of IC
β’ Given the truth table of πΉ π, π , π and π, compute πΌπΆ πΉ, π, π .
β’ Via πΌπΆ πΉ, π, π = limπββ
πΆπΆ(πΉπ, ππ, π)/π can
compute a sequence of upper bounds.
β’ But the rate of convergence as a function of π is unknown.
85
Open problem: Computability of IC
β’ Can compute the π-roundπΌπΆπ πΉ, π, π information complexity of πΉ.
β’ But the rate of convergence as a function of π is unknown.
β’ Conjecture:
πΌπΆπ πΉ, π, π β πΌπΆ πΉ, π, π = ππΉ,π,π1
π2.
β’ This is the relationship for the two-bit AND.
86
