“with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · annotation in...
TRANSCRIPT
![Page 1: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/1.jpg)
Annotation in Data Streams“with a little help from your friends”
! Amit Chakrabarti ! Dartmouth College! Graham Cormode ! AT&T Research Labs! Andrew McGregor ! University of Massachusetts, Amherst
![Page 2: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/2.jpg)
Data Stream Model[Morris ’78] [Munro, Paterson ’78] [Flajolet, Martin ’85]
[Alon, Matias, Szegedy ’96] [Henzinger, Raghavan, Rajagopalan ’98]
![Page 3: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/3.jpg)
Data Stream Model[Morris ’78] [Munro, Paterson ’78] [Flajolet, Martin ’85]
[Alon, Matias, Szegedy ’96] [Henzinger, Raghavan, Rajagopalan ’98]
• Stream: m elements from universe of size n
e.g., [x1, x2, ... , xm] = 3,5,3,7,5,4,8,7,5,4,8,6,3,2,6,4,7, ...
![Page 4: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/4.jpg)
Data Stream Model[Morris ’78] [Munro, Paterson ’78] [Flajolet, Martin ’85]
[Alon, Matias, Szegedy ’96] [Henzinger, Raghavan, Rajagopalan ’98]
• Stream: m elements from universe of size n
e.g., [x1, x2, ... , xm] = 3,5,3,7,5,4,8,7,5,4,8,6,3,2,6,4,7, ...
• Goal: Compute a function of stream, e.g., median, number of distinct elements, longest increasing sequence.
![Page 5: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/5.jpg)
Data Stream Model[Morris ’78] [Munro, Paterson ’78] [Flajolet, Martin ’85]
[Alon, Matias, Szegedy ’96] [Henzinger, Raghavan, Rajagopalan ’98]
• Stream: m elements from universe of size n
e.g., [x1, x2, ... , xm] = 3,5,3,7,5,4,8,7,5,4,8,6,3,2,6,4,7, ...
• Goal: Compute a function of stream, e.g., median, number of distinct elements, longest increasing sequence.
• The Catch:
• i) Limited working memory, i.e., sublinear(n,m)
• ii) Access data sequentially
• iii) Process each element quickly
![Page 6: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/6.jpg)
Data Stream Model[Morris ’78] [Munro, Paterson ’78] [Flajolet, Martin ’85]
[Alon, Matias, Szegedy ’96] [Henzinger, Raghavan, Rajagopalan ’98]
• Stream: m elements from universe of size n
e.g., [x1, x2, ... , xm] = 3,5,3,7,5,4,8,7,5,4,8,6,3,2,6,4,7, ...
• Goal: Compute a function of stream, e.g., median, number of distinct elements, longest increasing sequence.
• The Catch:
• i) Limited working memory, i.e., sublinear(n,m)
• ii) Access data sequentially
• iii) Process each element quickly
• Origins in ’70s but has become popular in last ten years because of growing theory and very applicable.
![Page 7: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/7.jpg)
Outsourcing Stream Processing
![Page 8: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/8.jpg)
Outsourcing Stream Processing
• Many problems require linear space :(
![Page 9: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/9.jpg)
Outsourcing Stream Processing
• Many problems require linear space :(
• Off-load computation to more powerful “helper”:
E.g., special hardware, multiple processing units, contractor who provides a commercial service.
![Page 10: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/10.jpg)
Outsourcing Stream Processing
• Many problems require linear space :(
• Off-load computation to more powerful “helper”:
E.g., special hardware, multiple processing units, contractor who provides a commercial service.
• Previous work in database community:
E.g., stream punctuation [Tucker et al. 05], proof infused streams [Li et al. 07], stream outsourcing [Yi et al. 08].
![Page 11: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/11.jpg)
Outsourcing Stream Processing
• Many problems require linear space :(
• Off-load computation to more powerful “helper”:
E.g., special hardware, multiple processing units, contractor who provides a commercial service.
• Previous work in database community:
E.g., stream punctuation [Tucker et al. 05], proof infused streams [Li et al. 07], stream outsourcing [Yi et al. 08].
• Today’s talk: What should model be and how powerful is it?
![Page 12: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/12.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
![Page 13: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/13.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
12
![Page 14: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/14.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
3
![Page 15: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/15.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
18
![Page 16: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/16.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
103
![Page 17: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/17.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
37
![Page 18: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/18.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
56
![Page 19: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/19.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
43
![Page 20: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/20.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
59
![Page 21: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/21.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
9
![Page 22: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/22.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
89
![Page 23: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/23.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
99
![Page 24: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/24.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
41
![Page 25: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/25.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
51
![Page 26: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/26.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
56
![Page 27: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/27.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
26
![Page 28: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/28.jpg)
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
* Numerical values not to actual scale.
26
![Page 29: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/29.jpg)
43
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
* Numerical values not to actual scale.
![Page 30: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/30.jpg)
43
Model Version 1: “Just trust the helper”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
* Numerical values not to actual scale.
• Don’t want to have to trust the third party.
![Page 31: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/31.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
![Page 32: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/32.jpg)
43
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
![Page 33: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/33.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
12
![Page 34: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/34.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
3
![Page 35: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/35.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
18
![Page 36: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/36.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
103
![Page 37: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/37.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
37
![Page 38: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/38.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
56
![Page 39: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/39.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
43
![Page 40: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/40.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
59
![Page 41: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/41.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
9
![Page 42: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/42.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
89
![Page 43: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/43.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
99
![Page 44: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/44.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
41
![Page 45: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/45.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
51
![Page 46: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/46.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
56
![Page 47: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/47.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
26
![Page 48: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/48.jpg)
Model Version 2: “Helper is prescient”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper first announces the answer: verification is easy.
• Can’t expect the helper to know the future.
26
![Page 49: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/49.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 50: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/50.jpg)
12
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 51: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/51.jpg)
3
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 52: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/52.jpg)
18
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 53: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/53.jpg)
103
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 54: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/54.jpg)
37
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 55: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/55.jpg)
56
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 56: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/56.jpg)
43
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 57: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/57.jpg)
59
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 58: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/58.jpg)
9
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 59: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/59.jpg)
89
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 60: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/60.jpg)
99
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 61: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/61.jpg)
41
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 62: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/62.jpg)
51
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 63: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/63.jpg)
56
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 64: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/64.jpg)
26
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 65: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/65.jpg)
26
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 66: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/66.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
3
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 67: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/67.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
9
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 68: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/68.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
12
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 69: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/69.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
18
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 70: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/70.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
26
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 71: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/71.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
37
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 72: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/72.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
41
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 73: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/73.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
43
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 74: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/74.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
51
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 75: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/75.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
56
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 76: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/76.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
56
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 77: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/77.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
59
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 78: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/78.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
89
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 79: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/79.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
99
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 80: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/80.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
103
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 81: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/81.jpg)
Model Version 3: “Helper is loquacious”
• Example Problem: Find median of m numbers from [m]. This requires Ω(m) space in the data stream model.
• Helper repeats stream in sorted order: Easy to find median
• Fingerprint to check annotation B is rearranged stream A:
• For prime q≥3m and r ∈R [q]: check FPA(r)=FPB(r) where
103FPS(x) =
�
i∈S
(x − i) mod q
cf. “Best-Order Streaming” [Das Sarma, Lipton, Nanongkai 09]
![Page 82: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/82.jpg)
Final(ish) Model
![Page 83: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/83.jpg)
• Problem: Given stream S, want to compute f(S):
• S=[x1, x2, x3, x4, x5, x6, x7, x8, x9, ... , xm]
Final(ish) Model
![Page 84: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/84.jpg)
• Problem: Given stream S, want to compute f(S):
• S=[x1, x2, x3, x4, x5, x6, x7, x8, x9, ... , xm]
• Helper: augments stream with “good” h-bit annotation:
• (S,a)=[x1, x2, x3, a3, x4, x5, x6, x7, a7, ... , xm, am]
Final(ish) Model
![Page 85: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/85.jpg)
• Problem: Given stream S, want to compute f(S):
• S=[x1, x2, x3, x4, x5, x6, x7, x8, x9, ... , xm]
• Helper: augments stream with “good” h-bit annotation:
• (S,a)=[x1, x2, x3, a3, x4, x5, x6, x7, a7, ... , xm, am]
Final(ish) Model
annotation is a function of previous elements
![Page 86: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/86.jpg)
• Problem: Given stream S, want to compute f(S):
• S=[x1, x2, x3, x4, x5, x6, x7, x8, x9, ... , xm]
• Helper: augments stream with “good” h-bit annotation:
• (S,a)=[x1, x2, x3, a3, x4, x5, x6, x7, a7, ... , xm, am]
• Verifier: using v bits of space and random string r, run verification algorithm to compute g(S,a,r) such that:
• a) Prr[g(S,a,r)=f(S)]≥1-δ
• b) Prr[g(S,a’,r)=⊥]≥1-δ for a’≠a
Final(ish) Model
annotation is a function of previous elements
![Page 87: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/87.jpg)
• Problem: Given stream S, want to compute f(S):
• S=[x1, x2, x3, x4, x5, x6, x7, x8, x9, ... , xm]
• Helper: augments stream with “good” h-bit annotation:
• (S,a)=[x1, x2, x3, a3, x4, x5, x6, x7, a7, ... , xm, am]
• Verifier: using v bits of space and random string r, run verification algorithm to compute g(S,a,r) such that:
• a) Prr[g(S,a,r)=f(S)]≥1-δ
• b) Prr[g(S,a’,r)=⊥]≥1-δ for a’≠a
• Goal: Minimize h and v for given error δ.
Final(ish) Model
annotation is a function of previous elements
![Page 88: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/88.jpg)
Better Median Protocol
• Problem: Find median of m numbers from [m].
• Thm: O(√m log m) annotation bits and O(√m log m) verification memory is sufficient to find the median.
• Thm: Any protocol for median requires
• (annotation “h”) x (verification memory “v”) = Ω(m).
![Page 89: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/89.jpg)
Upper Bound...
![Page 90: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/90.jpg)
Upper Bound...
• Define “cumulative frequency” vector: gi = |{j : xj ≤ i}|
![Page 91: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/91.jpg)
Upper Bound...
• Define “cumulative frequency” vector: gi = |{j : xj ≤ i}|
1 1 3 5 6 7 8 8 8 8 10 10 11 12 12 15 18 18 19 20 22 22 23 25 25
![Page 92: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/92.jpg)
Upper Bound...
• Define “cumulative frequency” vector:
• Easy to see i is median iff gi-1<m/2 and gi≥m/2
gi = |{j : xj ≤ i}|
1 1 3 5 6 7 8 8 8 8 10 10 11 12 12 15 18 18 19 20 22 22 23 25 25
![Page 93: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/93.jpg)
Upper Bound...
• Define “cumulative frequency” vector:
• Easy to see i is median iff gi-1<m/2 and gi≥m/2
gi = |{j : xj ≤ i}|
1 1 3 5 6 7 8 8 8 8 10 10 11 12 12 15 18 18 19 20 22 22 23 25 25
16th position
![Page 94: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/94.jpg)
Upper Bound...
• Define “cumulative frequency” vector:
• Easy to see i is median iff gi-1<m/2 and gi≥m/2
• Partition g into v = m1/2 segments of length h = m1/2
gi = |{j : xj ≤ i}|
1 1 3 5 6 7 8 8 8 8 10 10 11 12 12 15 18 18 19 20 22 22 23 25 25
![Page 95: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/95.jpg)
Upper Bound...
• Define “cumulative frequency” vector:
• Easy to see i is median iff gi-1<m/2 and gi≥m/2
• Partition g into v = m1/2 segments of length h = m1/2
• Verifier: ! a) Construct fingerprint of each segment
gi = |{j : xj ≤ i}|
1 1 3 5 6 7 8 8 8 8 10 10 11 12 12 15 18 18 19 20 22 22 23 25 25
![Page 96: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/96.jpg)
Upper Bound...
• Define “cumulative frequency” vector:
• Easy to see i is median iff gi-1<m/2 and gi≥m/2
• Partition g into v = m1/2 segments of length h = m1/2
• Verifier: ! a) Construct fingerprint of each segment
• ! b) Compute last entry in each segment
gi = |{j : xj ≤ i}|
1 1 3 5 6 7 8 8 8 8 10 10 11 12 12 15 18 18 19 20 22 22 23 25 25
![Page 97: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/97.jpg)
Upper Bound...
• Define “cumulative frequency” vector:
• Easy to see i is median iff gi-1<m/2 and gi≥m/2
• Partition g into v = m1/2 segments of length h = m1/2
• Verifier: ! a) Construct fingerprint of each segment
• ! b) Compute last entry in each segment
• ! c) Identify “interesting” segment
gi = |{j : xj ≤ i}|
1 1 3 5 6 7 8 8 8 8 10 10 11 12 12 15 18 18 19 20 22 22 23 25 25
![Page 98: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/98.jpg)
Upper Bound...
• Define “cumulative frequency” vector:
• Easy to see i is median iff gi-1<m/2 and gi≥m/2
• Partition g into v = m1/2 segments of length h = m1/2
• Verifier: ! a) Construct fingerprint of each segment
• ! b) Compute last entry in each segment
• ! c) Identify “interesting” segment
• Helper: ! Presents entirety of interesting segment
gi = |{j : xj ≤ i}|
1 1 3 5 6 7 8 8 8 8 10 10 11 12 12 15 18 18 19 20 22 22 23 25 25
![Page 99: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/99.jpg)
Better Median Protocol
• Problem: Find median of m numbers from [m].
• Thm: O(√m log m) annotation bits and O(√m log m) verification memory is sufficient to find the median.
![Page 100: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/100.jpg)
Better Median Protocol
• Problem: Find median of m numbers from [m].
• Thm: O(√m log m) annotation bits and O(√m log m) verification memory is sufficient to find the median.
• Thm: Any protocol for median requires
• (annotation “h”) x (verification memory “v”) = Ω(m).
![Page 101: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/101.jpg)
Lower Bound ...
![Page 102: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/102.jpg)
• Suppose algorithm “A” has parameters (h,v), error δ = 1/3
Lower Bound ...
![Page 103: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/103.jpg)
• Suppose algorithm “A” has parameters (h,v), error δ = 1/3
• Define “B”:
• Algorithm “B” has parameters (h,hv), error δ = (1/3)2-h
• If a valid, then expect (2/3)t runs to return median
• If a not valid, then expect (2/3)t runs give ⊥
Lower Bound ...
ALGORITHM “B”
1. Run t = Θ(h) copies of “A” in parallel
2. Use annotation a for all copies
3. Output majority answer if it exists, else ⊥
![Page 104: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/104.jpg)
• Suppose algorithm “A” has parameters (h,v), error δ = 1/3
• Define “B”:
• Algorithm “B” has parameters (h,hv), error δ = (1/3)2-h
• If a valid, then expect (2/3)t runs to return median
• If a not valid, then expect (2/3)t runs give ⊥
• Define “C”: Verifier ignores annotation, tries all 2h possible annotations and ensures error at most 1/3 (by union bound).
Lower Bound ...
ALGORITHM “B”
1. Run t = Θ(h) copies of “A” in parallel
2. Use annotation a for all copies
3. Output majority answer if it exists, else ⊥
![Page 105: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/105.jpg)
• Suppose algorithm “A” has parameters (h,v), error δ = 1/3
• Define “B”:
• Algorithm “B” has parameters (h,hv), error δ = (1/3)2-h
• If a valid, then expect (2/3)t runs to return median
• If a not valid, then expect (2/3)t runs give ⊥
• Define “C”: Verifier ignores annotation, tries all 2h possible annotations and ensures error at most 1/3 (by union bound).
• Algorithm “C” solves median of stream using O(hv) space
Lower Bound ...
ALGORITHM “B”
1. Run t = Θ(h) copies of “A” in parallel
2. Use annotation a for all copies
3. Output majority answer if it exists, else ⊥
![Page 106: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/106.jpg)
Our Results
• Median: Optimal trade-off of annotation & verification.
• Frequency Moments: Optimal trade-off for exact version and lower bounds for approximate version.
• Heavy Hitters: Optimal trade-off for exact and protocol for approximate version via CM-sketch verification.
• Graph Problems: Trade-offs for counting triangles, matchings, and connectivity. Optimal in some regimes.
![Page 107: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/107.jpg)
1.Frequency Moments
2.Counting Triangles
3.Beyond the Moraines...
1.Frequency Moments
2.Counting Triangles
3.Beyond the Moraines...
![Page 108: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/108.jpg)
1.Frequency Moments
2.Counting Triangles
3.Beyond the Moraines...
![Page 109: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/109.jpg)
Frequency Moments• Problem: Given m numbers from [n], compute
Fk =�
i∈[n]
f ki where fi = freq. of i
![Page 110: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/110.jpg)
Frequency Moments• Problem: Given m numbers from [n], compute
Fk =�
i∈[n]
f ki where fi = freq. of i
![Page 111: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/111.jpg)
Frequency Moments• Problem: Given m numbers from [n], compute
• Thm: O(k2√n log m) annotation bits and O(k√n log m) verification memory is sufficient to compute Fk.
• using “algebrization” ideas from [Aaronson, Widgerson 08]
Fk =�
i∈[n]
f ki where fi = freq. of i
![Page 112: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/112.jpg)
Frequency Moments• Problem: Given m numbers from [n], compute
• Thm: O(k2√n log m) annotation bits and O(k√n log m) verification memory is sufficient to compute Fk.
• using “algebrization” ideas from [Aaronson, Widgerson 08]
• Thm: Any protocol for Fk requires
• (annotation) x (verification memory) = Ω(n)
• and any constant factor approx. requires
• (annotation) x (verification memory) = Ω(n1-5/k).
• using ideas from [Klauck 03], [Alon, Mattias, Szegedy 99]
Fk =�
i∈[n]
f ki where fi = freq. of i
![Page 113: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/113.jpg)
Upper Bound (1/2)...
![Page 114: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/114.jpg)
Upper Bound (1/2)...
• Transform universe [n] into [√n] x [√n] : �
i ,j∈[√
n] fki ,j
![Page 115: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/115.jpg)
Upper Bound (1/2)...
• Transform universe [n] into [√n] x [√n] :
1 0 3 2 1 10 9 8 3
1 0 3
2 1 10
9 8 3“frequency vector”“frequency square”
�i ,j∈[
√n] f
ki ,j
![Page 116: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/116.jpg)
Upper Bound (1/2)...
• Transform universe [n] into [√n] x [√n] :
• Define where q large prime:f (x , y) ∈ Fq[x , y ]
1 0 3 2 1 10 9 8 3
1 0 3
2 1 10
9 8 3“frequency vector”“frequency square”
f (x , y) =�
(i ,j)∈S
pi ,j(x , y) where pi ,j(x , y) =
√n�
�=1:� �=i
x − �
i − �·
√n�
�=1:� �=j
y − �
j − �
�i ,j∈[
√n] f
ki ,j
![Page 117: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/117.jpg)
Upper Bound (1/2)...
• Transform universe [n] into [√n] x [√n] :
• Define where q large prime:f (x , y) ∈ Fq[x , y ]
f (i , j) = frequency of (i , j) in stream
1 0 3 2 1 10 9 8 3
1 0 3
2 1 10
9 8 3“frequency vector”“frequency square”
f (x , y) =�
(i ,j)∈S
pi ,j(x , y) where pi ,j(x , y) =
√n�
�=1:� �=i
x − �
i − �·
√n�
�=1:� �=j
y − �
j − �
�i ,j∈[
√n] f
ki ,j
![Page 118: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/118.jpg)
Upper Bound (1/2)...
• Transform universe [n] into [√n] x [√n] :
• Define where q large prime:
• Observe: f defined incrementally and:
f (x , y) ∈ Fq[x , y ]
degx(f ) = degy (f ) =√
n − 1
f (i , j) = frequency of (i , j) in stream
1 0 3 2 1 10 9 8 3
1 0 3
2 1 10
9 8 3“frequency vector”“frequency square”
f (x , y) =�
(i ,j)∈S
pi ,j(x , y) where pi ,j(x , y) =
√n�
�=1:� �=i
x − �
i − �·
√n�
�=1:� �=j
y − �
j − �
�i ,j∈[
√n] f
ki ,j
![Page 119: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/119.jpg)
Upper Bound (2/2)...
![Page 120: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/120.jpg)
Upper Bound (2/2)...
• Verifier: pick r ∈R [q]; compute f(r,y); and determine
C =�
j∈[√
n]
f (r , j)k
![Page 121: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/121.jpg)
Upper Bound (2/2)...
• Verifier: pick r ∈R [q]; compute f(r,y); and determine
• Helper: annotates O(√n log q) bits with
C =�
j∈[√
n]
f (r , j)k
s(x) =�
j∈[√
n]
f (x , j)k
![Page 122: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/122.jpg)
Upper Bound (2/2)...
• Verifier: pick r ∈R [q]; compute f(r,y); and determine
• Helper: annotates O(√n log q) bits with
• Verifier: checks C=s(r) and outputs:
C =�
j∈[√
n]
f (r , j)k
�
i∈[√
n]
s(i) = Fk
s(x) =�
j∈[√
n]
f (x , j)k
![Page 123: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/123.jpg)
Lower Bound...
![Page 124: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/124.jpg)
Lower Bound...
• Thm: Any protocol for Fk requires
• (annotation) x (verification memory) = Ω(n).
• Idea: Use MA lower bound for 2-party set-disjointness.
![Page 125: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/125.jpg)
Lower Bound...
• Thm: Any protocol for Fk requires
• (annotation) x (verification memory) = Ω(n).
• Idea: Use MA lower bound for 2-party set-disjointness.
• Thm: Any constant factor approx. for Fk requires
• (annotation) x (verification memory) = Ω(n1-5/k).
• Idea: New MA lower bound for t-party set-disjointness.
![Page 126: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/126.jpg)
1.Frequency Moments
2.Counting Triangles
3.Beyond the Moraines...
![Page 127: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/127.jpg)
Counting Triangles
![Page 128: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/128.jpg)
Counting Triangles• Problem: Given m edges on n nodes, find # triangles.
![Page 129: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/129.jpg)
Counting Triangles• Problem: Given m edges on n nodes, find # triangles.
• Thm: O(n3α log n) annotation bits and O(n3-3α log n) verification memory suffices for 0<α<1.
• using idea from [Bar-Yossef, Kumar, Sivakumar 02]
![Page 130: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/130.jpg)
Counting Triangles• Problem: Given m edges on n nodes, find # triangles.
• Thm: O(n3α log n) annotation bits and O(n3-3α log n) verification memory suffices for 0<α<1.
• using idea from [Bar-Yossef, Kumar, Sivakumar 02]
Construct induced stream S by replacing each (i,j) by (i,j,k) for each k≠i,j. Compute (F3(S)-2F2(S)+F1(S))/12.
![Page 131: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/131.jpg)
Counting Triangles• Problem: Given m edges on n nodes, find # triangles.
• Thm: O(n3α log n) annotation bits and O(n3-3α log n) verification memory suffices for 0<α<1.
• using idea from [Bar-Yossef, Kumar, Sivakumar 02]
Construct induced stream S by replacing each (i,j) by (i,j,k) for each k≠i,j. Compute (F3(S)-2F2(S)+F1(S))/12.
• Thm: O(n2 log n) annotation bits and O(log n) verification memory suffices.
![Page 132: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/132.jpg)
Counting Triangles• Problem: Given m edges on n nodes, find # triangles.
• Thm: O(n3α log n) annotation bits and O(n3-3α log n) verification memory suffices for 0<α<1.
• using idea from [Bar-Yossef, Kumar, Sivakumar 02]
Construct induced stream S by replacing each (i,j) by (i,j,k) for each k≠i,j. Compute (F3(S)-2F2(S)+F1(S))/12.
• Thm: O(n2 log n) annotation bits and O(log n) verification memory suffices.
• A is adjacency matrix. Prover give (A[u,v], A2[u,v]) for each u,v. Verify A by finger-printing and verify A2 by picking random s,r and checking (sA)(ArT)= sA2rT.
![Page 133: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/133.jpg)
Counting Triangles• Problem: Given m edges on n nodes, find # triangles.
• Thm: O(n3α log n) annotation bits and O(n3-3α log n) verification memory suffices for 0<α<1.
• using idea from [Bar-Yossef, Kumar, Sivakumar 02]
Construct induced stream S by replacing each (i,j) by (i,j,k) for each k≠i,j. Compute (F3(S)-2F2(S)+F1(S))/12.
• Thm: O(n2 log n) annotation bits and O(log n) verification memory suffices.
• A is adjacency matrix. Prover give (A[u,v], A2[u,v]) for each u,v. Verify A by finger-printing and verify A2 by picking random s,r and checking (sA)(ArT)= sA2rT.
• Thm: Triangles requires (annotation) x (verification)=Ω(n2).
![Page 134: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/134.jpg)
1.Frequency Moments
2.Counting Triangles
3.Beyond the Moraines...
![Page 135: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/135.jpg)
Interactive Proofs & StreamsForthcoming work from [Cormode, Yi ’09]...
![Page 136: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/136.jpg)
• Motivated by applications clouding computing:
1. Consider messages back and forth between prover and verifier after the stream has been observed.
2. Measure in terms of memory used by verifier and subsequent communication.
Interactive Proofs & StreamsForthcoming work from [Cormode, Yi ’09]...
![Page 137: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/137.jpg)
• Motivated by applications clouding computing:
1. Consider messages back and forth between prover and verifier after the stream has been observed.
2. Measure in terms of memory used by verifier and subsequent communication.
• Results:
1. Using log memory and log communication, many database problems can be solved exactly: self join size, frequency moments, range queries, index etc.
Interactive Proofs & StreamsForthcoming work from [Cormode, Yi ’09]...
![Page 138: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/138.jpg)
Sketch Verification
![Page 139: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/139.jpg)
Sketch Verification• Sketch: Let A be k by n measurement matrix and
compute Af: for stream [1, 2, 3, 2, 4,...],
• Af = a1+a2+a3+a2+a4+... where ai is ith column of A
![Page 140: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/140.jpg)
Sketch Verification• Sketch: Let A be k by n measurement matrix and
compute Af: for stream [1, 2, 3, 2, 4,...],
• Af = a1+a2+a3+a2+a4+... where ai is ith column of A
• Annotation Protocol: Helper gives Af in O(k log n) bits and verifier checks in O(log k) bits by composing fingerprint with A. Let B be k’ by k and compute
• BAf = Ba1+Ba2+Ba3+Ba2+Ba4 + ...
![Page 141: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/141.jpg)
Sketch Verification• Sketch: Let A be k by n measurement matrix and
compute Af: for stream [1, 2, 3, 2, 4,...],
• Af = a1+a2+a3+a2+a4+... where ai is ith column of A
• Annotation Protocol: Helper gives Af in O(k log n) bits and verifier checks in O(log k) bits by composing fingerprint with A. Let B be k’ by k and compute
• BAf = Ba1+Ba2+Ba3+Ba2+Ba4 + ...
• Tricky Part: Having helper guide verifier through the steps of extracting information from sketch.
![Page 142: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/142.jpg)
Sketch Verification• Sketch: Let A be k by n measurement matrix and
compute Af: for stream [1, 2, 3, 2, 4,...],
• Af = a1+a2+a3+a2+a4+... where ai is ith column of A
• Annotation Protocol: Helper gives Af in O(k log n) bits and verifier checks in O(log k) bits by composing fingerprint with A. Let B be k’ by k and compute
• BAf = Ba1+Ba2+Ba3+Ba2+Ba4 + ...
• Tricky Part: Having helper guide verifier through the steps of extracting information from sketch.
• Mysterious Issue: Most useful sketches are random but can’t trust helper to pick random bits.
![Page 143: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/143.jpg)
Thanks!
Summary
Model: Merlin-Arthur communication meets data streams and intractable stream problems become solvable!
Results: Annotation/verification trade-offs for classical stream problems. Some optimal and some open questions.
![Page 144: “with a little help from your friends”mcgregor/slides/09... · 2010-06-10 · Annotation in Data Streams “with a little help from your friends”! Amit Chakrabarti! Dartmouth](https://reader033.vdocument.in/reader033/viewer/2022041513/5e296a94b06707546d242816/html5/thumbnails/144.jpg)