dremel: interactive analysis of web-scale datasetsistoica/classes/... · dremel: interactive...
TRANSCRIPT
![Page 1: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/1.jpg)
Dremel: Interactive Analysis of Web-Scale Datasets
Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shivakumar, Matt Tolton, Theo Vassilakis
Presented by: Sameer Agarwal
![Page 2: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/2.jpg)
Dremel: Interactive Analysis of Web-Scale Datasets
![Page 3: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/3.jpg)
Interactive Queries on Large Data
• Input/Output
‐ Sequentially reading a Terabyte from disk in a second requires ~20,000 parallel reads!
• Processing
‐ CPU-intensive queries may need to run on thousands of cores to complete within a second.
‐ Dealing with failures and stragglers is essential.
![Page 4: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/4.jpg)
Interactive Queries on Large Data
• Input/Output
‐ Sequentially reading a Terabyte from disk in a second requires ~20,000 parallel reads! [Nested Columnar Storage]
• Processing
‐ CPU-intensive queries may need to run on thousands of cores to complete within a second.
‐ Dealing with failures and stragglers is essential.
![Page 5: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/5.jpg)
Interactive Queries on Large Data
• Input/Output
‐ Sequentially reading a Terabyte from disk in a second requires ~20,000 parallel reads! [Nested Columnar Storage]
• Processing
‐ CPU-intensive queries may need to run on thousands of cores to complete within a second. [Hierarchical Query Processing]
‐ Dealing with failures and stragglers is essential.
![Page 6: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/6.jpg)
Interactive Queries on Large Data
• Input/Output
‐ Sequentially reading a Terabyte from disk in a second requires ~20,000 parallel reads! [Nested Columnar Storage]
• Processing
‐ CPU-intensive queries may need to run on thousands of cores to complete within a second. [Hierarchical Query Processing]
‐ Dealing with failures and stragglers is essential. [Profiles, Duplicates or Ignores Them]
![Page 7: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/7.jpg)
Nested Columnar Storage
DocId: 10
Links
Forward: 20
Name
Language
Code: 'en-us'
Country: 'us'
Url: 'http://A'
Name
Url: 'http://B'
r1
![Page 8: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/8.jpg)
Nested Columnar Storage
A
B
C D
E *
*
*
. . .
. . .
r1
r2 r1
r2
r1
r2
r1
r2
Read Less; Cheaper Decompression!
![Page 9: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/9.jpg)
Nested Columnar Storage
message Document {
required int64 DocId;
optional group Links {
repeated int64 Backward; repeated int64 Forward;
}
repeated group Name {
repeated group Language {
required string Code;
optional string Country;
}
optional string Url;
}
}
DocId: 10
Links
Forward: 20
Forward: 40
Forward: 60
Name
Language
Code: 'en-us'
Country: 'us'
Language
Code: 'en'
Url: 'http://A'
Name
Url: 'http://B'
Name
Language
Code: 'en-gb'
Country: 'gb'
![Page 10: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/10.jpg)
Nested Columnar Storage
DocId: 10
Links
Forward: 20
Forward: 40
Forward: 60
Name
Language
Code: 'en-us'
Country: 'us'
Language
Code: 'en'
Url: 'http://A'
Name
Url: 'http://B'
Name
Language
Code: 'en-gb'
Country: 'gb'
value r d
10 0 0
20 0 0
DocId
value r d
http://A 0 2
http://B 1 2
NULL 1 1
Name.Url
value r d
en-us 0 2
en 2 2
NULL 1 1
en-gb 1 2
Name.Language.Code Name.Language.Country
Links.Forward
value r d
us 0 3
NULL 2 2
NULL 1 1
gb 1 3
value r d
20 0 2
40 1 2
60 1 2
![Page 11: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/11.jpg)
value r d
en-us 0 2
Name.Language.Code
r1.Name1.Language1.Code: 'en-us'
DocId: 10
Links
Forward: 20
Forward: 40
Forward: 60
Name
Language
Code: 'en-us'
Country: 'us'
Language
Code: 'en'
Url: 'http://A'
Name
Url: 'http://B'
Name
Language
Code: 'en-gb'
Country: 'gb'
DocId: 20
Links
Backward: 10
Backward: 30
Forward: 80
Name
Url: 'http://C'
r1
r2
Building Columns
Repetition (r) and definition (d) levels encode the structural delta between the current value and the previous value. (r): Length of common path prefix (d): Number of fields in the path that could be optional but are actually present
![Page 12: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/12.jpg)
value r d
en-us 0 2
Name.Language.Code
r1.Name1.Language1.Code: 'en-us‘
r1.Name1.Language2.Code: 'en'
DocId: 10
Links
Forward: 20
Forward: 40
Forward: 60
Name
Language
Code: 'en-us'
Country: 'us'
Language
Code: 'en'
Url: 'http://A'
Name
Url: 'http://B'
Name
Language
Code: 'en-gb'
Country: 'gb'
DocId: 20
Links
Backward: 10
Backward: 30
Forward: 80
Name
Url: 'http://C'
r1
r2
Building Columns
value r d
en-us 0 2
en 2 2
![Page 13: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/13.jpg)
Name.Language.Code
r1.Name1.Language1.Code: 'en-us‘
r1.Name1.Language2.Code: 'en‘
r1.Name2
DocId: 10
Links
Forward: 20
Forward: 40
Forward: 60
Name
Language
Code: 'en-us'
Country: 'us'
Language
Code: 'en'
Url: 'http://A'
Name
Url: 'http://B'
Name
Language
Code: 'en-gb'
Country: 'gb'
DocId: 20
Links
Backward: 10
Backward: 30
Forward: 80
Name
Url: 'http://C'
r1
r2
Building Columns
value r d
en-us 0 2
en 2 2
NULL 1 1
![Page 14: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/14.jpg)
Name.Language.Code
r1.Name1.Language1.Code: 'en-us‘
r1.Name1.Language2.Code: 'en‘
r1.Name2
r1.Name3.Language1.Code: 'en-gb'
DocId: 10
Links
Forward: 20
Forward: 40
Forward: 60
Name
Language
Code: 'en-us'
Country: 'us'
Language
Code: 'en'
Url: 'http://A'
Name
Url: 'http://B'
Name
Language
Code: 'en-gb'
Country: 'gb'
DocId: 20
Links
Backward: 10
Backward: 30
Forward: 80
Name
Url: 'http://C'
r1
r2
Building Columns
value r d
en-us 0 2
en 2 2
NULL 1 1
value r d
en-us 0 2
en 2 2
NULL 1 1
en-gb 1 2
![Page 15: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/15.jpg)
Name.Language.Code
r1.Name1.Language1.Code: 'en-us‘
r1.Name1.Language2.Code: 'en‘
r1.Name2
r1.Name3.Language1.Code: 'en-gb’
r2.Name1
DocId: 10
Links
Forward: 20
Forward: 40
Forward: 60
Name
Language
Code: 'en-us'
Country: 'us'
Language
Code: 'en'
Url: 'http://A'
Name
Url: 'http://B'
Name
Language
Code: 'en-gb'
Country: 'gb'
DocId: 20
Links
Backward: 10
Backward: 30
Forward: 80
Name
Url: 'http://C'
r1
r2
Building Columns
value r d
en-us 0 2
en 2 2
NULL 1 1
value r d
en-us 0 2
en 2 2
NULL 1 1
en-gb 1 2
value r d
en-us 0 2
en 2 2
NULL 1 1
en-gb 1 2
NULL 0 1
![Page 16: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/16.jpg)
Retrieving Columns
Name.Language.Country Name.Language.Code
Links.Backward Links.Forward
Name.Url
DocId
1
0
1
0
0,1,2
2
0,1 1
0
0
![Page 17: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/17.jpg)
Retrieving Columns
Name.Language.Country Name.Language.Code
Links.Backward Links.Forward
Name.Url
DocId
1
0
1
0
0,1,2
2
0,1 1
0
0
![Page 18: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/18.jpg)
Retrieving Columns
DocId
Name.Language.Country 1,2
0
0
value r d
10 0 0
20 0 0
DocId
Name.Language.Country
value r d
us 0 3
NULL 2 2
NULL 1 1
gb 1 3
![Page 19: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/19.jpg)
Retrieving Columns
value r d
10 0 0
20 0 0
DocId
Name.Language.Country
value r d
us 0 3
NULL 2 2
NULL 1 1
gb 1 3
DocId: 10
Name
Language
Country: 'us'
Language
Name
Name
Language
Country: 'gb'
DocId: 20
Name
s1
s2
![Page 20: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/20.jpg)
Hierarchical Query Processing
storage layer (e.g., GFS)
. . .
. . .
. . . leaf servers
(with local
storage)
intermediate
servers
root server
client
20
![Page 21: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/21.jpg)
• Optimized for Select-Project-Aggregate queries.
‐ Single Scan over Data
‐ Recursive Reducers
• Defers discussion of joins, indexing, updates etc. to future work.
• Scheduler’s Secret Sauce.
Hierarchical Query Processing
![Page 22: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/22.jpg)
Duplicate/Ignore Stragglers
percentage of processed tablets
processing time per tablet (sec)
Duplicates
or Ignores
Stragglers
![Page 23: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/23.jpg)
Comments/Critiques
![Page 24: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/24.jpg)
Does Dremel really require a new execution engine?
![Page 25: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/25.jpg)
What’s really novel about Aggregation Trees?
• Very similar to the MapReduce model (Leaf servers run Map tasks and Aggregators are Reduce tasks)
• Partial Aggregates/Recursive Reducers have already been proposed by Traditional Databases as well as SCOPE/Dryad.
![Page 26: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/26.jpg)
Can we make other tradeoffs?
• Input/Output
‐ Sequentially reading a Terabyte from disk in a second requires ~20,000 parallel reads!
• Processing
‐ CPU-intensive queries may need to run on thousands of cores to complete within a second.
‐ Dealing with failures and stragglers is essential.
![Page 27: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/27.jpg)
Can we make other tradeoffs?
• Input/Output
‐ Sequentially reading a Terabyte from disk in a second requires ~20,000 parallel reads! [Sampling? In-memory RDDs?]
• Processing
‐ CPU-intensive queries may need to run on thousands of cores to complete within a second.
‐ Dealing with failures and stragglers is essential.
![Page 28: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/28.jpg)
Can we make other tradeoffs?
• Input/Output
‐ Sequentially reading a Terabyte from disk in a second requires ~20,000 parallel reads! [Sampling? In-memory RDDs?]
• Processing
‐ CPU-intensive queries may need to run on thousands of cores to complete within a second. [Better Data Partitioning?]
‐ Dealing with failures and stragglers is essential.
![Page 29: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/29.jpg)
Can we make other tradeoffs?
• Input/Output
‐ Sequentially reading a Terabyte from disk in a second requires ~20,000 parallel reads! [Sampling? In-memory RDDs?]
• Processing
‐ CPU-intensive queries may need to run on thousands of cores to complete within a second. [Better Data Partitioning?]
‐ Dealing with failures and stragglers is essential. [Giving Answers with Bounded Errors/Confidence Intervals?]
![Page 30: Dremel: Interactive Analysis of Web-Scale Datasetsistoica/classes/... · Dremel: Interactive Analysis of Web-Scale Datasets Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey](https://reader033.vdocument.in/reader033/viewer/2022041712/5e48c995c7d0f317552faa70/html5/thumbnails/30.jpg)
Thank You!