breaking the memory wall in monetdb
DESCRIPTION
It regarding memory...and the usage of memory now-a-days..n how to overcome this problem.we use MonetDb to overcome this problem, monetDB is a open source column structure.TRANSCRIPT
![Page 1: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/1.jpg)
Breaking the Memory Wall in MonetDB
Presented By- M.Sheshasai Reddy (07D91A1225)
CSE 704 1
![Page 2: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/2.jpg)
Memory Wall
• Growing disparity of speed between CPU and memory outside CPU chip
• 20-40 % of instructions in a program reference memory
• It is a performance bottleneck to access main memory
CSE 704 2
![Page 3: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/3.jpg)
Want to break the Memory Wall?
• Average time to access memory can be written as:
tavg=p x tc + (1-p) x tm
where p = probability of a cache hit
tc = time to access cache
tm = DRAM access time
• To reduce tavg, increase p (or reduce tc)
• Exploit cache
CSE 704 3
![Page 4: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/4.jpg)
Exploit the power of cache by :
• Vertical Storage
• Bulk query Algebra
• Cache conscious algorithms
• Memory Access cost modeling
CSE 704 4
![Page 5: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/5.jpg)
MonetDB
• 1994 - Now
• Open source Column Store
• Has been successfully applied in high-performance applications for data mining, OLAP, GIS, XML Query, text and multimedia retrieval.
CSE 704 5
![Page 6: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/6.jpg)
MonetDB Architecture
CSE 704 6
![Page 7: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/7.jpg)
Exploit the power of cache by :
• Vertical Storage
• Bulk query Algebra
• Cache conscious algorithms
• Memory Access cost modeling
CSE 704 7
![Page 8: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/8.jpg)
MonetDB- Vertical storageID Day Discount
10 4/4/98 0.19511 9/4/98 0.06512 1/2/98 0.17513 7/2/98 0
CSE 704 8
OID ID100 10101 11102 12103 13104 14
OID Day100 4/4/98101 9/4/98102 1/2/98103 7/2/98104 1/2/99
OID Discount100 0.195101 0.065102 0.175103 0104 0.065
![Page 9: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/9.jpg)
MonetDB-Vertical Storage
• Uses Decomposed Storage Model (DSM)
• Each column stored in a separate Binary Association Table (BAT)
• BAT <surrogate,value>• <head, tail>• Two simple memory arrays • Internally, columns are stored using
memory-mapped files• Provides O(1) database lookup mechanism
CSE 704 9
![Page 10: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/10.jpg)
Exploit the power of cache by :
• Vertical Storage
• Bulk query Algebra
• Cache conscious algorithms
• Memory Access cost modeling
CSE 704 10
![Page 11: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/11.jpg)
Bulk Query Algebra
• Low-level algebra called BAT Algebra
• Breaks complex expressions into a sequence of BAT algebra operators
• Implementing these operations are simple array operations
• No need of Expression Interpreting engine
CSE 704 11
![Page 12: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/12.jpg)
Example
• BAT algebra expression
R:bat[:oid, :oid]:=select(B:bat[:oid,:int], V:int)
• C code implementing the above expression
for (i = j = 0; i <n; i++)
if (B.tail[i] == V)
R.tail[j++] = i;
CSE 704 12
![Page 13: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/13.jpg)
MonetDB Vertical Storage
• Advantage: – For loop creates high instruction locality and
thus eliminates the cache miss problem
• Disadvantage:– Materialize intermediate results
CSE 704 13
![Page 14: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/14.jpg)
Exploit the power of cache by :
• Vertical Storage
• Bulk query Algebra
• Cache conscious algorithms
• Memory Access cost modeling
CSE 704 14
![Page 15: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/15.jpg)
MonetDB- Cache concious algorithms• Partitioned Hash Join
– Relations partitioned into H separate clusters
– Each cluster fits in L2 memory cache
• ProblemNumber of clusters (H) exceeds the number of
- TLB entries TLB thrashing- Cache lines Cache thrashing
• Solution
-Multi-pass radix cluster
CSE 704 15
![Page 16: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/16.jpg)
Multipass radix cluster• Multiple clustering passes
• Limit the number of clusters per pass
• Avoids TLB/Cache thrashing
CSE 704 16
![Page 17: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/17.jpg)
Exploit the power of cache by :
• Vertical Storage
• Bulk query Algebra
• Cache conscious algorithms
• Memory Access cost modeling
CSE 704 17
![Page 18: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/18.jpg)
Memory Access cost modeling
• Cost models are created that take the cost of memory access into account
• Main goal is to predict the number and kind of cache misses for all cache levels.
• Complex data access patterns of database algorithms are studied.
CSE 704 18
![Page 19: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/19.jpg)
Basic Access Patterns
• Single sequential traversal s_trav(R)
• Single random traversal r_trav(R)
• Repetitive sequential traversal rs_trav(r,d,R)
• Repetitive random traversal rr_trav(r,d,R)
R= region representing a database table
CSE 704 19
![Page 20: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/20.jpg)
Compound Access Patterns
• Database operations accessing more than one data region
• Can be created by combining basic access patterns
• Basic access pattern can be combined either– Sequentially– ConcurrentlyExample: s_trav(U) s_trav(W) for W<-
select(U)
CSE 704 20
![Page 21: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/21.jpg)
Sequential Execution
• Total number of cache misses is at most the sum of the cache misses of all patterns
• However if the data region for patterns is same, cache miss can be saved
CSE 704 21
Concurrent Execution• Patterns are competing for the same
cache
• Considering this, total number of cache misses are calculated
![Page 22: Breaking the memory wall in MonetDB](https://reader033.vdocument.in/reader033/viewer/2022061110/5453d867b1af9f37608b4863/html5/thumbnails/22.jpg)
Conclusion
• MonetDB has redefined database architecture in the face of an ever-changing computer architecture
• Principle of Vertical storage is currently being adopted by many commercial systems
• HBase, C-Store, Apache Cassandra are some of the other column stores
CSE 704 22