Download - IMC Summit 2016 Breakout - Nikita Shamgunov - Propelling IoT Innovation with Predictive Analytics
Topics• The On-Demand Economy
• From In-Memory Compu8ng to In-Memory Databases
• Renewable Energy and PowerStream
• Demo and Q&A
(c) Nikita Shamgunov and MemSQL
Consumers are condi.oned to instant services, like Uber, Stripe, and Airbnb
(c) Nikita Shamgunov and MemSQL
Racing to meet internal and external expecta1ons for speed and
personaliza,on
(c) Nikita Shamgunov and MemSQL
Enterprises must move from overnight to
Real-&me, intra-day opera&ons
(c) Nikita Shamgunov and MemSQL
Achieving sub 100 ms latency• Real-'me monitoring and analy'cs on streaming video
• Proac'vely diagnose issues in real-'me
• Deliver be9er viewer experience
(c) Nikita Shamgunov and MemSQL
Massive Ingest AND Analy1cs• Instant accuracy to the latest repin
• Build real-5me analy5c applica5ons
• 1 GB/sec totaling 72 TB/day
(c) Nikita Shamgunov and MemSQL
In-Memory Databases...• Use memory instead of disk
• Do not (need to) save data on disk
(c) Nikita Shamgunov and MemSQL
In-Memory Databases...• Use memory instead of disk
• Do not (need to) save data on disk
(c) Nikita Shamgunov and MemSQL
In-Memory Databases...• Use memory instead of disk
• Do not (need to) save data on disk
• Put the whole dataset in memory
(c) Nikita Shamgunov and MemSQL
In-Memory Databases...• Use memory instead of disk
• Do not (need to) save data on disk
• Put the whole dataset in memory
(c) Nikita Shamgunov and MemSQL
In-Memory Databases...• Use memory instead of disk
• Do not (need to) save data on disk
• Put the whole dataset in memory
Well, some)mes...
(c) Nikita Shamgunov and MemSQL
Wikipedia says...
In-memory databases primarily rely on main-memory for storage.
(c) Nikita Shamgunov and MemSQL
In-Memory Databases• Are durable to disk (and respect ACID)
• Can spill on disk or pin data in-memory (and take advantage of it)
(c) Nikita Shamgunov and MemSQL
In-Memory Databases• Are durable to disk (and respect ACID)
• Can spill on disk or pin data in-memory (and take advantage of it)
• Tradeoffs are suited to systems with lots of memory
(c) Nikita Shamgunov and MemSQL
In-Memory Databases• Are durable to disk (and respect ACID)
• Can spill on disk or pin data in-memory (and take advantage of it)
• Tradeoffs are suited to systems with lots of memory
• Tend to be distributed systems
(c) Nikita Shamgunov and MemSQL
In-Memory Databases• Are durable to disk (and respect ACID)
• Can spill on disk or pin data in-memory (and take advantage of it)
• Tradeoffs are suited to systems with lots of memory
• Tend to be distributed systems
• Have a different set of boClenecks
(c) Nikita Shamgunov and MemSQL
Why?• Memory is ge,ng cheaper (about 40% every year)
• Cache is the new RAM (RAM is the new disk, disk is the new tape, etc)
(c) Nikita Shamgunov and MemSQL
Why?• Memory is ge,ng cheaper (about 40% every year)
• Cache is the new RAM (RAM is the new disk, disk is the new tape, etc)
• In-memory databases leverage SSD (no random writes)
(c) Nikita Shamgunov and MemSQL
Why?• Memory is ge,ng cheaper (about 40% every year)
• Cache is the new RAM (RAM is the new disk, disk is the new tape, etc)
• In-memory databases leverage SSD (no random writes)
• NVRAM is coming (and could be cheaper than SSD)
(c) Nikita Shamgunov and MemSQL
Why?• Memory is ge,ng cheaper (about 40% every year)
• Cache is the new RAM (RAM is the new disk, disk is the new tape, etc)
• In-memory databases leverage SSD (no random writes)
• NVRAM is coming (and could be cheaper than SSD)
In-memory databases are tuned to modern hardware and modern workloads
(c) Nikita Shamgunov and MemSQL
Demo Sequence• Powerstream user interface
• Showcase largest windfarms
• Real-8me simula8ons
• Witness live opera8ons
• Ease of new pipeline setup
• Ka>a subscrip8on
(c) Nikita Shamgunov and MemSQL
Enabling predic.ve analy.cs• Use exis(ng models from SAS
• Create models in Spark MLlib
• Predic(ve scoring as part of the pipeline
(c) Nikita Shamgunov and MemSQL