big data overview
DESCRIPTION
An introduction to big data. What's big data, why we'd want it , how is it applicable to CSPs, short intro to Hadoop (some of the info is in the slide notes)TRANSCRIPT
![Page 1: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/1.jpg)
BIG DATAArnon Rotem-Gal-Oz
Director of Technology Research, AmdocsThe blind men and the elephant. Poem by John Godfrey Saxe (Cartoon originally copyrighted by the authors; G. Renee Guzlas, artists http://www.nature.com/ki/journal/v62/n5/fig_tab/4493262f1.html
![Page 2: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/2.jpg)
1880 US Census
![Page 3: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/3.jpg)
![Page 4: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/4.jpg)
HollerithTabulating Machine
Hollerith photos by Martin Wichary : http://www.flickr.com/photos/mwichary/4358926764/in/photostream/
![Page 5: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/5.jpg)
![Page 6: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/6.jpg)
Source: Silicon Angle http://siliconangle.com/blog/2013/11/13/how-big-is-big-data-really/
Big data happens when the data you have to process is bigger than what you can process in the given time with current
technologies
![Page 7: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/7.jpg)
Myth: Big data = keep all data
Source: Big Data Public Private Forum : http://www.big-project.eu/sites/default/files/D2.2.1_First%20draft%20of%20Technical%20white%20papers_FINAL_v1.01_0.pdf
![Page 8: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/8.jpg)
Source: Big Data Public Private Forum : http://www.big-project.eu/sites/default/files/D2.2.1_First%20draft%20of%20Technical%20white%20papers_FINAL_v1.01_0.pdf
![Page 9: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/9.jpg)
Some Telco Numbers
Source: Wikipediahttp://upload.wikimedia.org/wikipedia/commons/5/50/Telephone_operators,_1952.jpg
![Page 10: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/10.jpg)
So, what do we do with all this data?
Source: Wikipedia http://upload.wikimedia.org/wikipedia/commons/0/06/UPS_Truck.jpg
![Page 11: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/11.jpg)
It’s the insights, stupid*
* With apologies to Bill Clinton
![Page 12: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/12.jpg)
Source: Silicon Angle http://siliconangle.com/blog/2013/11/13/how-big-is-big-data-really/
Big data analytics is when sample = N
• Big data happens when the data you have to process is bigger than what you can process in the given time with current technologies
![Page 13: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/13.jpg)
![Page 14: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/14.jpg)
“My daughter got this in the mail!, She’s still in high school,
and you’re sending her coupons for baby clothes and cribs? Are you trying to encourage her to
get pregnant?”
• Source: Forbes http://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-teen-girl-was-pregnant-before-her-father-did/
![Page 15: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/15.jpg)
We need to watch out thatAnalytics won’t get too creepy
![Page 16: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/16.jpg)
When people hear big data they think
fast data
Source: Steve Jones Cap Geminihttp://www.no.capgemini.com/node/778541
![Page 17: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/17.jpg)
Subscribers
Collect& Filter Correlate
(simplified) Network proactive care flow
Account
Event Store
Identify & Predict NetworkFailures
ReimburseVIPs
Prioritize technicians
Identify impact on
high valued Accounts
![Page 18: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/18.jpg)
Source: Silicon Angle http://siliconangle.com/blog/2013/11/13/how-big-is-big-data-really/
Big data is when we can handle data fast enough to make a difference
• Big data happens when the data you have to process is bigger than what you can process in the given time with current technologies
• Big data analytics is when sample = N
![Page 19: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/19.jpg)
Technology space
![Page 20: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/20.jpg)
The Elephant in the room
![Page 21: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/21.jpg)
Hadoop Stack
Map/Reduce
HDFS
HBase
PigHive
ZooKeeper
Oozie MahoutGiraph
![Page 22: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/22.jpg)
Schema on read
![Page 23: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/23.jpg)
Move data to computation
![Page 24: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/24.jpg)
Maybe we should rethink moving data to computation…
Source : http://my-inner-voice.blogspot.co.il/2012/06/haddop-101-paper-by-miha-ahronovitz-and.html
![Page 25: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/25.jpg)
![Page 26: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/26.jpg)
Map/reduce
Source: http://www.bodhtree.com/blog/2012/10/18/ever-wondered-what-happens-between-map-and-reduce/
![Page 27: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/27.jpg)
Customer Segmentation
First name
Last name
ARPU Age Device Country …
Mr. Smith 100 22 iPhone 5s,White USA
John Doe 87 42 Samsung Galaxy S5,Gold France
Lady In Red 105 21 Samsung Note 3, White UK
…
Uluru, Australia by Stuart Edwards (cc) http://en.wikipedia.org/wiki/Uluru#mediaviewer/File:Uluru_Panorama.jpg
![Page 28: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/28.jpg)
K-Means
ARPU
Age
Source : http://pypr.sourceforge.net/kmeans.html
![Page 29: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/29.jpg)
K=3AR
PU
Age
ARPU
Age
Source : http://pypr.sourceforge.net/kmeans.html
![Page 30: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/30.jpg)
New paradigms
Map/Reduce
HDFS
HBase
PigHive
ZooKeeper
Oozie MahoutGiraph
![Page 31: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/31.jpg)
New Paradigms
Map/Reduce
HDFS
HBase
Pig HiveZoo
Keeper
Oozie Mahout
YARN
Giraph
![Page 32: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/32.jpg)
New Paradigms
Map/Reduce
HDFS
HBase
Pig HiveZoo
Keeper
Oozie Mahout
YARN
Giraph SparkStorm
Slider
Flink
Impala
Tez
Presto
![Page 33: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/33.jpg)
Amdocs Analytics & Data Management Heritage
2013
• Proactive Care• TerraScale• Network optimization
• Real time analytics platform
• Single product catalog
• BSS–OSS Integration
• CRM-Billing Integration
OSSAnalytics Platform,
16 Analytics Patents
• aLDM logical data model
• Policy control
Network AnalyticsCRM
2000 2008
Acqu
isiti
ons
Portf
olio
![Page 34: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/34.jpg)
34Information Security Level 2 – Sensitive© 2014 – Proprietary and Confidential Information of Amdocs
Touchpoints & Applications
CRM Self Service E-MailPCRF SMS OtherWi-Fi OffloadCampaign Mng. • • • • • • •
Operational Envelope & Platform Administration
• Security Management
• Configuration Management
• Services Inventory
• Performance Management
• Fault Management
• LoggerCollect & Ingest
Transform & Enrich
Aggregate & Correlate
Drive Insight
Close the Loop
Machine Learn &
ScoreApplication-Ready Data and Analytics/ML Insights
Entities and Profiles
Detailed Data
OSSProbes Social RAN Inventory Usage &
ChargingCRM
Real-Time & Batch Connectors
Insight Platform
Marketing
AnalyticalApplication Framework:
Dashboards & Visualisation
Decisioning Engine
Dynamic Micro Segmentation
Network Care Operations
![Page 35: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/35.jpg)
Source: Silicon Angle http://siliconangle.com/blog/2013/11/13/how-big-is-big-data-really/
• Big data happens when the data you have to process is bigger than what you can process in the given time with current technologies
• Big data analytics is when sample = N
• Big data is when we can handle data fast enough to make a difference
![Page 36: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/36.jpg)
Additional takeaways
• CSPs have always been in the big data business – they just didn’t know it
• Big data is not a panacea • Hadoop is shaping up as the big data OS– Though there are alternatives arriving from the
cloud arena (mesos, kubernetes)
![Page 37: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/37.jpg)
What we covered here is not even
the tip of the iceberg
Source: wikimedia http://commons.wikimedia.org/wiki/File:Iceberg.jpg
![Page 38: Big data Overview](https://reader035.vdocument.in/reader035/viewer/2022070300/54102f818d7f72aa0e8b461b/html5/thumbnails/38.jpg)
Arnon Rotem-Gal-Oz Director of Technology Research, [email protected] / [email protected]