building big data applications with sql-mapreduce and...
TRANSCRIPT
![Page 1: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/1.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.1
Title of Presenter | Date
Building Big Data Applications with SQL-MapReduce and Aster Developer Express
Peter Pawlowski, Software Engineer
![Page 2: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/2.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.2
Big Data: The Trend
Many companies have anever-increasing amount of data.
![Page 3: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/3.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.3
Big Data: The Challenge
How do we enable all these companies to build useful
applications using this data?
“Big Data Applications”
![Page 4: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/4.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.4
Not Everyone is Google
![Page 5: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/5.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.5
Our Goal: Big Data Applications Made Easy
1. We enable storing terabytes to petabytes of data in our scalable, parallel database.
2. We provide a framework for running rich, interesting analytics on this data.
![Page 6: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/6.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.6
Our Goal: Big Data Applications Made Easy
1. We enable storing terabytes to petabytes of data in our scalable, parallel database.
2. We provide a framework for running rich, interesting, and fast analytics on this data.
![Page 7: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/7.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.7
Aster Data nCluster
Workers (Parallel Processing)
Loading/Export
Queries
Backup
Queries/Results
Reports, Advanced Analytics, Applications(SQL, SQL-MapReduce, ODBC , JDBC)
Queen
Data Data
![Page 8: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/8.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.8
We started with a SQL interface
• Great fit for many data problems
• Many people know it
• Plugs into many applications & tools
![Page 9: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/9.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.9
… but SQL …
• Is a bad fit for many kinds of analytic applications
• Can be hard to write & maintain
• Can be hard to execute efficiently
![Page 10: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/10.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.10
Enter SQL-MapReduce
• SQL-MapReduce is our framework for pushing analytic code into the database
• Enables much richer analytics to happen on the data in the database
![Page 11: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/11.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.11
Two Steps to SQL-MapReduce
• Write a SQL-MapReduce function in Java*
• Invoke it from SQL
* Other languages supported, too. See our website.
![Page 12: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/12.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.12
![Page 13: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/13.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.13
Test Environment
Deploy3Test2Develop1
SQL / SQL-MR App
SQL / SQL-MR App
Aster Data Developer Express
![Page 14: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/14.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.14
Step 1: Develop
![Page 15: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/15.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.15
Step 2: Test Locally
Desktop Test Environment
Input OutputInput Files
Output Files
• Run tests on your own machine without requiring an nCluster instance.
• Configure with JSON.
Application
Results
![Page 16: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/16.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.16
Step 2: Test Locally
Desktop Test Environment
Input OutputInput Files
Output Files
• Run tests on your own machine without requiring an nCluster instance.
• Configure with JSON.
Application
Results
![Page 17: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/17.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.17
Step 2: Test Locally
![Page 18: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/18.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.18
Step 3: Deploy
![Page 19: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/19.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.19
Step 3: Deploy
Aster Data nCluster
App App AppApp App App
SQL / SQL-MR App
• Push application into Aster database directly from within IDE
• Initiate and validate application with actual data stored inside nCluster
![Page 20: Building Big Data Applications with SQL-MapReduce and ...wiki.eclipse.org/images/f/f8/Aster_Eclipse_Slides.pdf · 20 Confidential and proprietary. Copyright © 2010 Aster Data Systems,](https://reader034.vdocument.in/reader034/viewer/2022042123/5e9e9e6b51e4ee41272e9218/html5/thumbnails/20.jpg)
Confidential and proprietary. Copyright © 2010 Aster Data Systems, Inc.20
Aster Data Developer Express is freely available
Download atwww.asterdata.com