a modern framework for amazon elastic mapreduce (bdt309) | aws re:invent 2013
DESCRIPTION
If you've ever developed code for processing data, you know what a mess it can be—especially on Hadoop. You lack debugging tools, instant feedback, automated tests, and a sane deploy. Mortar has developed a modern framework for data processing on Hadoop and Amazon Elastic MapReduce. It is a free, open framework providing instant, step-by-step execution visibility, automated testing, reusable components, and one-button deployment. See how Mortar demonstrates this framework on Amazon EMR on a sample data set to solve a big data problem.TRANSCRIPT
![Page 1: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/1.jpg)
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Mortar Data: K Young (CEO), Jeremy Karn (Lead Engineer)
November 15, 2013
A Modern Framework for EMR
Friday, November 15, 13
![Page 2: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/2.jpg)
K Young Jeremy Karn
Friday, November 15, 13
![Page 3: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/3.jpg)
This talk is technical
Friday, November 15, 13
![Page 4: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/4.jpg)
Great: HadoopGreater: Amazon EMR
Friday, November 15, 13
![Page 5: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/5.jpg)
A Modern Framework
Friday, November 15, 13
![Page 6: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/6.jpg)
A Modern Framework
Friday, November 15, 13
![Page 7: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/7.jpg)
Goal: 10x productivity• Collaboration• Efficient, Free Development• Testing / Debug• Reproducibility• And More...
Friday, November 15, 13
![Page 8: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/8.jpg)
For data science(not a database)
Friday, November 15, 13
![Page 9: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/9.jpg)
Iterate Locally
DeployCloud Execution EMR + Mortar
Friday, November 15, 13
![Page 10: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/10.jpg)
Demo
Find the most popularprojects on GitHub
Friday, November 15, 13
![Page 11: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/11.jpg)
Goal: Collaboration• Easily run code from others• Contribute changes back
Friday, November 15, 13
![Page 12: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/12.jpg)
Goal: Efficient, Free Development• Rapid iteration• No cost• Resilient to errors
Friday, November 15, 13
![Page 13: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/13.jpg)
Goal: Testing & Debug• Automated testing• See what is happening at runtime
Friday, November 15, 13
![Page 14: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/14.jpg)
Goal: Reproducibility• Know what was run• 1-button deploy• Rollback
Friday, November 15, 13
![Page 15: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/15.jpg)
Goal: Miscellaneous goals• Easy scheduling• Easily use other technologies: Python, Amazon
DynamoDB, MongoDB• Results easy to locate• Managed cluster lifetime• More granular API
Friday, November 15, 13
![Page 16: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/16.jpg)
A Modern Framework for Amazon EMR
• Collaboration• Efficient, Free Development• Testing / Debug• Reproducibility• And More...
What you saw
Friday, November 15, 13
![Page 17: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/17.jpg)
Next steps• bit.ly/mortar-reinvent• Documentation: help.mortardata.com• Follow: @mortardata
Friday, November 15, 13
![Page 18: A Modern Framework for Amazon Elastic MapReduce (BDT309) | AWS re:Invent 2013](https://reader033.vdocument.in/reader033/viewer/2022051818/54b3a7f14a79599c5c8b477a/html5/thumbnails/18.jpg)
Please give us your feedback on this presentation
As a thank you, we will select prize winners daily for completed surveys!
BDT309 Thank You
Friday, November 15, 13