building a recommendation engine with spring and hadoop
DESCRIPTION
Speaker: Michael Minella Big Data Track The Amazon’s and Google’s of the world have had Ph.D.’s locked up in back rooms for years creating algorithms to get you to click on things and subsequently buy stuff. One of the big things that those smart people have been working on are recommendation engines. Today, a recommendation engine isn’t something that only the Amazon’s of the world can have. With an hour, and a handful of open source tools, we’ll build a recommendation engine based on the data from the website we probably spend the most time on…StackOverflow. We’ll use Spring XD and Spring Batch to orchestrate the full lifecycle of Hadoop processing (ingest, process, export) and use Apache Mahout to provide us with the recommendation processing. A basic understanding of Hadoop concepts (what Map/Reduce is) and Spring (basic D/I configuration) is expected for this talk.TRANSCRIPT
![Page 1: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/1.jpg)
BUILDING
ENGINES
WITH SPRING
![Page 2: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/2.jpg)
MICHAEL MINELLATWITTER: @MICHAELMINELLA
HOME PAGE: SPRING.IO/TEAM/MMINELLA
![Page 3: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/3.jpg)
WHAT I’M NOT
![Page 4: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/4.jpg)
![Page 5: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/5.jpg)
https://github.com/SpringOne2GX-2014/
![Page 6: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/6.jpg)
THANK YOUSEBASTIAN SCHELTERPAT FERREL
![Page 7: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/7.jpg)
![Page 8: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/8.jpg)
13
![Page 9: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/9.jpg)
![Page 10: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/10.jpg)
RECOMMENDATION
ALGORITHMS
![Page 11: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/11.jpg)
![Page 12: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/12.jpg)
L E T ’ S S E T S O M E
EXPECTATIONS
![Page 13: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/13.jpg)
![Page 14: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/14.jpg)
![Page 15: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/15.jpg)
![Page 16: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/16.jpg)
![Page 17: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/17.jpg)
![Page 18: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/18.jpg)
SCALE OF THE PROBLEM
![Page 19: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/19.jpg)
MILLIONS OF
USERS
![Page 20: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/20.jpg)
100,000’s OF
ITEMS
![Page 21: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/21.jpg)
TOOLS AND
TECHNOLOGIES
![Page 22: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/22.jpg)
1SPRING BOOT
![Page 23: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/23.jpg)
2MYSQL
![Page 24: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/24.jpg)
3HADOOP
![Page 25: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/25.jpg)
4SPRING XD
![Page 26: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/26.jpg)
5MAHOUT
![Page 27: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/27.jpg)
SPRING XDEXTREME DATA
![Page 28: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/28.jpg)
APPLICATIONCOMPLEXITY
![Page 29: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/29.jpg)
L O T S O F
BOILERPLATE
![Page 30: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/30.jpg)
MANY DOMAINS TO
BRIDGE
![Page 31: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/31.jpg)
I N C O N S I S T E N T
APIS
![Page 32: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/32.jpg)
SOURCE, CHANEL, SINK
DATA FLOW MODEL
ADAPTER, CHANEL, FILTER, TRANSFORMER, ETC
EIP PATTERNS
=
![Page 33: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/33.jpg)
JOB, CONNECTOR
IMPORT/EXPORT
JOB, ITEMREADER/ITEMWRITER
BATCH PROCESSING
=
![Page 34: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/34.jpg)
WORKFLOW, ACTION
WORKFLOWORCHESTRATION
JOB, STEP
BATCH PROCESSING
=
![Page 35: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/35.jpg)
SPRING XDEXTREME DATA
![Page 36: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/36.jpg)
SPRING
Ingestion
Orchestration
Extraction
Real-time
Analytics
![Page 37: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/37.jpg)
D I S T R I B U T E D
RUNTIME
![Page 38: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/38.jpg)
STREAMING
BATCH&
![Page 39: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/39.jpg)
![Page 40: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/40.jpg)
--directory=/xd/dir1
filter --expression=“payload?.price > 3.00” |
http | hdfs--port=8181
![Page 41: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/41.jpg)
BATCH PROCESSING FOR
HEAVY LIFTING
![Page 42: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/42.jpg)
JOB
![Page 43: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/43.jpg)
STEP
![Page 44: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/44.jpg)
TASKLET
![Page 45: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/45.jpg)
CHUNK
![Page 46: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/46.jpg)
SPRING FOR
APACHE HADOOP
![Page 47: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/47.jpg)
![Page 48: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/48.jpg)
TOTAL LINES OF CUSTOM CODE
47 Lines of Java
29 Lines of XML
6 Spring XD Shell Commands
![Page 49: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/49.jpg)
RECOMMENDATION
ALGORITHMS
![Page 50: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/50.jpg)
PREDICTING THE
FUTURE
![Page 51: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/51.jpg)
C O L L A B O R AT I V E
FILTERING
![Page 52: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/52.jpg)
TWO OPTIONS
![Page 53: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/53.jpg)
USER BASED
![Page 54: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/54.jpg)
USER ITEM 1ITEM 2ITEM 3ITEM 4ITEM 5
DEREK
MICHAEL
PHIL
DARREL ?
![Page 55: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/55.jpg)
USER BASED
![Page 56: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/56.jpg)
USER BASED
![Page 57: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/57.jpg)
ITEM BASED
![Page 58: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/58.jpg)
?
ITEM DEREKMICHAELPHILDARREL
ITEM 1
ITEM 2
ITEM 3
ITEM 4
ITEM 5
![Page 59: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/59.jpg)
ITEM BASED
![Page 60: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/60.jpg)
ITEM BASED
![Page 61: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/61.jpg)
PEOPLE ARE
FUNNY
![Page 62: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/62.jpg)
![Page 63: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/63.jpg)
![Page 64: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/64.jpg)
![Page 65: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/65.jpg)
![Page 66: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/66.jpg)
USER_ID, TAG_ID, VOTES
TAG_ID, TAG_ID, SCORE
![Page 67: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/67.jpg)
![Page 68: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/68.jpg)
LOOKING INTO THE
FUTURE
![Page 69: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/69.jpg)
SNAPSHOTS AHEAD!
![Page 70: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/70.jpg)
![Page 71: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/71.jpg)
MAP REDUCE
![Page 72: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/72.jpg)
M A P R E D U C E
PROBLEMS
![Page 73: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/73.jpg)
A P I I S V E R Y
LOW LEVEL
![Page 74: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/74.jpg)
H I G H
LATENCY
![Page 75: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/75.jpg)
N O T A LWAY S A
GOOD FIT
![Page 76: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/76.jpg)
![Page 77: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/77.jpg)
POTENTIALLY
FASTER
![Page 78: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/78.jpg)
HIGHER LEVEL
APIS
![Page 79: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/79.jpg)
scala> textFile.count()
res0: Long = 126
![Page 80: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/80.jpg)
![Page 81: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/81.jpg)
USER_ID, TAG_ID, VOTES
TAGID,TAGID:RANK…
![Page 82: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/82.jpg)
U S E A
SEARCH ENGINE1
![Page 83: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/83.jpg)
D ATA
NORMALIZATION2
![Page 84: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/84.jpg)
![Page 85: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/85.jpg)
![Page 86: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/86.jpg)
Learn More. Stay Connected.
Spring BatchProject: spring.io/spring-batchGithub: github.com/spring-projects/spring-batchJira: jira.spring.io/browse/BATCH
Spring BootProject: spring.io/spring-bootGithub: github.com/spring-projects/spring-boot
Spring XDProject: spring.io/spring-xdGithub: github.com/spring-projects/spring-xdJira: jira.spring.io/browse/XD
Twitter: twitter.com/springcentral
YouTube: spring.io/video
LinkedIn: spring.io/linkedin
Google Plus: spring.io/gplus
![Page 87: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/87.jpg)
Servers by Jaime Carrion
from The Noun Project
Question by Jessica Lock
from The Noun Project
Check Box by Hrag Chanchanian
from The Noun Project
Crane by Kenneth Von Alt
from The Noun Project
Nut by Naomi Atkinson
from The Noun Project
Funnel by Volodin Anton
from The Noun Project
Circuit by Piotrek Chuchla
from The Noun Project
Puzzle by Matthew Hall
from The Noun Project
Database by Anton Outkine
from The Noun Project
Network by Mister Pixel
from The Noun Project
Puzzle by Eric M. Ellis
from The Noun Project
People by Wilson Joseph
from The Noun Project
Maze by Gilbert Bages
from The Noun Project
Fork by Dmitry Baranovskiy
from The Noun Project
Algebra by Ilsur Aptukov
from The Noun Project
Thumbs Up by Jørgen Bovolden
from The Noun Project
Scale by Edward Boatman
from The Noun Project
Users by Vittorio Maria Vecchi
from The Noun Project
Flow Chart by Michael Wohlwend
from The Noun Project
Running by Dimiter Petrov
from The Noun Project
Move by Dmitry Baranovskiy
from The Noun Project
Running by Dimiter Petrov
from The Noun Project
Abacus byAlice Mortaro
from The Noun Project
Stopwatch by Scott Lewis
from The Noun Project
Lego by jon trillana
from The Noun Project
Lego by jon trillana
from The Noun Project
Lego by jon trillana
from The Noun Project
Lego by Jake Dunham
from The Noun Project
![Page 88: Building a Recommendation Engine with Spring and Hadoop](https://reader034.vdocument.in/reader034/viewer/2022042817/559c1e2d1a28abd3298b4678/html5/thumbnails/88.jpg)
TheEnd