[@indeedeng] imhotep workshop
DESCRIPTION
Link to video: http://youtu.be/LBDZFtqL-ck?list=UURVEh0SlyrZNTeIbEDwj3wQ We are excited to announce the open source availability of Imhotep, the interactive data analytics platform that powers data-driven decision making at Indeed. In a previous talk, we explained how we developed Imhotep, a distributed system for building decision trees for machine learning. We went on to describe how we build large scale interactive analytics tools using the same platform. Next we showed how our engineering and product organizations use Imhotep to focus on key metrics at scale. During this session, Product Manager Tom Bergman provided examples of valuable insights that can be gained by using Imhotep. After the presentation, attendees explored their own data in Imhotep. Product engineers were on hand to answer questions.TRANSCRIPT
![Page 1: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/1.jpg)
Imhotep Workshop
http://go.indeed.com/iws
![Page 2: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/2.jpg)
engineering.indeed.com/talks
![Page 3: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/3.jpg)
@IndeedEng Workshop:Interactive Analytics
with Imhotep
![Page 4: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/4.jpg)
Tom BergmanProduct Manager
![Page 5: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/5.jpg)
Is anybody there???
![Page 6: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/6.jpg)
Does this thing work???
![Page 7: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/7.jpg)
Are we winning???
![Page 8: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/8.jpg)
Harder questions???
![Page 9: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/9.jpg)
Imhotep is Indeed’s highly scalable
open-source analytics platform.
What is Imhotep?
![Page 10: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/10.jpg)
● Imhotep Daemons
● Imhotep Query Language (IQL)
● IQL Web Client
● TSV/CSV Uploader
Imhotep open source included:
![Page 11: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/11.jpg)
● Easy upload & compression
● Fast, Interactive queries
What does Imhotep do?
![Page 12: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/12.jpg)
Imhotep Philosophy
![Page 13: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/13.jpg)
● Quickly refine your questions
Interactive
![Page 14: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/14.jpg)
SOME TIME LATER…
Time to the right question
Oh, bummer. Wrong question. Let’s try again.
Nope. Nope. YES!
Next question?
Hadoop and the Hadoop elephant logo are trademarks of the Apache Software Foundation.
![Page 15: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/15.jpg)
Ask all the questions!
Cool! Really?
Wow... Awesome
Oh… Ah! INSIGHT! …
![Page 16: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/16.jpg)
● Data should not be down-sampled
Ground Truth
![Page 17: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/17.jpg)
![Page 18: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/18.jpg)
● Web-based to facilitate sharing
Show me the data
![Page 19: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/19.jpg)
![Page 20: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/20.jpg)
![Page 21: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/21.jpg)
● Instantaneous sharing
Cache Rules Everything Around Me
![Page 22: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/22.jpg)
● Easily queryable
Easy Access
![Page 23: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/23.jpg)
● Dataset >
● Document >
● Field >
● Term >
Imhotep Data Structures
● DB Table
● Denormalized Row
● Column
● Value
![Page 24: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/24.jpg)
Imhotep Query Language (IQL)
![Page 25: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/25.jpg)
IQL - Imhotep Query Language
Expressive SQL-like language for aggregate analytics.
![Page 26: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/26.jpg)
IQL queries - requirements
Dataset Date range
![Page 27: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/27.jpg)
IQL queries - optional
DatasetDate range
FiltersGroup byMetrics
![Page 28: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/28.jpg)
from searchresults
‘2013-12-05’
‘2013-12-10’
where country=ie
and jobagedays<1
group by time(1d)
select clicked, count()
IQL - Dataset
Dataset
![Page 29: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/29.jpg)
from searchresults
‘2013-12-05’
‘2013-12-10’
where country=ie
and jobagedays<1
group by time(1d)
select clicked, count()
IQL - Date Range
Date Range
![Page 30: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/30.jpg)
from searchresults
‘2013-12-05’
‘2013-12-10’
where country=ie
and jobagedays<1
group by time(1d)
select clicked, count()
IQL - Filters
Filters
![Page 31: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/31.jpg)
from searchresults
‘2013-12-05’
‘2013-12-10’
where country=ie
and jobagedays<1
group by time(1d)
select clicked, count()
IQL - Group by
Groups
![Page 32: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/32.jpg)
IQL - Metrics
from searchresults
‘2013-12-05’
‘2013-12-10’
where country=ie
and jobagedays<1
group by time(1d)
select clicked, count()
Metrics
![Page 33: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/33.jpg)
IQL - Example Results
![Page 34: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/34.jpg)
![Page 35: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/35.jpg)
Extract Transform Load (ETL)
![Page 36: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/36.jpg)
● Up to your data schema
Extract
![Page 37: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/37.jpg)
● De-Normalize Data
● Formating
Transform
![Page 38: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/38.jpg)
● Jobsearch
● Ad Clicks
● Resume Contacts
Example Datasets
![Page 39: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/39.jpg)
● TSV/CSV Uploader
Load
![Page 40: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/40.jpg)
● TSV/CSV Uploader
● Java API
Load
![Page 41: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/41.jpg)
● Massive compression
● Fast Boolean Search
Inverted Index
![Page 42: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/42.jpg)
● Run on any* computer/s
● Can deploy to AWS using handy
AWS CloudFormation script
Opensource Package
![Page 43: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/43.jpg)
● Create S3 buckets
● Create EC2 Key Pair
● Run CloudFormation script
CloudFormation Setup
![Page 44: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/44.jpg)
DEMO
![Page 45: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/45.jpg)
Q&A
![Page 46: [@IndeedEng] Imhotep Workshop](https://reader036.vdocument.in/reader036/viewer/2022062320/5598cdd81a28ab8d338b4637/html5/thumbnails/46.jpg)
Helpful Workshop Linkshttp://go.indeed.com/iws