airbnb tech talk: josh wills on the life of a data scientist

The Life of a Data ScientistMay 23, 2012

About Me

• jwills@cloudera.com and @josh_wills• Formerly of Google (2008 – 2011)

• Worked on the ad auction• Led the team that build the data infrastructure for Google+

• Before that: a bunch of startups• Sometimes as a software engineer, sometimes as a statistician

• Math degree from Duke and a half-finished PhD from The University of Texas at Austin

• Now: Director of Data Science at Cloudera

@josh_wills, #hacker

@josh_wills, #ThoughtLeader

What is a Data Scientist?

One Definition…

… versus Another

Why Is Everyone Talking About Them?

Because They Make Things Fun.

Data Scientists Power The Products You Love

The Job Isn’t New. The Impact Is.

How Do I Become One?

The Standard Reply

Personality Trait #1: Relentless, but in a Lazy Way

Personality Trait #2: (Acquired) Humility

Step 1: Study Math

But…I didn’t study math.

Alternate Step 1: Study (Computer) Science

Things People Don’t Know About Computer Science

Things Scientists Don’t Know About Statistics

Problem Solving In Context

Phase 2: Stuff You Still Don’t Know

Statisticians: How to Work on a Engineering Team

• Modular software design

• Unit tests• Code reviews• Automated build and

test infrastructure• Source code

management

Software Engineers: How to Carry Out an Analysis

Industrial Machine Learning

Data Scientists and Hadoop

Data Analyst

“If my tools and data can’t answer a question, then the question doesn’t get answered.”

Data Scientist

“If my tools and data can’t answer a question, then I go get better tools and data.”

Incredibly Common Question

“When should I use Hadoop instead of a relational database?”

The Unit of Analysis Problem: Three Symptoms

First Symptom: COUNT DISTINCT

Second Symptom: Cursors

Third Symptom: ALTER TABLE OF_DOOM

The Unit of Analysis Problem

• Data warehouses are optimized to analyze transactions• Awesome for finance

and ERP• Not ideal for product

and marketing

• A function of what databases are good at

What Are You Trying to Analyze?

Simple Entities• Static attributes• Flat data structure• Transient• Examples

• SKUs• Line items from an invoice• Log messages

Complex Entities• Evolving attributes• Hierarchical data structure• Persistent• Examples

• Customers• Suppliers• Website visitors

Choosing Our Own Data Format

• We get to structure our data in the way that works best for the problem we are solving• Flexible• Evolvable• Compact• Fast

serialization/deserialization

Spell Correction: The Drosophila of Data Science

Simple Counts on Complex Objects

The Uncanny Valley for Statisticians on Hadoop

The Business of Data Science

Where You Should Work: The Two Options

A Startup

Close to the Money

Dealing for Data

Education and Growth

Questions?@josh_wills

airbnb tech talk: josh wills on the life of a data scientist

Education

machine learning and hadoop present and future josh wills...

wills point texas, wills point edc

josh wills, mlconf 2013

airbnb adcopy

what is airbnb

airbnb search architecture: presented by maxim charkov,...

"the airbnb story" by riley newman, former director of data...

overview of the airbnb community in denmark - airbnb...

josh wills, director of data science, cloudera at mlconf sea...

inside airbnb · 2019. 10. 14. · inside airbnb: the face...

airbnb slides

inside airbnb - brooklyn...

1 © cloudera, inc. all rights reserved. engines,...

sf's airbnb legislation

wills—holographic wills and testamentary...

airbnb€¦ · airbnb and resy team up to offer in-app...

airbnb final

airbnb ppt

classifying what destinations airbnb users...

inside...