Download - [@IndeedEng] Data-Driven Product Design
![Page 1: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/1.jpg)
Data-DrivenProduct Design
J Christopher GarciaChris Hyams
![Page 2: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/2.jpg)
DesignJ Christopher Garcia
User Experience Lead, Indeed
![Page 3: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/3.jpg)
![Page 4: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/4.jpg)
![Page 5: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/5.jpg)
![Page 6: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/6.jpg)
2008
![Page 7: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/7.jpg)
2013
![Page 8: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/8.jpg)
![Page 9: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/9.jpg)
![Page 10: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/10.jpg)
![Page 11: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/11.jpg)
114 tests244 variations
84,000 combinations
![Page 12: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/12.jpg)
![Page 13: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/13.jpg)
![Page 14: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/14.jpg)
![Page 15: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/15.jpg)
Direct observation can be inefficient
![Page 16: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/16.jpg)
Before and afterdoesn't work
![Page 17: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/17.jpg)
Traffic doesn't look like this
![Page 18: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/18.jpg)
Hourly traffic looks like this...
![Page 19: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/19.jpg)
...and daily traffic looks like this
![Page 20: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/20.jpg)
...and monthly traffic looks like this
![Page 21: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/21.jpg)
![Page 22: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/22.jpg)
![Page 23: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/23.jpg)
Local maximum
![Page 24: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/24.jpg)
![Page 25: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/25.jpg)
![Page 26: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/26.jpg)
![Page 27: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/27.jpg)
![Page 28: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/28.jpg)
![Page 29: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/29.jpg)
![Page 30: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/30.jpg)
![Page 31: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/31.jpg)
![Page 32: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/32.jpg)
What we've learnedUI Principles
![Page 33: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/33.jpg)
Clickable items should look clickable
![Page 34: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/34.jpg)
![Page 35: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/35.jpg)
![Page 36: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/36.jpg)
![Page 37: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/37.jpg)
![Page 38: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/38.jpg)
![Page 39: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/39.jpg)
Users read
![Page 40: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/40.jpg)
![Page 41: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/41.jpg)
![Page 42: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/42.jpg)
![Page 43: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/43.jpg)
![Page 44: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/44.jpg)
![Page 45: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/45.jpg)
![Page 46: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/46.jpg)
![Page 47: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/47.jpg)
a word is worth a thousand icons
![Page 48: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/48.jpg)
![Page 49: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/49.jpg)
![Page 50: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/50.jpg)
![Page 51: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/51.jpg)
![Page 52: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/52.jpg)
![Page 53: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/53.jpg)
![Page 54: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/54.jpg)
+14%more search results pages
![Page 55: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/55.jpg)
BEWARE MULTIPLE CHANGES
![Page 56: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/56.jpg)
![Page 57: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/57.jpg)
![Page 58: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/58.jpg)
![Page 59: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/59.jpg)
![Page 60: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/60.jpg)
![Page 61: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/61.jpg)
Searches: -2.0%Job Clicks: -1.7%Saved jobs: -7.6%
![Page 62: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/62.jpg)
![Page 63: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/63.jpg)
![Page 64: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/64.jpg)
![Page 65: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/65.jpg)
![Page 66: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/66.jpg)
![Page 67: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/67.jpg)
Searches: -1.8%Job Clicks: -0.8%Saved jobs: 0%
![Page 68: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/68.jpg)
![Page 69: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/69.jpg)
Really?
![Page 70: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/70.jpg)
Really.
![Page 71: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/71.jpg)
![Page 72: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/72.jpg)
![Page 73: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/73.jpg)
![Page 74: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/74.jpg)
![Page 75: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/75.jpg)
![Page 76: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/76.jpg)
![Page 77: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/77.jpg)
![Page 78: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/78.jpg)
![Page 79: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/79.jpg)
![Page 80: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/80.jpg)
![Page 81: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/81.jpg)
![Page 82: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/82.jpg)
![Page 83: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/83.jpg)
![Page 84: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/84.jpg)
ProductChris Hyams
SVP, Product & Engineering
![Page 85: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/85.jpg)
I help people get jobs.
![Page 86: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/86.jpg)
What's best for the job seeker?
![Page 87: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/87.jpg)
How do we know what's best for job seekers?
![Page 88: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/88.jpg)
Test and measure EVERYTHING
![Page 89: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/89.jpg)
Measurement
![Page 90: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/90.jpg)
1. Log everything
![Page 91: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/91.jpg)
jobsearch indexabredistimeacmetimeaddltimeadscadsdelayadsibadscbadsiboostojcboostojibsjcbsjcwiabsjibsjindappliesbsjindappviewsbsjrevbsjwiackcntckszcountsctkagectkagedaysdayofweekdcpingtimedomTotalTimeds-mpo
dsmissdstimefeatempfjfreekwacfreekwarevfreesjcfreesjrevfrmtimegalatdelayiplatiplongjslatdelayjsvdelaykwackwacdelaykwaikwarevkwcntlacinsizelacsgsizelmstimempotimemprtimenavTotTimendxtime
ojcojclongojcshortojcwiaojiojindappliesojindappviewsojwiaoocscpageprcvdlatencyprimfollowcntprvwojiprvwojlatprvwojopentimeprvwojreqradscradsirecidlookupbudgetrectimeredirCountredirTimerelfollowcntrespTimereturnvisitrojc
rojirqcntrqlcntrqqcntrrsjcrrsjirrsjrevrsavailrsjcrsjirsusedrsviableserpsizesjcsjcdelaysjclongsjcntsjcshortsjcwiasjisjindappliessjindappviewssjrevsjwiasllatsllong
sqcsqisugtimesvjsvjnostarsvjstartadsctadsitimetimeofdaytotcnttotfollowcnttotrevtottimetsjctsjcwiatsjitsjindappliestsjindappviewstsjrevtsjwiaunqcntvpwacinsizewacsgsize
![Page 92: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/92.jpg)
acmepageacmereviewmodacmeserviceacmesessionadclickadcrequestadcrevadschanneladsclickadsenseclickadveadvtagghttpaggjiraaggjobaggjob_waldorfaggsherlockaggsourcehealthagstimingapiapijsvapisearcharchiveindexarchiveindex_shingled_testbincarclicksclickclickanalyticscobranddctmismatchdrawdupepairsdupepairs_minidupepairs_olddupepairsalldupepairsall_miniejcheckeremilyops
feedbridgeglobalnavgooglebot_organichomepageimpressionindeedapplyjhstjobalertjobalertorganicjobalertsearchjobalertsponsoredjobexpirationjobexpiration2jobexpiration3jobprocessedjobqueueblockjobsearchjssquerykeywordAdlocsvclucyindexermainmechanicalturkmindyopsmobhomepagemobilmobilemobileorganicmobilesponsoredmobrecjobsmobsearchmobviewjobmyindeedmyindfunnelmyindpagemyindrezcreatemyindsessionoldopsesjasx
organicorgmodelorgmodelsubsetorgmodelsubset90passportaccountpassportpagepassportsigninramsaccessrecjobsrecommendserviceresumedataresumesearchrexcontactsrexfunnelreximpressionrexsearchrezSrchSearchrezalertrezalertfunnelrezfunnelrezjserrrezsrchrequestrezviewsearchablejobsseosessionsjmodelsponsoredsysadappinfosysadapptimingtestndxtestndx1testndx2tmpusrsvccacheusrsvcrequestviewjobwebusersignin
![Page 93: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/93.jpg)
16 million jobs100 million UV
3 billion searches
![Page 94: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/94.jpg)
1TB per day
![Page 95: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/95.jpg)
Everything you ever wanted to know about logging but were afraid to ask
Logrepo: Enabling Data-Driven Decisions
![Page 96: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/96.jpg)
2. Index data
![Page 97: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/97.jpg)
3. Build tools toquery indexes
![Page 98: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/98.jpg)
[another tech talk]
![Page 99: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/99.jpg)
Tools answer questions
![Page 100: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/100.jpg)
Mobile searches per hour inJP vs. UK?
![Page 101: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/101.jpg)
Resume creation by country?
![Page 102: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/102.jpg)
Impressions in job alerts byemail domain?
![Page 103: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/103.jpg)
Percent of app downloads fromiOS, Android, Windows?
![Page 104: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/104.jpg)
How quickly does a datacenter take on traffic after a failover?
![Page 105: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/105.jpg)
Test EVERYTHING
![Page 106: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/106.jpg)
TextVisual designLayoutControl flowNew featuresAlgorithms [also another tech talk]
Everything
![Page 107: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/107.jpg)
5% 5%grp:bluebtntst0 grp:bluebtntst1
split:
log:
![Page 108: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/108.jpg)
Measure
![Page 109: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/109.jpg)
statistical relevance
![Page 110: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/110.jpg)
p-value
![Page 111: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/111.jpg)
Big changes require less data
Conversion Improvement p-value Trials
10% +30% 0.038 1,000
source: http://go.indeed.com/jyusg
![Page 112: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/112.jpg)
Small changes require more data
Conversion Improvement p-value Trials
10% +30% 0.038 1,000
10% +1% 0.042 750,000
source: http://go.indeed.com/ommgd
![Page 113: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/113.jpg)
![Page 114: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/114.jpg)
Product Principles
![Page 115: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/115.jpg)
There's no such thing as a no-brainer
![Page 116: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/116.jpg)
![Page 117: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/117.jpg)
-30%
![Page 118: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/118.jpg)
Corollary: if a test fails,it means the test failed
![Page 119: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/119.jpg)
-30%
![Page 120: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/120.jpg)
![Page 121: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/121.jpg)
+25%
![Page 122: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/122.jpg)
Little things can makea big difference
![Page 123: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/123.jpg)
![Page 124: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/124.jpg)
organic
![Page 125: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/125.jpg)
sponsored
![Page 126: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/126.jpg)
+34%
![Page 127: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/127.jpg)
Corollary: be skepticalof big changes
![Page 128: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/128.jpg)
Does color really matter that much?
![Page 129: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/129.jpg)
a A
a B
a C +34%
+10%
![Page 130: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/130.jpg)
Are people just boredof blue?
![Page 131: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/131.jpg)
+29% +22%
![Page 132: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/132.jpg)
Will the effect wear off?
![Page 133: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/133.jpg)
![Page 134: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/134.jpg)
Get it right,then do it right
![Page 135: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/135.jpg)
![Page 136: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/136.jpg)
Location backfill: job clicks
+8%
![Page 137: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/137.jpg)
Location backfill: search time
+1,500%
![Page 138: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/138.jpg)
Location backfill: performance fix
+1,500%
![Page 139: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/139.jpg)
Location backfill: performance fix
+1,500% +0%
![Page 140: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/140.jpg)
Location backfill: job clicks
+8%
![Page 141: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/141.jpg)
Location backfill: clicks after fix
+8% +9%
![Page 142: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/142.jpg)
Measure everything(aka, Beware of unintended consequences)
![Page 143: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/143.jpg)
April Fools' 2010
![Page 144: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/144.jpg)
![Page 145: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/145.jpg)
![Page 146: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/146.jpg)
![Page 147: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/147.jpg)
Organic clicks
+1.8%
![Page 148: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/148.jpg)
Job alert signups
-17%
![Page 149: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/149.jpg)
Revenue
+15.5%
![Page 150: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/150.jpg)
Revenue
+15.5%
![Page 151: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/151.jpg)
![Page 152: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/152.jpg)
![Page 153: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/153.jpg)
![Page 154: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/154.jpg)
![Page 155: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/155.jpg)
Job alert signups: top v. bottom
-17% +6%-21%
![Page 156: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/156.jpg)
Corollary: attention is a zero-sum game
(usually)
![Page 157: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/157.jpg)
SearchClickRevenueEmail signupResume create...
Relative value of actions
![Page 158: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/158.jpg)
![Page 159: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/159.jpg)
Different for everyone
![Page 160: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/160.jpg)
Location auto-complete
![Page 161: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/161.jpg)
Zero-results page
![Page 162: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/162.jpg)
![Page 163: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/163.jpg)
![Page 164: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/164.jpg)
Zero-results pages
-2.7%
![Page 165: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/165.jpg)
Organic clicks
+0%
![Page 166: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/166.jpg)
Clicks w/ auto-complete
+8%
![Page 167: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/167.jpg)
Keyword ad revenue
+1,410%
![Page 168: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/168.jpg)
Keyword ad revenue
+1,410%
![Page 169: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/169.jpg)
keyword ads
![Page 170: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/170.jpg)
![Page 171: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/171.jpg)
![Page 172: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/172.jpg)
![Page 173: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/173.jpg)
![Page 174: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/174.jpg)
Keyword ad revenue after fix
![Page 175: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/175.jpg)
Measure everything,every time
![Page 176: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/176.jpg)
![Page 177: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/177.jpg)
Getting Started
![Page 178: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/178.jpg)
![Page 179: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/179.jpg)
![Page 180: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/180.jpg)
![Page 181: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/181.jpg)
Fewer tests
![Page 182: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/182.jpg)
Focus on big wins
![Page 183: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/183.jpg)
Conversion Improvement p-value Trials
10% +30% 0.038 1,000
10% +1% 0.042 750,000
source: http://go.indeed.com/ommgd
![Page 184: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/184.jpg)
Straw man
Determine # of trials in a weekDetermine minimum improvementOne new test each weekIf success, roll out; if not, kill & move on
![Page 185: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/185.jpg)
What We Get
![Page 186: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/186.jpg)
Quality
![Page 187: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/187.jpg)
Risk tolerance
![Page 188: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/188.jpg)
Culture of measurement
![Page 189: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/189.jpg)
(Empirically)better products
![Page 190: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/190.jpg)
fin
![Page 191: [@IndeedEng] Data-Driven Product Design](https://reader033.vdocument.in/reader033/viewer/2022042813/5457d513af795963388b45d2/html5/thumbnails/191.jpg)