[2c6]everyplay_big_data
DESCRIPTION
[2C6]Everyplay_Big_DataTRANSCRIPT
![Page 1: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/1.jpg)
![Page 2: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/2.jpg)
Tuomas Rinta, Development Director Everyplay / Unity Technologies
FROM BIG DATA TO ACTIONABLE ANALYTICS
![Page 3: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/3.jpg)
So what is ?
![Page 4: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/4.jpg)
![Page 5: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/5.jpg)
and numbers
• Live in about 1000 games across iOS and Android • Nearly 100 million game sessions recorded daily • About 2 billion events of usage data generated every week
![Page 6: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/6.jpg)
Why do we care about big data?
• Mobile games, especially free-‐to-‐play, live and die by their metrics
• Providing a service for game developers must have proven value, and each opFmizaFon counts
![Page 7: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/7.jpg)
So let’s talk about how we use big data, and how we got
started
![Page 8: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/8.jpg)
Our goal “How do we create a metrics-‐driven product based on big data?”
![Page 9: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/9.jpg)
This needs to be as quick as possible
Collect data
Analyze Create
A/B tests
Improve product
![Page 10: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/10.jpg)
Challenges • We ship an SDK – and normal update cycle by clients can be as long as 6-‐12 months, not very dynamic – This conflicts with the fast improvement cycle – Technology must adapt to supporFng big data
• The product evolves constantly – AnalyFcs requirements change constantly
![Page 11: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/11.jpg)
SDK is instrumented to send everything
the user does to the servers
Scribe
Amazon S3
Real-tim
e production system
Batch data processing
Apache Pig
![Page 12: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/12.jpg)
Tackling evolving analy;cs
![Page 13: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/13.jpg)
Issues with big data and analyFcs • AnalyFcs requirements change • RedshiS is based on PostgreSQL, so there needs to be a scheme – Schemes are the most restricFve factor with RedshiS
• How does that work with evolving analyFcs? • Everything would be easy if there weren’t billions of rows of data…
![Page 14: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/14.jpg)
How should data be reported? • Choosing how the end-‐user instrumentaFon sends events is crucial
• Bad format of events can make analyFcs from big data nearly impossible
• You don’t always know before-‐hand what you need
![Page 15: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/15.jpg)
Two possible approaches Separate events Example of video sharing: openVideoEditor trimButtonPressed undoTrimPressed activateFacecamRecording finishFacecamRecording shareButtonPressed • More flexible with a schema-
based database • Requires much more
data processing • Combining events can be
a hassle
Conversions with properties Example of video sharing: {event: “videoShareComplete”, {properties: [ {didTrimVideo: true}, {isVideoTrimmed: false}, {didUseFacecam: true}, {isFacecamEnabled: true}, {totalDuration: 1241} ] } } • Problematic with a schema-
based database • Easier and faster to process • All relevant data is pre-
aggregated
![Page 16: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/16.jpg)
“What about Postgre and JSON?” • Yes, Postgre allows parsing of JSON documents which allows arbitrary format of event data
• However, when your data gets big, this comes with a warning…
![Page 17: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/17.jpg)
Comparing querying fields and JSON Normal query: select count(*) from events where created > ‘2014-09-01’ and event_type=‘recordSessionClosed’; Vs. JSON-‐based: select count(*) from events where created > ‘2014-09-01’ and json_extract_path_text(event_json, ‘event_type’) = ‘recordSessionClosed’
![Page 18: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/18.jpg)
Results
0
200
400
600
800
1000
1200
1400
Normal JSON
Execution time (in seconds)
![Page 19: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/19.jpg)
So what’s the best soluFon? • Combining single-‐event sending with extra JSON-‐ properFes
• Querying the JSON-‐properFes is slow, so we store only informaFon that is not needed that much there (drill-‐down informaFon)
![Page 20: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/20.jpg)
How do we then analyse the data? • Most on-‐the-‐market soluFons fell short due to
– Pricing – Features – Availability
• Turned out to be easier to “roll your own”
![Page 21: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/21.jpg)
Solving an actual problem “What are the worst drop-‐off points for uploading a replay?”
![Page 22: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/22.jpg)
Tools • SQL • JavaScript • Google Charts visualisaFon library
![Page 23: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/23.jpg)
![Page 24: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/24.jpg)
Why JavaScript for processing? • Dynamic, fast, relaFvely well-‐known • Excellent libraries for data visualisaFon
– Highcharts, Google Charts, D3.js, Dygraph • Good for visualizing data, but that’s it
![Page 25: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/25.jpg)
Keys to a successful data-‐driven product • Plan ahead for analyFcs and leave room for an evolving product
• If metrics and analyFcs are not easily accessible by decision makers, they are worthless – self-‐updaFng dashboards are one of the main keys to success
• Build A/B tesFng and data-‐driven behaviour directly into your product, don’t hack it on later
![Page 26: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/26.jpg)
Thank you! Questions, comments? Email: [email protected] Twitter: @trinta developers.everyplay.com
![Page 27: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/27.jpg)
Q&A
![Page 28: [2C6]Everyplay_Big_Data](https://reader034.vdocument.in/reader034/viewer/2022051817/547de507b4af9fce158b54a6/html5/thumbnails/28.jpg)
THANK YOU