Download - Building a successful business after launch
![Page 1: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/1.jpg)
Building a successful business after launch through rapid iteration
Brett G. Durrett (@bdurrett)VP, Engineering & Operations
IMVU, Inc.
![Page 2: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/2.jpg)
Building a successful business afterlaunch through rapid iteration
Tweet this talk!
Brett G. Durrett (@bdurrett)
VP, Engineering & Operations, IMVU, Inc.
GDC Online, Austin, TX - October 7, 2010
#imvugdc
![Page 3: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/3.jpg)
2
About Brett
• IMVU
– Social entertainment
– 2005 – present
– VP Engineering & Operations
• There.com
– Virtual world, RIP 2010
– Various executive roles, 1999-2004
• Asylum Entertainment
– Game developer
– 1992-1999… think Gameboy -> Playstation era
– CEO
![Page 4: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/4.jpg)
3
![Page 5: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/5.jpg)
4
![Page 6: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/6.jpg)
5
![Page 7: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/7.jpg)
6
![Page 8: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/8.jpg)
7
![Page 9: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/9.jpg)
8
![Page 10: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/10.jpg)
9
![Page 11: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/11.jpg)
10
![Page 12: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/12.jpg)
11
IMVU’s Big Product Launch!
![Page 13: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/13.jpg)
12
IMVU’s Big Product Launch!
Didn’t Happen That Way
Most of our product was built
after launch
![Page 14: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/14.jpg)
13
![Page 15: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/15.jpg)
14
![Page 16: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/16.jpg)
15
Goals for this Session
• Describe IMVU’s development systems
• Provide examples that
– Generate ideas
– Inspire you to develop faster and better
– Help you get started
• Convince you that this is possible at your company
![Page 17: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/17.jpg)
16
Rapid Iteration
Big Project
![Page 18: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/18.jpg)
17
Big Project
Need solid understanding of your destination, exactly
how to get there
Hard to change course once the mass is in motion
Plan, check plan, re-plan, review plan, execute plan
Big risk, big rewards!
You probably get one shot
![Page 19: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/19.jpg)
18
Rapid Iteration
React to the road ahead, obstacles unknown
Frequently adjust and fix (pit stops)
Smaller risk, incremental rewards
Ability to change course, or destination
![Page 20: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/20.jpg)
19
IMVU went with rapid iteration
![Page 21: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/21.jpg)
20Actually… we started with something like this
![Page 22: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/22.jpg)
21
Iteration Cycle
Build
MeasureLearn
![Page 23: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/23.jpg)
22
Iteration Cycle
Build
MeasureLearn
![Page 24: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/24.jpg)
23
Guidance for Making “Build” Rapid
1. Customer-facing changes can be made at any time
![Page 25: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/25.jpg)
24
Build Today
Engineer commits code
20 minutes later it is live in production
Repeat about 50 times per day
![Page 26: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/26.jpg)
25
Does This Really Work?
“Maybe this is just viable for a single developer … your site will be down. A lot.”
“It seems like the author either has no customers or very understanding customers”
Responses to February 2009 posting about Continuous Deployment at IMVU
(at the time IMVU had a $12 million run rate)
![Page 27: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/27.jpg)
All tests
pass?
Local tests
pass, engineer
commits code
Lots and lots of
tests run
Code deployed
to all servers
Metrics
good?
Code deployed
to % of servers
Metrics
still
good?
Rollback
(Blocks)
Revert commit
(Blocks)
No
Yes
No
Yes
No
Yes
Win!
![Page 28: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/28.jpg)
27
Benefits of Continuous Deployment
• Rapid iteration using real customer metrics
• Regressions easy to find, correct
• Releases have zero overhead
• Engineers deploy code first day on job!
#imvugdc
![Page 29: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/29.jpg)
28
The Cluster Immune System
• Actively watches key metrics
– Business: revenue, registration, logins
– System: loads, database connections
• Two phase
– Initial roll-out to ~5% hosts
– Complete roll-out to 100% host
• Roll-back is automatic if metrics suffer
![Page 30: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/30.jpg)
29
Finding and Fixing Problems
• Each release has few changes, 1-3 commits
• Production issues correlate with check-in timestamp
• No overhead to producing a new release to correct issue
Identifying cause
takes minutes
![Page 31: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/31.jpg)
30
Schema Changes
Problems
• Difficult to roll-back schema
• Alter statements lock / impact customers
Solutions
• New schema has formal review process
• No alter on large tables, create new table
– Copy on read
– Complete migration with background job
• NoSQL back-end
![Page 32: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/32.jpg)
31
New Features
• Developed on trunk, not branch
– “hidden” from customers by A/B experiment
– 100% control, add QA to experiment
• Deployed daily during development
• Slow roll-out by increasing experiment %
– Experiment closed = fully launched
![Page 33: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/33.jpg)
32
Challenges with Continuous Deployment
• Won’t catch issues that fail slowly– SELECT * FROM growing_table WHERE 1
• Some critical areas cause hard lock-ups
– MySQL
– Memcached
• Lack of test coverage of older code
– Not an issue if you start with test coverage
• Outsourcing
![Page 34: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/34.jpg)
33
The Beginning of Build at IMVU
1. Check-in to CVS repository
2. Occasionally rsync to live server (manual)
3. Pray for the best
…and there was no automated test coverage
#imvugdc
![Page 35: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/35.jpg)
34
Getting Started – Extreme Basics
1. Continuous integration system
2. Production monitoring and alerting
– System performance
– Business metrics
– Trending is nice too
3. Simple deploy / roll-back system
![Page 36: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/36.jpg)
35
Commit to Making Forward Progress
• Require coverage for all new code
• Add coverage for failures / regressions
• Understand and fix root cause of failures
– Failures in production
– Intermittent test failures
![Page 37: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/37.jpg)
36
Expect Some Hurdles
• Production failures
• Different overhead
– Tests
– Build systems
• Production failures
• Frustration
• Process changes
• Production failures
![Page 38: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/38.jpg)
37
Iteration Cycle
Build
MeasureLearn
![Page 39: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/39.jpg)
38
Guidance for Making “Measure” Rapid
1. Make it simple to add instrumentation
2. Aim for real-time measurements of all business-critical data
![Page 40: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/40.jpg)
39
“Measure” Today
• Split-test experiment system
– Multiple branches
– Roll-out system
• Real-time business metrics
– Monitoring
– Alerting
– Trending
• User tests
![Page 41: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/41.jpg)
40
Value of Experiments
• Freedom to try new things
• Validates hypothesis (or not)
• Identifies surprises (big ones)
• Identifying anomalies
![Page 42: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/42.jpg)
41
Simple Experiment System
• Define experiment in code, branch
• Web interface for adjusting parameters
• Ability to compare metrics vs. control
• Make it easy to do this! Encourage use!
![Page 43: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/43.jpg)
42
Easy Experiment Setup$experiment = array(
‘name’ -> ‘DemoGDC’,
'initial_rollout' => '0',
'branches' => array(
'Enhanced' => 0.5,
// can specify multiple weighted branches
)
);
// Helper function for readability
function user_should_see_test_feature( $uid = NULL ) {
return get_branch_for_customer( $uid,
$experiment ) ) == 'Enhanced';
// assigns customer if first time called
}
// The experiment branch
if( user_should_see_test_feature( $uid ) ) {
// behavior for experiment
} else {
// control behavior
}
![Page 44: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/44.jpg)
43
Experiment Admin Interface
![Page 45: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/45.jpg)
44
User Testing
You are not your user!
• Don’t rely on summarized data
– Have the whole team watch
• Users better at expressing pain than telling you what you should build
• Possible with cheap equipment / no overhead
![Page 46: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/46.jpg)
45
Monitoring, Alerting, Trending
• System metrics
– Loads
– Response times
• Business metrics
– Revenue
– Key customer behavior
![Page 47: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/47.jpg)
46
System Trending
![Page 48: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/48.jpg)
47
Application Trending
![Page 49: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/49.jpg)
48
System Alerting
![Page 50: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/50.jpg)
49
Business Metric Alerting
![Page 51: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/51.jpg)
50
Predictive Monitoring
• Sounds great!
– But we can’t make it work
• Problems with variation in business
• False alerts lead to ignored alerts
![Page 52: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/52.jpg)
51
The Beginning of Measure at IMVU
1. Simple A/B experiment system
2. No cluster monitoring
3. Manual SQL queries
![Page 53: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/53.jpg)
52
Getting Started with Measure
1. Enable experiments
2. Deploy a packages for trending & alerting
3. Add a simple counter system (memcached)
– increment / set key functions for your app
– simple template for monitoring to read value
4. Start counting something (revenue is good)
#imvugdc
![Page 54: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/54.jpg)
53
Monitoring and Alerting Options
• Nagios – alerting, scales well, state is an issue
• Cacti – trending, simple to start, difficult to scale
• Ganglia – trending, scales well, work to centralize
• Zabbix – trending and alerting
• Munin – trending
![Page 55: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/55.jpg)
54
Consider Monitoring
• Basic system (load, disk, availability)
• Customer lifetime funnel
• Key business metrics
• As you learn, anything that impacts these
We are constantly adding new monitoring
![Page 56: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/56.jpg)
55
Iteration Cycle
Build
MeasureLearn
![Page 57: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/57.jpg)
56
Guidance for Making “Learn” Rapid
1. Regularly review data from “measure”
2. Make process change part of the process
3. Share data widely as possible
![Page 58: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/58.jpg)
57
“Learn” Today
• Data frequently reviewed
– Postmortem failures
– Scrum retrospectives
– Weekly business review
– Experiment results
• Iterate on processes after every sprint
• Data shared with all employees
![Page 59: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/59.jpg)
58
Experiment System Output
#imvugdc
![Page 60: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/60.jpg)
59
ROI Analysis
• Experiments simplify understanding ‘R’
– Measure revenue per user in experiment
• Scrum simplifies understanding ‘I’
– Track “actual time” on each task
![Page 61: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/61.jpg)
60
Beware Data!
• Do you understand what was measured?
• Do the results make sense?
• Have you reached a local maximum?
• Is your view too narrowly focused?
• Is it statistically significant?
#imvugdc
![Page 62: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/62.jpg)
61
Statistical Significance Made Simple
1. Define N as “the number of trials.”
2. Define D as “half the difference between the 'winner' and the 'loser'.”
3. The test result is statistically significant if D2 is bigger than N.
This is a great blog post: http://blog.asmartbear.com/easy-statistics-for-adwords-ab-testing-and-hamsters.html
T1 T2 T3
Control 8 32 62
Experiment 4 19 40
N 12 51 102
D 2 6.5 11
D2 4 42.25 121
Significant? No No Yes
Requires 50/50 split between control and
experiment. Provides 95% confidence interval
using a Chi-squared test
![Page 63: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/63.jpg)
62
Review and Revise
• Regularly review your operating data
– Business performance
– Experiment results
– Retrospectives for process
• Change processes based on learning
– Stop doing, Start doing
![Page 64: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/64.jpg)
63
Now What, Mr. Rapid Iteration?
![Page 65: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/65.jpg)
64
Failure is a Learning Opportunity
• Postmortem undesired results
– Service failure
– Business failure
– Organizational problems
• Produce action items that prevent recurrence
– And actually do them
![Page 66: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/66.jpg)
65
Toyota “5 Why” Method
• Get to root cause, not just symptom
• Ask “why” 5 levels deep for each issue
• Identify corrective measures for each cause
– Respond with appropriate level of investment
![Page 67: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/67.jpg)
66
Example 5 Why
Y1: Why were logins failing
A1: The master database has too many connections
Y2: Why did the master have too many connections?
A2: A new feature was using slow-running queries
Y3: Why were there slow queries?
A3: They were implemented with a cache time of 0
Y4: Why would queries have a cache time of 0?
A4: Ned, the new engineer, did not know that slow queries on the
master have to be cached
Y5: Why didn’t Ned know about caching slow queries?
A5: Well, he’s new and we don’t discuss it in boot camp and he cut and
pasted from another query as an example
![Page 68: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/68.jpg)
67
Transparency
• Share data!
– More informed decisions
– Empowered staff
• Share the good and the bad
• Don’t penalize failures
– People avoid sharing data if they are penalized
– Failure is an opportunity to learn & improve
#imvugdc
![Page 69: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/69.jpg)
68
The Beginning of “Learn” at IMVU
1. A/B testing (no roll-out)
2. Extremely basic reporting on funnel
3. Shared in daily standup and by e-mail
![Page 70: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/70.jpg)
69
Making Iteration Rapid
• Make it easy
• Measure!
– Look for anything that slows the iteration cycle
• Incremental improvements
– Baby steps on your way to rapid iteration
– Pick one thing to improve
![Page 71: Building a successful business after launch](https://reader034.vdocument.in/reader034/viewer/2022042515/5553452fb4c9059e688b47c1/html5/thumbnails/71.jpg)
70
Thank You! Any Questions?
Brett G. Durrett
Twitter: @bdurrett
Tweet this talk!
IMVU recognized as:
Inc. 500
http://bit.ly/dv52wK
Red Herring 100:
http://bit.ly/bbz5Ex
Best Place to Work:
http://bit.ly/aAVdp8
(and we're hiring)
http://www.imvu.com/jobs#imvugdc