continuous innovation

Chris Dolezalek – October, 2012

Con8nuous Innova8on In a Rapidly Evolving Market Experimen*ng and being agile with products, processes, …



The path to enlightenment lies not in the footsteps of another

What is the one mantra to follow

religiously?

Follow no mantra religiously

初心 Beginner’s Mind


The Pivot That Elusive Hockey S*ck

BroadVision, Inc. pivots to become fastest growing soDware company on Nasdaq

SoDlab, GmbH pivots to become Germany’s 2nd fastest growing independent soDware company

PowerSoD developed PowerBuilder in 1991, went public in 1993 and was acquired for ~$900 million in 1995

hRp://www.sunarsports.com/products/Easton-‐Stealth-‐S19-‐Sr.-‐Composite-‐Hockey-‐S*ck

1. The Ten Faces of Innova*on by Thomas Kelley & Jonathan LiRman hRps://itunes.apple.com/us/book/the-‐ten-‐faces-‐of-‐innova*on/id421032598?mt=11

Google $0 to $37.9B in 10 years ”Innova(ng at a breakneck pace; rolling out a new service capability prac(cally every month1.”


Hire for Tomorrow not Today • The top 10 in-‐demand jobs in 2010

did not exist in 2004 “Did You Know? Shift Happens (August 2011)“ http://www.youtube.com/watch?v=EJpSqeQbh4o

Skills vs Curiosity & Passion • "CQ + PQ > IQ”

Curiosity & Passion Quo:ents are more important than Intelligence Quo:ent Thomas Friedman, The World is Flat

APtude MaQers • “It is not their ap:tude

but their aFtude that will determine their al:tude.” Jesse Jackson, Washington Post 21 May 1978

Lean Staffing Hiring Your Most Valuable Resources


Don’t stop at Hiring

Invest in spinning up At IMVU, we dedicate a mentor to every new engineer that will sit next to them and spend 6 weeks spinning them up according to a boot camp guide -‐ including a day-‐one push of a change to produc*on. Spinning Marathon voor WAR CHILD -‐ Some rights reserved By isafmedia (cropped)


What makes for the most produc*ve & innova*ve employee?

Happiness = Passion, Engagement

• Support your team Perspec8ve: Managers work for the engineers to enable them to be produc8ve

• Sell vs Tell = No show, No go Winning the hearts and minds bring engagement as opposed to bodies in seats. 5-‐Whys: Company Direc8ves Alignment = What are you working on and Why is it important?

• Be Transparent & Invite Challenges Share business objec8ves and reward those who ques8ons them

• Let them pick their team & project LiRle startups/laboratories

• Let them Hack Hack weeks once a quarter

• Let them experiment • Let them choose their tools • Let them determine when done is

Story Point Sizing

• Keep teams small, agile and fun (BVSN 5 man teams vs whole companies)

Manage Your Most Valuable Resources


• Run Experiments

• Minimal Viable Products

• Con*nuous Deployment

• Smart Test Coverage

• Use an Immune System

• Post Mortems, & Retrospec*ves & 5 Whys


Con8nuous Innova8on Validate Hypotheses with Data

Time: Experiment A: up ~6% over 2 weeks prior

Num

ber of uses of new

feature


Time: Subgroup A1: up ~48% over 2 weeks prior Subgroup A2: down ~68% over 2 weeks prior

Con8nuous Innova8on Validate Hypotheses with the Right Data

Num

ber of uses of new

feature


IMVU con*nually runs process experiments as a way to improve our processes and to avoid falling into dysfunc*onal rou*nes.

Experiment as individuals but also as teams

5 Teams 5 Different Processes


Minimal Viable Products

• Different Kinds of MVPs – A heavily used system (IMVU’s Landing Pages Millions of monthly hits)

– New Feature Sets (inventory management into exis*ng product)

– A brand new subsystem (a new game with a new game/fun loop)

– A paradigm shiD (an iPhone App)

hRp://www.rise.net/blog/minimum-‐viable-‐product


How ofen do you deploy?

• Waterfall Development : every 6-‐12 months

• Agile Methodology: every 1-‐3 weeks

• Con8nuous Deployment: every day

• IMVU every 7 Minutes

Deploy, Iterate & Learn at New Speeds Production releases


Test-‐Driven Development Where/How to start?

Test Driven Development hRp://www.flickr.com/photos/brunobord/3987593006/lightbox Some rights reserved by brunobord


Immune Systems Small Change-‐Sets = Easy Debugging


5 Whys Turning Losses into Wins

1. Why did the server grind to a halt? DB query took forever.

2. Why did the query take so long? Un-‐indexed query on large table.

3. Why didn’t the engineer realize the query would be slow? Local tests ran fine against a small test db.

4. Why didn’t the engineer know to run a “try” against produc:on? It was his first database change as a new hire.

5. Why did his mentor not tell him? The mentor had leD for the day.

6. Why did the new hire not know to only check-‐in with mentor? This was the first mentee for that mentor and he hadn’t told him.

Follow-‐Up 1) Update Spin-‐Up Doc for Mentor and Mentee to start with a clear

statement that mentee should not check into produc*on without mentor or without having reached that stage.

Follow-‐Up 2) Add notes to DB Query sec*on of spin-‐up doc on use of “try” Post-‐Mortems = Teachable Moments ; Fix what needs fixing

Ques*on mark sign photo by Colin_K on Flickr


Requirement Opacity

Granularity versus deadlines: It’s ok not to have the nth degree of detail of what you’re building in 6 months’ 8me.


Con8nuous Deployment Flow

hRp://www.slideshare.net/bgdurreR/con*nuous-‐deployment-‐at-‐lean-‐la

Hypothesis Build on VMs


Experiments & Alterna8ves

• Experiments: Vo*ng by doing, requires func*onality

• User Sessions: Hard to get good sample size quickly

• Customer Advocates: Par*cipates in forums, user groups, …

• Anthropologist: Do field research to observe interac*ons

• Surveys: What people say rather than do

• Dog-‐fooding: Use your product

• Mavens: Good insights into established market

hRp://www.gladwell.com/*ppingpoint/index.html


An Enhancement / Story is Born

• Project Brief Goes to a Exec Checkpoint – User stories – Success criteria by date x… (engagement, revenue, …)

• Preview Mee*ng by Product Owner – To: Tech Lead, QA Lead, Data Analyst

• Tech Design, Test Plan, Data Design

• Planning Mee*ng – Story point sizing – High and low es*mate jus*fica*on – Risk buckets for stories vs risk included in task es*mates


Delivering Innova*on to Risk Averse, Enterprise Customers

• In 1985, I came to Edinburgh from Munich to help sell the Bank of Scotland on transi*oning to our new product that resulted from our second pivot at SoDlab.

• What unfolded was not quite according to script.

• They were not at all excited about adop*ng innova*on, having in 8 previous systems suppor*ng 180 of their IT staff. But, they did…


Apply Beginner’s Mind and Con8nuously Innovate…

Your Product Direc*on, Your Company Direc*on,

Your Development Processes, Your Resource Management, Your Marke*ng Strategy, Your Technology Choices, Your Analy*cs Approaches,

Understanding Your Customers & Prospects, .. Everything You Do and Every Approach You Take.

Chris Dolezalek

初心 Beginner’s Mind

Chris Dolezalek – October, 2012 Chris Dolezalek – October, 2012 22

Thank You!

Chris Dolezalek


References & Other Links • Eric Ries, IMVU, former CTO of Engineering -‐ The Lean Startup

How Today's Entrepreneurs Use Con*nuous Innova*on to Create Radically Successful Businesses hRp://www.amazon.com/Lean-‐Startup-‐Entrepreneurs-‐Con*nuous-‐Innova*on/dp/0307887898/ref=sr_1_1?ie=UTF8&qid=1318352510&sr=8-‐1 hRp://itunes.apple.com/us/book/the-‐lean-‐startup/id422540072?mt=11

• BreQ DurreQ, IMVU, CEO – Con8nuous Deployment… Con8nuous Deployment: Possibility or Pipe Dream? hRp://bos*nnova*on.com/2011/11/21/con*nuous-‐deployment-‐possibility-‐or-‐pipe-‐dream Con8nuous Deployment & Learning Fast with A/B Tes8ng hRp://www.slideshare.net/bgdurreR/con*nuous-‐deployment-‐at-‐lean-‐la hRp://www.slideshare.net/bgdurreR/learning-‐fast-‐with-‐ab-‐tes*ng-‐and-‐con*nuous-‐deployment hRp://www.mediabistro.com/The-‐Challenges-‐of-‐Con*nuous-‐Deployment-‐Social-‐Developer-‐Summit-‐467-‐ondemandvideo.html

• James Birchler, IMVU, VP Engineering Learning from Experiments hRp://www.slideshare.net/jamesbirchler/learning-‐from-‐experiments-‐at-‐imvu

• Flickr, 10+ Deploys Per Day: Dev and Ops Coopera*on at Flickr hRp://www.slideshare.net/jallspaw/10-‐deploys-‐per-‐day-‐dev-‐and-‐ops-‐coopera*on-‐at-‐flickr

• Carol Dweck, Mindset: The New Psychology of Success hRp://www.amazon.com/Mindset-‐Psychology-‐Success-‐Carol-‐Dweck/dp/0345472322/ref=sr_1_1?s=books&ie=UTF8&qid=1318359458&sr=1-‐1 hRp://itunes.apple.com/us/book/mindset/id422549774?mt=11 hRp://www.youtube.com/watch?v=XHW9l_sCEyU

• Did You Know (a.k.a ShiD Happens) Blog Post, Slides, Videos hRp://www.lps.k12.co.us/schools/arapahoe/fisch/fischbowlpresenta*ons.htm hRp://shiDhappens.wikispaces.com

• Paul Stoltz, Peak Learning hRp://www.peaklearning.com

– Erik Weihenmayer & Paul Stoltz, The Adversity Advantage hRp://itunes.apple.com/us/book/the-‐adversity-‐advantage/id381515439?mt=11 hRp://www.amazon.com/Adversity-‐Advantage-‐Everyday-‐Struggles-‐Greatness/dp/1439199493/ref=sr_1_1?s=books&ie=UTF8&qid=1318360603&sr=1-‐1

– Paul Stoltz, Adversity Quo*ent at Work hRp://itunes.apple.com/ca/book/adversity-‐quo*ent-‐work/id385756581?mt=11 hRp://www.amazon.com/Adversity-‐Quo*ent-‐Work-‐Finding-‐Capacity/dp/0060937211/ref=sr_1_sc_1?s=books&ie=UTF8&qid=1318360551&sr=1-‐1-‐spell

• Michael Jordan, "Failure" Nike Commercial hRp://www.youtube.com/watch?v=45mMioJ5szc

• Malcolm Gladwell, Outliers & Tipping Point hRp://www.gladwell.com/outliers/index.html hRp://itunes.apple.com/us/book/outliers/id357396748?mt=11 hRp://www.gladwell.com/*ppingpoint/index.html hRp://itunes.apple.com/us/book/the-‐*pping-‐point/id357658331?mt=11

• Is Con8nuous Innova8on Too Risky? hRp://www.forbes.com/sites/stevedenning/2012/02/10/is-‐radical-‐management-‐too-‐risky


References on Experimen8ng

• Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, Ya Xu http://www.exp-platform.com/Documents/puzzlingOutcomesInControlledExperiments.pdf

• Puzzling outcomes in A/B testing - Greg Linden http://glinden.blogspot.co.uk/2012/07/puzzling-outcomes-in-ab-testing.html

• Microsoft EXP - Experimentation Platform - Ronny Kohavi Accelerating software innovation through trustworthy experimentation http://www.exp-platform.com/Pages/default.aspx

• What types of things does Netflix A/B test aside from member sign-up? http://www.quora.com/What-types-of-things-does-Netflix-A-B-test-aside-from-member-sign-up

• From Zero to a Million Users - Dropbox and Xobni lessons learned - Adam Smith http://www.slideshare.net/adamsmith1/from-zero-to-a-million-users-dropbox-and-xobni-lessons-learned

• Data-‐Driven Startups -‐ July 23, 2010 http://davidcancel.com/data-driven-startups

• Analy8cs Maturity hRp://www.forbes.com/sites/piyankajain/2012/06/22/what-‐is-‐your-‐organiza*ons-‐analy*cs-‐maturity/


Appendix, Drill Down Slides


"It is one of the happy incidents of the federal system, that a single courageous state may, if its citizens choose, serve as a laboratory; and try novel social and economic experiments without risk to the rest of the country.”

-‐ Louis D. Brandeis, 1932 Supreme Court Jus*ce

IMVU con*nually runs process experiments as a way to improve our processes and to avoid falling into dysfunc*onal rou*nes.

No Meeting Mondays

…

No Meeting Mondays

Prototype, Test, Refactor

Defined Milestones & Iteration

Experiment as individuals but also as teams

Interrupt Lanes/Engineers

Risk Buckets vs Risk factored into

task estimates


Zynga Numbers • Zynga’s daily highs of concurrent players is equivalent to

everyone in the city of Paris playing together at the same *me. That happens every day. The games are always changing and the devices they’re played on change constantly. Zynga releases over 100 updates across all its games every day. The company once released more than 1000 updates in one week. Zynga players make a million ac*ons per second.

• Zynga runs about 130 experiments in its games every day. ADer releasing a new feature Zynga can find out within minutes if players enjoyed them. Beyond fun is social — what maRer is that friends and family con*nue to play. This is a player’s Ac*ve Social Network (ASN). It’s a true barometer of how social a game is.

hRp://www.insidesocialgames.com/2012/06/26/live-‐from-‐zynga-‐unleashed


Flickr’s Perspec*ve

hQp://www.slideshare.net/jallspaw/10-‐deploys-‐per-‐day-‐dev-‐and-‐ops-‐coopera8on-‐at-‐flickr


What’s Under Your Hood? Improving, replacing and dele*ng old code can vital, but when and how?

hRp://www.commercialmotor.com/big-‐lorry-‐blog/now-‐thats-‐more-‐like-‐it-‐for-‐biglorryblog-‐one-‐horsepower-‐truck-‐exclusive-‐picture


VALVE SoDware’s Game Design Process

hRp://www.valvesoDware.com/publica*ons/2009/GDC2009_ValvesApproachToPlaytes*ng.pdf hRp://www.geekwire.com/2011/experiments-‐video-‐game-‐economics-‐valves-‐gabe-‐newell

Valve is the company that created the Steam Game Distribu*on system.


Con*nuous Improvement -‐ Iterate

1. Release Minimal Viable Products (MVPs), 2. Get Customer Feedback / Collect Metrics (don’t presume you know what’ll work), 3. Analyze Changes, 4. Make Itera*ve Changes 5. Go back to Step 2

hRp://buu700.com/steverant : Also note that he men*ons a "ship early and iterate" ethos at Amazon, and also how a clear order from the top was what successfully drove them into building SOA pla|orms despite the high ini*al cost.


Counter Innova8on

• How to make your offering less appealing to… – Fraudsters – Scammers – Spammers – Cyber-‐Bullies

Knowing your users may need to go beyond knowing the ones you want to keep.


Priori8za8on

When priori*zing, deciding what’s at the top of the list, though hard, oDen maRers less than what’s at the boRom…

1. iOS App

2. Android App

3. Infrastructure Upgrade

4. Enhance Adver*sing interface


On Le}ng Go

Shoo8ng the dogs (lePng go): • Loss aversion and the “sunk cost fallacy”

Many people have strong misgivings about "was*ng" resources (loss aversion). They may feel they've passed a point of no return. Economists would label this behavior "irra*onal": it is inefficient because it misallocates resources by depending on informa*on that is irrelevant to the decision being made. Colloquially, this is known as "throwing good money aDer bad“.

• Sunk Cost Dilemma The dilemma of having to choose between con*nuing a project of uncertain prospects already involving considerable sunk costs, or discon*nuing the project. Given this choice between the certain loss of the sunk costs when stopping the project versus possible – even if unlikely – long-‐term profitability when going on, policy makers tend to favor uncertain success over certain loss.

h<p://en.wikipedia.org/wiki/Sunk_costs#Loss_aversion_and_the_sunk_cost_fallacy h<p://en.wikipedia.org/wiki/Sunk_cost_dilemma


Mobile App Release verses Con8nuous Deployment

App Stores require extended *me to market + Mobile users expect higher fidelity apps

= Mismatch from con*nuous web-‐deployment Reaching back to trusted prac*ces:

Clearer Requirements Release Candidates

Stable Libraries Code Freezes

Merging Changes Rigorous Release Tes8ng

Bug Triages Release Sign-‐Offs

Release Reference Copies Risk-‐Awareness

Down8me Avoidance Server Isola8ons

continuous innovation

Technology

ves5whys chrisdolezalekoctober

dierent processes imvucon

ent thomasfriedman

weeks con8nuousdeployment

onality usersessions

pqiq curiositypassionquo

months agilemethodology

ck broadvision