Comparing and ContrastingComparing and ContrastingSoftware Engineering ResearchSoftware Engineering Research

in Industry and Academiain Industry and Academia

Christopher Scaffidi

Carnegie Mellon University

22

What unexamined assumptions will shapeWhat unexamined assumptions will shapeyour choice of a career?your choice of a career?

“In graduate school there was an implicit, not-very-clearly-stated assumption that the successful graduates go to become professors, and the failed ones go to industry.”

- CMU SCS PhD ’04 Mihai Budiu

(now happily working atMicrosoft Research)

[Bud07]

Introduction--- Disagree --- --- Agree ---

Do current SCS students agree with him?

33

Where do your peers choose to work?Where do your peers choose to work?

98 SE PhD graduates in 2007

Introduction

105 CSD graduates

May 2000-Oct 2005 [Pfe]Includes degrees characterized as SEin 235 CS+CE departments covered in the 2007 Taulbee Survey [Tau07]

41% of Year 2007 MIT PhDschose industry [MIT07]

44

OutlineOutline

1. Focus today = research, not development

2. Literature and statistics• Research traits

• Statistics on hiring & funding

3. Interviews of people in research in industry• Differences among companies, and vs academia

• Advice from industry researchers

Outline

55

““Research” vs “development” Research” vs “development” ≈≈ “making knowledge” vs “making products”“making knowledge” vs “making products”

“Software engineering research answers questions about methods of development or analysis, about details of designing or evaluating a particular instance, about generalizations over whole classes of systems or techniques, or about exploratory issues concerning existence or feasibility.”

– Mary Shaw [Sha03]

Focusing today on research

66

Technology maturation has several phases.Technology maturation has several phases.

Basic Research: undirected investigationof fundamental ideas

Concept formulation: generalization,circulation, and publication of ideas

Development and extension: preliminary trials of approach

Enhancement and exploration: stabilization and derivation

Popularization: commercialization

Basic Research

Concept formulation

Development and extension

Enhancement and exploration

Popularization

Research-intense

[Red85]

Focusing today on research

Development-intense

77

Industry research tends to be more Industry research tends to be more utilitarianutilitarian than academic research than academic research

• According to literature, industry and academia differ in– How they choose interesting questions

– What results they produce in response

– How they evaluate the “goodness” of results

(Not a hard-and-fast rule, of course.)

(The rule is more true for tool-focused research than other research.)

(And some people work for industry and academia.)

This analysis structure is based on [Dyb05] [New94] [Sha03] [Sha03i]

Relative utilitarianism

88

Industrial questions: need-driven.Industrial questions: need-driven.Academic questions: knowledge-driven.Academic questions: knowledge-driven.Industrial QuestionsMethod of development

“How can we automate X?”

Feasibility“Is it possible to do X at all?”

Selection: Need-driven

ExamplesReusable functions in Excel

Can we let users write functions using Excel syntax, rather than that nasty bit of VBScript? [Jon03]

Academic QuestionsOften the same on the surface!

Selection: Knowledge-driven

ExamplesForms/3

Can we give useful features (e.g.: abstraction, graphics, animation) without breaking the “first-order declarative evaluation model”? [Bur01]

Relative utilitarianism

99

Industrial results: use-oriented.Industrial results: use-oriented.Academic results: truth-oriented.Academic results: truth-oriented.

Industrial ResultsTool prototype

“Tool embodies model or technique”

Specific Solution“Solution to application problem”

Selection: Use-oriented

ExamplesNIST ATP Grants

Aim to foster product-oriented technology[Rue97]

Academic ResultsOften similar on the surface, but the

end-game for academics (when the research phase ends) is publication rather than a product.

Selection: Truth-oriented

ExamplesForms/3

Designed around cognitive dimensions[Gre90]

Relative utilitarianism[Cha90] [Hil92]

1010

Industrial validation: product-minded.Industrial validation: product-minded.Academic validation: proof-minded.Academic validation: proof-minded.

Industrial ValidationExperience

“Used on real examples”

Selection: Product-minded

ExamplesBenchmarks

Compatibility / Standards

Preliminary integration with products

Case studies

Academic ValidationAnalysis or Experiments

Analytical or empirical evaluation

Examples “Here’s a [toy] example of how it works”

Selection: Proof-minded

ExamplesFormal proofs

Other formal analysis

Lab experiments

Examples

Relative utilitarianism[Sha03i][Zel98]

1111

Advantages of industry approachAdvantages of industry approach

• Increased likelihood that research results will directly impact products. In industry…

– Research questions have a tighter tie to product teams’ needs (maybe less need to persuade product people).

– Research results are produced with the target platform in mind (maybe less need to port & rewrite).

– Research evaluations often involve case studies in the target context (maybe fewer unanticipated requirements in porting to products).

Relative utilitarianism[Pfl00] [Pot93] [Pun06]

1212

Advantages of academic approachAdvantages of academic approach

• Increased likelihood that research will yield highly generalized, long-term benefits. In academia…

– Research questions are often unrelated to existing product teams’ needs (may be more revolution than evolution)

– Research results are communicated through publications and public settings (maybe less reinventing of the wheel)

– Research validation ultimately aims to convincingly demonstrate generalized effectiveness (maybe fewer wheels needed for many different products)

Relative utilitarianism[Pot93]

1313

Industry research appears to be more Industry research appears to be more accessible to new PhDsaccessible to new PhDs

• Up Next: A few statistics on– Numbers of new PhDs, versus available slots

– Funding, salaries, and job choices

Relative accessibility

1414

There aren’t enough slots There aren’t enough slots for all of us to become academic faculty.for all of us to become academic faculty.New CS+CE PhDs Granted Number of CS+CE Faculty

Approx 230 PhD-granting depts in US + Canada responded to each year’s survey [Tau03-07]

(about 130 faculty retire/die/etc per year)(2007 faculty count should show an uptick ~50)

(note: “faculty” doesn’t include postdocs or staff scientists)

Relative accessibility

900 1500 = 67% decrease

1515

Funding from teaching and external sources Funding from teaching and external sources looks sketchy.looks sketchy.

Newly Declared CS+CE Majors

Among undergrads attending those 230 PhD-granting depts in Taulbee surveys [Tau03-07]

External Funding Sources

In approx 125 deptsthat received external funding [Tau03-07]

Relative accessibility

$0.8B $0.6B = 31% decrease

23K 13K = 43% decrease

1616

Corporate R&D (more D than R) Corporate R&D (more D than R) is huge and growing.is huge and growing.

Approximately 25% increase

Based on the 1000 companies that spend the most on R&D [Boo06]

However: Less than 10% of industrial R&D is “basic research”.NSF says 3.8%; AIP says 8.9% -- [NSF08][Lea02]; see also [Duk04][Sci07]

Growth in Corporate R&D Budgets

Relative accessibility

R&D BudgetMicrosoft/MSR $7.1 B (2007)

IBM $6.2 B (2007)

Intel $5.9 B (2006)

Sun $2.0 B (2006)

Google $1.2 B (2006)

US publicly traded tech companies

$92 B (2006)

[Ann], [Kes06]

1717

Pay in academia is Pay in academia is not as bad as rumors suggest.not as bad as rumors suggest.

*=[Pay] **=[Tau07]

Assumptions: (1) Most of us have at least 5 yrs experience at a relatively senior software engineering position(2) We could graduate with a PhD and get a developer job using this experience, or we could graduate witha PhD and be a new hire in academia, or we could leave without a PhD and be like an average CMU grad.If you don’t like these assumptions, then try your own--see citations below. All pay includes bonuses.

1818

Neither industry nor academiaNeither industry nor academiais right for everybody.is right for everybody.

1499 CS + CE PhD graduates

98 SE PhD graduates

235 PhD-granting depts, 2007 [Tau07]

105 CSD graduatesMay 2000-Oct 2005 [Pfe]

235 PhD-granting depts, 2007

What this means for us

1919

So what does this mean for usSo what does this mean for usif we work in industry research?if we work in industry research?

• 30-minute structured interviews– Non-academic researchers at 7 organizations

• IBM, Microsoft/MSR, Intel, Google, Nokia, Adobe, SEI

• Subjects were chosen based on convenience/availability Not a systematic investigation (eg: no careful sampling procedure) Not meant to give generalizable knowledge about industry Not meant to test or evaluate some hypothesis Just meant to tell us about some particular companies

where we might possibly spend a big chunk of our lives

What this means for us

2020

Organization structure and funding modelsOrganization structure and funding modelsreinforce connections to productsreinforce connections to products

Company trait Mentioned byResearchers meet regularly with product teams All interviewees

They periodically show prototypes to product teams Most interviewees

Some research funding comes from product teams IBM

Advance development teams exist to help with tech transfer Nokia

Embedding research in advanced development ups the pace Adobe

Researchers may interact directly with end-customers SEI

Yet most interviewees reported some intellectual freedom• Picking their own research questions (albeit within company scope)• Being able to publish• Establishing external collaborations• Feeling light to moderate pressure to produce short-term results (usually <≈ 3 years)

What this means for us

2121

Not Not all evaluation criteria are product-focused.all evaluation criteria are product-focused.

Impact can be indirectExample: Some MSR researchers focus on providing better tools or methodsto help product teams produce better products.

What this means for us

Individual evaluation may differ from project evaluationExample: Adobe evaluates projects based on product impact but rewards individuals for publications as well as product impact.

Impact On

2222

Process for publicationProcess for publication

IBM MSR SEI Nokia Intel Google Adobe

Patent-check X X X X X

NDA concerns? X

Vetting process? X X X

Key concernsClients’internaldetails

Secret chip sets

Secret data sets

Business Strategy

(very roughly in decreasing order of approximate publication count)

What this means for us

2323

Two final key industry traits:Two final key industry traits:no students, and non-discrete promotionsno students, and non-discrete promotions

• Students are in short supply in industry research Interns are in hot demand Researchers do “grunt work” themselves Researchers collaborate with one another

• Promotions in industry are less “discrete” than tenuring Lots of little pay scale steps along the way Nobody gets fired by “missing tenure”

(though some decide research isn’t for them, and go into development)

Funding is subject to continuous/non-discrete shrinkage

(And, of course, academic researchers seek external grants to pay students and other project costs. Industry researchers usually have salaries but sometimes seek internal grants for other costs.)

What this means for us

2424

Tips from CMU Career CenterTips from CMU Career Center

• Use the alumni directory [Alm]; email previous graduates– Ask why they chose their job

– Ask what they like/dislike

– Ask if they feel intellectually stifled

– Ask about the task mixture (research / management / coding / etc)

• Know thyself– Do you like a lot of freedom to set your own course?

– Or do you fear that you will be spinning your wheels?

– Do you like to teach, or does it feel like a distraction?

– Could you stomach the high risk/reward in startups?

Closing

2525

Final thoughts from intervieweesFinal thoughts from interviewees

• Industry can be a great choice– It offers great flexibility, mentoring and collegiality - SEI

– It is a fun place to work - Google

– Take time to carefully consider your industry options - Intel

• Tips for getting a job in industry– When interviewing, know what you want to work on - MSR

– In your research, remember the ultimate user - Nokia

– Potential hires are judged by what they have built and finished - Adobe

Closing

2626

Thank YouThank You

• …to my interviewees for their time and helpful feedback

• …to CMU Career Center and Frank Pfenning for info

• …to Mary Shaw and Brad Myers for their thoughts

• …to NSF and EUSES for funding this literature survey (ITR-0325273 and CCF-0438929)

Closing

2727

References (1)References (1)

[Alm] CMU Alumni Directory, http://alumni.cmu.edu/index.cfm

[Ann] IBM, Microsoft, Sun, Intel, and Google Annual Reports

ftp://ftp.software.ibm.com/annualreport/2007/2007_ibm_annual.pdf

http://www.microsoft.com/msft/reports/ar07/downloads/MS_2007_AR.doc

http://www.sun.com/aboutsun/investor/annual_reports/sun_ar06.pdf

http://media.corporate-ir.net/media_files/irol/10/101302/2006IntelAnnualReport.pdf

http://investor.google.com/pdf/2006_Google_AnnualReport.pdf

[Boo06] Booz Allen Hamilton, The Global Innovation 1000, 2006, http://www.strategy-business.com/resiliencereport/resilience/rr00039?pg=all

[Bud07] M. Budiu. Academics Love Themselves, Mihai Budiu’s Blog, June 23, 2007, http://www.budiu.info/blog/2007/06/23/academics-love-themselves/

[Bur01] M. BURNETT, J. ATWOOD, R. WALPOLE DJANG, and J. REICHWEIN. Forms/3: A First-Order Visual Language To Explore the Boundaries of the Spreadsheet Paradigm. Journal of Functional Programming, Vol. 11, No. 02, 2001, pp. 155-206.

[Cha90] C. Chang and G. Trubow. Joint Software Research Between Industry and Academia. Software, IEEE, Vol. 7, No. 6, 1990, pp. 71-77.

[Duk04] C. Duke. Creating Economic Value from Research Knowledge, The Industrial Physicist, American Institute of Physics, August-September 2004, pp. 29-31.

[Dyb05] T. Dyba, B. Kitchenham, and M. Jorgensen. Evidence-Based Software Engineering for Practitioners. Software, IEEE, Vol. 22, No. 1, 2005, pp. 58-65.

[Gre89] T. Green. Cognitive Dimensions of Notations. Proceedings of the Fifth Conference of the British Computer Society, Human-Computer Interaction Specialist Group on People and Computers V Table of Contents, 1990, pp. 443-460.

[Hil92] C. Hilsum. The Proper Roles for Academic and Industrial Research. Engineering Management Journal, Vol. 2, No. 4, 1992, pp. 189-199.

[Jon03] S. Peyton Jones, A. Blackwell, and M. Burnett. A User-Centred Approach To Functions in Excel. ICFP '03: Proceedings of the Eighth ACM SIGPLAN International Conference on Functional Programming, ACM, 2003, pp. 165-176.

[Kes06] M. Kessler, Some tech companies cut R&D budgets, USA Today, June 13, 2006, http://www.usatoday.com/tech/news/2006-06-13-tech-research_x.htm

Closing

2828

References (2)References (2)[Lea02] A. Leath, Science & Engineering Indicators: Trends in U.S. R&D Expenditures, AIP Bulletin of Science Policy News,

http://www.aip.org/fyi/2002/099.html[MIT07] MIT Class of 2007 Graduating Student Survey, http://web.mit.edu/career/www/infostats/graduation07.pdf[New94] W. Newman. A Preliminary Analysis of the Products of HCI Research, Using Pro Forma Abstracts. Proceedings of the SIGCHI

Conference on Human Factors in Computing Systems: Celebrating Interdependence, 1994, pp. 278-284.[NSF08] Natinoal Science Foundation, Research and Development: Essential Foundation for U.S. Competitiveness in a Global Economy, Jan

2008, http://www.nsf.gov/statistics/nsb0803/nsb0803.pdf (see p. 6)[Pay] Payscale.com, data retrieved March 21, 2008.[Pfe] F. Pfenning, personal communication, March 21, 2008.[Pfl00] S. Pfleeger and W. Menezes. Marketing Technology To Software Practitioners. Software, IEEE, Vol. 17, No. 1, 2000, pp. 27-33.[Pot93] C. Potts. Software-Engineering Research Revisited. Software, IEEE, Vol. 10, No. 5, 1993, pp. 19-28.[Pun06] T. Punter, R. Krikhaar, and R. Bril. Sustainable Technology Transfer. Proceedings of the 2006 International Workshop on Software

Technology Transfer in Software Engineering, 2006, pp. 15-18.[Red85] S. Redwine Jr and W. Riddle. Software Technology Maturation. Proceedings of the 8th International Conference on Software

Engineering, 1985, pp. 189-200.[Rue97] R. Ruegg. The Advanced Technology Program's Evaluation Plan and Progress. 7th International Forum on Technology Management

(IFTM), Kyoto, Japan, November, 1997, pp. 3-7.[Sci07] J. Scinta, Industrial Research Institute's R&D Trends Forecast for 2007, Industrial Reserach Institute, Jan-Feb 2007,

http://www.iriinc.org/Template.cfm?Section=Home&CONTENTID=5994&TEMPLATE=/ContentManagement/ContentDisplay.cfm[Sha03] M. Shaw. What Makes Good Research in Software Engineering?. International Journal on Software Tools for Technology Transfer

(STTT), Vol. 4, No. 1, 2002, pp. 1-7.[Sha03i] M. Shaw. Writing Good Software Engineering Research Papers. Software Engineering, 2003. Proceedings. 25th International

Conference on, 2003, pp. 726-736.[Tau03-07] Taulbee Surveys, Computing Research Association, 2003-2007, http://www.cra.org/statistics/[Zel98] M. Zelkowitz, D. Wallace, and D. Binkley. Culture Conflicts in Software Engineering Technology Transfer. NASA Goddard Software

Engineering Workshop, 1998.

Closing

2929

Not Not all evaluation criteria are product-focused.all evaluation criteria are product-focused.

CompanyImpact on

Adobe Google Nokia MSR IBM Intel SEI

Products X X X X X

Patents X X X X

Publications X X X X X X

Publicity / PR X

Community X X X

Impact can be indirectExample: Some MSR researchers focus on providing better tools or methodsto help product teams produce better products.

What this means for us

Individual evaluation may differ from project evaluationExample: Adobe evaluates projects based on product impact but rewards individuals for publications as well as product impact.