case studies in industry - what we have learnt
TRANSCRIPT
![Page 1: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/1.jpg)
What We Have
Technische Universität München
Case Studies in Industry
Daniel Méndez
Technical University of MunichGermany
CESI 2016Austin, Texas
@mendezfe
Joint work withStefan Wagner (University of Stuttgart)
Based on material from a joint work with:Andreas Jedlitschka (Fraunhofer Institute for Experimental Software Engineering)Stefan Wagner (University of Stuttgart)
Feedback fromAntonio Vetrò (Technical University of Munich)
Jonas Eckhardt@binsanoj
![Page 2: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/2.jpg)
Compliments
Critique
![Page 3: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/3.jpg)
(…)
Background
![Page 4: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/4.jpg)
But before we start…
![Page 5: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/5.jpg)
Reviewer Request
“ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”
![Page 6: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/6.jpg)
Reviewer Request
“ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”
Selective perception
“Everything went great!”
![Page 7: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/7.jpg)
Reflection
Reviewer Request
“ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”
Selective perception
“Everything went great!”
![Page 8: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/8.jpg)
Reflection
Reviewer Request
“ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”
ExcuseSelective perception
“Everything went great!”
“Well, maybe not everything, but it wasn’t our fault […]”
![Page 9: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/9.jpg)
Reflection
Reviewer Request
“ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”
ExcuseSelective perception Denial
“Everything went great!”
“Well, maybe not everything, but it wasn’t our fault […]”
“Seriously, how should I have reacted?”
![Page 10: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/10.jpg)
Reflection
Reviewer Request
“ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”
ExcuseSelective perception Denial
“Everything went great!”
Acceptance
“Well, maybe not everything, but it wasn’t our fault […]”
“Seriously, how should I have reacted?”
“Ok, this was really naive | stupid | biased | not really scientific | …”
![Page 11: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/11.jpg)
Reflection
Reviewer Request
“ At the workshop, it would be interesting to hear the authors [Jonas] reflect on what they [the authors] learned from the process of reflecting on the experiences and writing the paper […]”
ExcuseSelective perception Denial Active learning
“Everything went great!”
Acceptance
“Well, maybe not everything, but it wasn’t our fault […]”
“Seriously, how should I have reacted?”
Learning by doing…it wrong.
“Ok, this was really naive | stupid | biased | not really scientific | …”
![Page 12: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/12.jpg)
Finding contacts
Image source: https://c1.staticflickr.com/1/74/196191428_7dc71b7bf7_z.jpg?zz=1
![Page 13: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/13.jpg)
» Integrate studies into ongoing / planned research projects
» Actively approach
– alumni of your university
– companies in your area
– practitioners at (local) events
Finding contacts
Image source: https://c1.staticflickr.com/1/74/196191428_7dc71b7bf7_z.jpg?zz=1
![Page 14: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/14.jpg)
Convincing contacts to participate
![Page 15: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/15.jpg)
Convincing contacts to participate
» Find appropriate terminology (technology transfer)
» Find appropriate incentives for project partners
» Respond to their local needs
» Provide early feedback / results
» In funded projects, probability is higher that practitioners collaborate
![Page 16: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/16.jpg)
Dealing with uncertainty
• What will the stakeholder characteristics be?• Availability• Skills• Motivation to participate• Commitment to goals• Representativeness• …
• What will the data look like?• Quality of the data• Quantity of the data
![Page 17: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/17.jpg)
Dealing with uncertainty
• What will the stakeholder characteristics be?• Availability• Skills• Motivation to participate• Commitment to goals• Representativeness• …
• What will the data look like?• Quality of the data• Quantity of the data
» Take early samples and test them!
» Test instruments and data quality via pilots
» Get to know your subjects early
» Be flexible, always ask yourself (and others): “What is the potential of the data I’m getting?”
![Page 18: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/18.jpg)
Context variables (and phenomena)• hard to determine (or unavailable)• hard to measure
» Context often hard to describe (hence, hard to reproduce by others)
Variables actually reported
Properly characterising the context
Variables you should measure
Variables youcan measure
![Page 19: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/19.jpg)
Context variables (and phenomena)• hard to determine (or unavailable)• hard to measure
» Context often hard to describe (hence, hard to reproduce by others)
Variables actually reported
Properly characterising the context
» Orient yourself on
» other (similar) studies or
» on classification schemes from the area of software process models (“Tailoring”)
» Focus on whole context, not only on cases for which you want to draw conclusions
» Ask yourself: “What information would I myself need to understand and replicate the study under the assumption it would yield same or similar results?”
Variables you should measure
Variables youcan measure
![Page 20: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/20.jpg)
Working with sensitive data
![Page 21: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/21.jpg)
Working with sensitive data
» Clarify the rules:
» what you need for the case study
» what can be stored and reported (internally and externally)
» Plan review and acceptance cycles for reports and publications
» The “easier” way (sometimes): Joint publication
![Page 22: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/22.jpg)
Possible distortions in data collection:• biased researchers
(trying to sell own PhD topic)• weak moderation skills• hidden agendas of respondents
(missing trust!)• (…)
Skills matter
![Page 23: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/23.jpg)
Possible distortions in data collection:• biased researchers
(trying to sell own PhD topic)• weak moderation skills• hidden agendas of respondents
(missing trust!)• (…)
Skills matter
» Don’t be biased!
» Involve other researchers in:
» Design of instruments
» Data collection (e.g. interviews)
» Continuously practice your
» Moderation skills
» Rhetoric and listening skills
» Build trust by showing loyalty and respect
![Page 24: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/24.jpg)
Possible problem: Support by management…only by management.
» No backup in projects (organisational culture)
Beware the ivory tower
![Page 25: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/25.jpg)
Possible problem: Support by management…only by management.
» No backup in projects (organisational culture)
» Make clear study plan including needed subjects & cases (and formally agree on it)
» Talk early to subjects
» Find active supporter (not sponsor) with own interests in project (= their own incentives)
Beware the ivory tower
![Page 26: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/26.jpg)
Case study data should be disclosed• reliability and trustworthiness?• reproducibility and replicability?
» The success of empirical SE relies on a collaborating community
(Data) Openness and transparency
![Page 27: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/27.jpg)
Case study data should be disclosed• reliability and trustworthiness?• reproducibility and replicability?
» The success of empirical SE relies on a collaborating community
(Data) Openness and transparency
» If possible, make your (anonymised) data accessible to others
http://openscience.us/repo/
![Page 28: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/28.jpg)
Properly reporting the results
Case study results need to be disseminated • to practice• to academia
» How to properly report?
![Page 29: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/29.jpg)
Properly reporting the results
Case study results need to be disseminated • to practice• to academia
» How to properly report?
» Report results according to target group:
» Small presentations to management w/ focus on results
» Peer-reviewed publication w/ (selected) research scope
» (Technical) Reports with full data (and analysis)
» …
» Rely on established guidelines for reporting on case studies
![Page 30: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/30.jpg)
KEY TAKE AWAYS
![Page 31: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/31.jpg)
Case Studies are a little bit like dating
• Finding someone sharing your interest is difficult… getting his attention to get further with you even more.
• You need deal with uncertainties
• You need to compromise (be pragmatic and, maybe, also a little bit of opportunistic)
• Social skills — especially listening skills — matter
• Sometimes, it takes very long to get lucky
![Page 32: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/32.jpg)
Case Studies are a little bit like dating
• Finding someone sharing your interest is difficult… getting his attention to get further with you even more.
• You need deal with uncertainties
• You need to compromise (be pragmatic and, maybe, also a little bit of opportunistic)
• Social skills — especially listening skills — matter
• Sometimes, it takes very long to get lucky
But there is one difference
• Null results matter
• In contrast to your dating experiences, we want to hear about your case study experiences (especially the failed ones)
![Page 33: Case Studies in Industry - What We Have Learnt](https://reader034.vdocument.in/reader034/viewer/2022042907/58e533791a28abac7e8b5e2f/html5/thumbnails/33.jpg)
Thank you!
Daniel Méndez
@mendezfe
• Slides will be made available (and probably tweeted)
• Approach him if you need material (studies, templates, …)