Critical Research Review

Neil ErnstEmpiRE 2015

Format Small groups:• 1: Cognitive Effectiveness of Conceptual Modeling Languages: Examining

Professional Modelers • 2: Inviting Everyone to Play: Gamifying Collaborative Requirements

Engineering • 3: Towards a Theory on the Interaction Between Requirements

Engineering and Systems Architecting• Phase 1: overview of research approach and problem (proposer)• Phase 2 (45 mins): breakout group questions and discussion. Write-up key

observations• Phase 3: Plenary presentation of observations to entire EMPIRE group

Guidelines• Peer review of research approach• Critique, not criticize: goal is better research outcomes. Be helpful.• Keep clear on what is shareable to wider group• Show raw data if possible• “borrowing smart people’s brains to work on the problem for an

hour”

Getting started: possible questions to ask• Is this problem investigation or treatment design/evaluation?• What is the research question?• Who are the subjects? Are they suitable?• How many / how representative are the units of analysis?• What is the theoretical perspective?• What is the core empirical approach? Quantitative? Qualitative?• Threats to validity:

• Externalizeable? Internally coherent? Constructs sensible?

• Replicable? What replications would make sense?• What use is envisioned for the results?