A summary deck describing some ground rules and potential starting points for conducting research reviews.

1. Critical Research Review
Neil Ernst
EmpiRE 2015

2. Format
Small groups:
• 1: Cognitive Effectiveness of Conceptual Modeling Languages:
Examining Professional Modelers
• 2: Inviting Everyone to Play: Gamifying Collaborative Requirements
Engineering
• 3: Towards a Theory on the Interaction Between Requirements
Engineering and Systems Architecting
• Phase 1: overview of research approach and problem (proposer)
• Phase 2 (45 mins): breakout group questions and discussion. Write-up
key observations
• Phase 3: Plenary presentation of observations to entire EMPIRE group

3. Guidelines
• Peer review of research approach
• Critique, not criticize: goal is better research outcomes. Be helpful.
• Keep clear on what is shareable to wider group
• Show raw data if possible
• “borrowing smart people’s brains to work on the problem for an
hour”

4. Getting started: possible questions to ask
• Is this problem investigation or treatment design/evaluation?
• What is the research question?
• Who are the subjects? Are they suitable?
• How many / how representative are the units of analysis?
• What is the theoretical perspective?
• What is the core empirical approach? Quantitative? Qualitative?
• Threats to validity:
• Externalizeable? Internally coherent? Constructs sensible?
• Replicable? What replications would make sense?
• What use is envisioned for the results?