Evidence-Based Management for Researchers British Academy of Management December 13, 2012

1. London, Dec 13, 2012
Postgraduate Course
British Academy of Management
Linking Research Questions
to Appropriate Designs

2. What is the best design?
Postgraduate Course
 Randomized controlled study?
 Grounded theory approach?
 Cohort / panel study?
 Qualitative field research?
 Longitudinal study?
 Post-test only study?
 Survey?
 Action research?
 Case study?

3. Postgraduate Course
What is the BEST car?

4. Postgraduate Course
(more / less appropriate)

5. What is the best design?
Postgraduate Course
quants vs quallies, positivists vs post structuralist, etc

6. Postgraduate Course

7. Types of questions
Postgraduate Course
Does it work?
Does it work better than ....?
Does it have an effect on ....?
Effect
What is the success factor for ....?
What is required to make it work ...?
Will it do more good than harm?

8. Types of questions: non-effect
Postgraduate Course
Needs: What do people want or need?
Attitude: What do people think or feel?
Experience: What are peoples’ experiences?
Prevalence: How many / often do people / organizations ...?
Procedure: How can we implement ...?
Process: How does it work?
Explanation: Why does it work?
Economics: How much does it cost?

9. Postgraduate Course
Effect vs Non-effect

10. Postgraduate Course
Internal validity

11. Internal validity
Postgraduate Course
internal validity = indicates to what extent the
results of the research may be biased and is thus
a comment on the degree to which alternative
explanations for the outcome found are possible.

13. Causal relations
Postgraduate Course
We are pattern seeking primates:
we are predisposed to see order
and causal relations in the world

14. Causality
Postgraduate Course
So, when do we know there is causal relation?
Three criteria:
1. the "cause" and the "effect" are related
2. the "cause" precedes the "effect" in time
3. there are no plausible alternative explanations for the
observed effect

15. ‘Effect’ questions
Postgraduate Course
Considerations for research:
1. Are the "cause" and the "effect” related?
effect size
1. Does the "cause" precede the "effect" in time?
before and after measurement
2. Are there no plausible alternative explanations for
the observed effect?
randomization, blinding, control group

16. Bias & Confounding
Postgraduate Course
Research shows:
Shoe size > quality of handwriting
Smoking youngsters > better lung function

17. Postgraduate Course
Research shows:
Charismatic leaders
Successful companies

18. Levels of internal validity
Postgraduate Course

19. Which design for which question?
Postgraduate Course
Explanation

20. Postgraduate Course
Different types of research questions
require different types of research
designs, but ...

21. But 1: feasibility
Postgraduate Course
Best research design?

22. But 1: feasibility
Postgraduate Course

23. But 2: it’s a trade off
Postgraduate Course
All research designs are flawed – though each is
flawed differently:
 Controlled studies: high internal validity, but sometimes
less suited to generalization
 Surveys and field research: low internal validity, but
sometimes more useful for management practice.
McGrath, J. (1981). Dilemmatics, The Study of Research Choices and Dilemmas.
American Behavioral Scientist, 25(2).

24. Postgraduate Course
SO:
Evidence accrual requires convergence of findings
derived from multiple studies that research the same
constructs and variables with different research
designs and in different contexts.

25. But 3: stage of development
Postgraduate Course
The appropriateness of a research design is also determined by
the body of evidence’s stage of development.
mature: extensively studied > controlled studies to identify
independent, dependent and moderating variables
nascent: open-ended the research questions > exploratory
qualitative studies to shape the understanding of the topic .
Edmondson, A. C., & McManus, S. E. (2007).
Methodological fit in management field research.
Academy of Management Review, 32(4), 1155-1179.

26. Postgraduate Course
Figure 5: Controlled vs uncontrolled studies

27. In summary
Postgraduate Course
Different types of research questions require different
types of research designs, but ...
 feasibility
 past research
 stage of development
also have to be taken into account!