The document introduces the ECOVAL framework for increasing ecological validity in usability testing. ECOVAL provides guidelines for manipulating different attributes of the testing environment, such as using real vs mock objects. It also presents GAMBIT, a supporting tool that allows producing low-cost interactive prototypes to conduct remote and unobtrusive usability tests with an ecological validity ratio of 1:1.17 compared to paper tests. A case study demonstrates how applying ECOVAL and using GAMBIT resulted in early detection of usability problems, increased user productivity, and decreased costs compared to traditional paper-based testing.

2. Suzanne Kieffer
ECOVAL: A Framework for Increasing the
Ecological Validity in Usability Testing
Jean VanderdoncktUgo Braga Sangiorgi
Université catholique de Louvain
Louvain School of Management
Louvain-la-Neuve, Belgium

3. Ecological validity
Experimental
design
Natural
setting
x x
x
xx
x
xx
x
x
x
x
x
x
x
x
Environment
Tasks
Stimuli

4. Ecological validity
High
Low
Experimental
design
Natural
setting
≈
Experimental
design
Natural
setting
≠

5. Internal validity
Was the study well done?
External validity
Can we generalize the findings to
particular persons, settings and times?
Ecological validity
To which extent can we generalize the
findings?

7. User interface design
Ecological
validity
Overall
validity
Conclusions
of usability
evaluations

8. What’s the problem with ecological validity?
 No formal definition that enables
 To objectively quantify the ecological validity of
experimental designs
 To capture the critical attributes of the real-world
environment

9. LOGICAL IDITYVALECO ECOVAL
A framework for increasing the ecological validity in usability testing

10. Outline
 The ECOVAL framework
 ECOVAL
 Guidelines
 GAMBIT
 Case study
 Operability and usefulness of ECOVAL
 ROI of increased ecological validity
 Recommendations

12. The ECOVAL framework

13. The guidelines

14. No objects

15. Mock objects

16. Real objects

17. Only verbalized

18. Mimicked/verbalized

19. For real

20. GAMBIT – the supporting tool
 Production of prototypes
 Easy
 Quick
 Cost-effective
 Real user interface performances
 Interaction
 Navigation
 Responsiveness

21. GAMBIT – how to
1
Produce
screens

22. GAMBIT – how to
2
Create
behaviour

23. GAMBIT – how to
3
Conduct
experiment

24. GAMBIT – how to
4
Download
log file

25. • Q1: To which extent is ECOVAL
applicable for controlling the
ecological validity?
• Q2: What are the possible
outcomes/benefits of
increased ecological validity ?

26. Case study

27. Hot-Dip Galvanizing on Continuous Lines

29. Monitoring report
Task Control Result
Task Control Result
Artefact:
paper checklist
Section > Subsection
Function – Name - Date – Shift
Section > Subsection
300 items a day

31. Timeline
Task
analysis
User
profile
Usability
goal
setting
Screen
design
Usability
evaluation

32. Usability evaluation
Paper versus GAMBIT prototype

33. Ecological validity
Ratio Paper-to-Gambit for ecological validity of 1:1.17

34. Balsamiq mockups

35. Experimentations
Paper GAMBIT
5 key users x 2
10 participants (30’) 18 participants (1 hour)
5 key users x 3
+ 3 user representatives
V1
6 participants
V2
12 participants

36. New version of the prototype

37. Apparatus

38. Results and discussion

39. Early detection of usability problems
 Increased user efficiencies
 Increased user satisfaction

40. Cost-benefit analysis
130
hours
18
hours

41. Cost-benefit analysis (continued)
$74,480
$194,480

42. Paper GAMBIT
Ecological
validity
1 1.17
Data
collection
Manual & face-to-face
Difficult & tedious
Unstructured process
Obtrusive
Automated & remote
Comfortable
Structured process
Unobtrusive
Outcomes
Compliant with
the literature
Early detection of usability problems
Increased user productivity
Decreased late design changes
Increased organizational efficiency

43. Recommendations

44. Take away
 {ECOVAL, Guidelines, GAMBIT}
 Case study
 Operability
 Usefulness
 ROI
 Recommendations

45. Contact details
suzanne.kieffer@uclouvain.be
ugo.sangiorgi@uclouvain.be
jean.vanderdonckt@uclouvain.be
Thank you!