Presentation of a usability study a group of MSc students at Sussex University (including myself) carried out on the Android game, X-Construction Lite

1. X-Construction Lite: HCI evaluation
Title
Andy Keavey, Mariana de Rojas-Morao,
Merve Yildirim, Queen Ajifa Ododo, Tom Hume
Text
Tom Hume Human-Computer Interaction, assignment 1

2. Title
Text
Screenshot of page from
Android Marketplace
Tom Hume Human-Computer Interaction, assignment 1

3. Overview of gameplay
First screen of instructions (animates) Second screen of instructions
Game-play screen Results of play screen
Tom Hume Human-Computer Interaction, assignment 1

4. Evaluation approach
1. Expert review
1.1. Cognitive walkthrough, written up
1.2. Evaluation vs Nielsen heuristics
2. User testing
2.1. Pre-test questionnaire
2.2. User test of first level (video recorded)
2.3. Interview with user whilst reviewing recording
2.4. Post-test questionnaire
Tom Hume Human-Computer Interaction, assignment 1

5. User evaluation
8 test subjects recruited
(all Android owners, familiar with UI)
Tom Hume Human-Computer Interaction, assignment 1

6. Results of cognitive walkthrough
Issue # reports
Zoom functions misused ✓✓✓
Errors placing girders ✓✓✓
Misunderstanding help messages ✓✓✓
Unsure how to proceed ✓✓
Game crashed on start ✓
Problem starting play ✓
Confused by pop-up adverts ✓ “I successfully drew some
girders, but I struggled a bit
because they were going
“Some of my girders went out of view at the wrong direction most
the top of the screen. I used the zoom out
of the time”
gesture consistent with Google Maps and
could see a much better view of the task”
Tom Hume Human-Computer Interaction, assignment 1

7. Results of heuristic evaluation
Agreement Disagreement
Match between system & real world User control & freedom
Consistency & standards Error prevention
Flexibility & efficiency Recognition over recall
Aesthetics & minimalism
Tom Hume Human-Computer Interaction, assignment 1

8. Results of quantitative data
300
Seconds to complete first level
225
150
75
0
0 1 2 3 4
# attempts to complete first level
Tom Hume Human-Computer Interaction, assignment 1

9. Results of questionnaire data
Age of test subjects Female Male
40
30 25%
20
10
75%
0
8 7 5 4 6 1 3 2
Estimated vs actual fun Android gamer Not android gamer
8
6
4 38%
2
63%
0
1 2 3 4 5 6 7 8
Estimated Actual
Tom Hume Human-Computer Interaction, assignment 1

10. Results of user testing
Issue # users
Misunderstood girder placement ✓✓✓✓✓✓
Didn’t understand help/instructions ✓✓✓✓✓
Mixed up replaying/restarting level ✓✓✓✓
Couldn’t get the train started ✓✓✓
Panned unintentionally ✓✓
Couldn’t tell they were complete ✓
Couldn’t find girder removal tool ✓
Confused by an advert ✓
Misunderstood tool instructions ✓
Couldn’t find instructions again ✓
Tom Hume Human-Computer Interaction, assignment 1

11. Key issue: misunderstood girder placement
Tom Hume Human-Computer Interaction, assignment 1

12. Key issue: mixed up replaying/restarting level
Tom Hume Human-Computer Interaction, assignment 1

13. Conclusions
Good news
• Most players complete the first level, first time
• Most players enjoy the game more than they thought
• Many expert evaluators are now regular players
Bad news
• 75% players, 3/5 experts had issues with placing girders
• 62% players don’t read help, 3/5 experts misunderstand it
• 50% players mix up replaying and restarting the level
• 37% players, 2/5 experts have trouble starting their train
• 20% players pan unintentionally, 3/5 experts had zoom issues
Tom Hume Human-Computer Interaction, assignment 1

14. Recommendations
1. Warn players on the first level when they place a girder which
doesn’t finish attached to an end-point;
2. Clarify language in help messages and screens;
3. Improve the iconography, labelling or placement of the “replay
failure” and “restart level” icons to reduce confusion between
them;
4. Teach players to use the “play” button by animating it on the
first level when they have a workable bridge;
5. Allow pinch-and-zoom to zoom into the bridge during play.
Tom Hume Human-Computer Interaction, assignment 1

15. Critique of methods
Hard to record video in sunlight
Users were self-selecting, to a degree
Slight inconsistencies in questionnaire usage
Tom Hume Human-Computer Interaction, assignment 1

16. Questions?
Tom Hume Human-Computer Interaction, assignment 1