A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014
•
1 like•486 views
An invited talk at ISMAR 2016 workshop on Perceptual Issues in Augmented Reality! Citation: A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014 Arindam Dey, Mark Billinghurst, Robert W. Lindeman, J. Edward Swan II IEEE International Symposium on Mixed and Augmented Reality 2016, Pages: 49-51. Merida, Mexico.
Recommended
Recommended
More Related Content
Similar to A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014
Similar to A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014 (20)
Recently uploaded
Recently uploaded (20)
A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014
- 1. A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014 Arindam Dey, Mark Billinghurst, Robert W. Lindeman, J. Edward Swan II arindam.dey@unisa.edu.au
- 2. • Earlier Surveys • Methodology • Results • Key Insights A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014
- 3. Earlier Surveys of AR Usability Studies Time Venues Papers# Swan & Gabbard (2005) 1992 - 2004 ISMAR,ISWC, VR, Presence 21 Dünser et al. (2007) 1992 - 2007 ACM, IEEE digital libraries 165 Bai & Blackwell (2012) 2001 - 2010 IEEE ISMAR 71 Our Survey 2005 - 2014 Scopus 291
- 4. • Earlier Surveys • Methodology • Results • Key Insights A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014
- 5. Methodology Scopus Keyword Search Unique (1,147) Reported User Study (604) Quality Check (396) Norming (5) Review / Data Collection Impact (291)
- 6. Methodology Scopus Keyword Search Unique (1,147) Quality Check (396) Impact (291) Keywords from Denser et al. (2007): "augmented reality" AND "user evaluation(s)" "augmented reality" AND "user study/-ies" "augmented reality" AND "feedback" "augmented reality" AND "experiment(s)" "augmented reality" AND "pilot study" "augmented reality" AND participant AND study "augmented reality" AND participant AND experiment "augmented reality" AND subject AND study "augmented reality" AND subject AND experiment Additional keywords for our survey: "augmented reality" AND "user experience(s)” "augmented reality" AND "usability” “augmented reality” AND “evaluation(s)” AND “participant” “augmented reality” AND “evaluation(s)” AND “user” “augmented reality” AND “empirical” AND “participant” “augmented reality” AND “empirical” AND “user” “augmented reality” AND “empirical” Reported User Study (604)
- 7. Methodology Scopus Keyword Search Unique (1,147) Description of the user study included • Task • Design • Participant demographics Quality Check (396) Impact (291) Reported User Study (604)
- 8. Methodology Scopus Keyword Search Unique (1,147) Average Citation Count (ACC) = Total Lifetime Citation / Lifetime in Years ACC >= 1.5 included Quality Check (396) Impact (291) Reported User Study (604)
- 9. Methodology Norming (5) Review / Data Collection • Five papers were randomly selected • All reviewers reviewed the papers • Discussed the process in a group • Agreed on reviewing consistency • Started reviewing papers individually
- 10. Methodology • Distributed papers equally among four • Regular meetings (Skype) • Doubt resolution • Data collected through Google Form 291 papers with 369 studies were reviewed Norming (5) Review / Data Collection
- 11. Methodology • Application areas & keywords • Experimental design & data collection • Participant demographics • Experimental tasks & environments • Types of experiment (pilot, formal etc.) • Types of senses augmented and displays used Focused Attributes
- 12. • Earlier Surveys • Methodology • Results • Key Insights A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014
- 13. Results Publication Year 2014 2013 2012 2011 2010 2009 2008 2007 2006 2005 NumberofPapers 60 50 40 30 20 10 0 Interaction Collaboration Tourism and Exploration Navigation and Driving Industrial Application Entertainment and Gaming Education Medical Perception Application Areas • Most papers in Interaction (67) and Perception (51) • Tourism (8), Collaboration (12), and Entertainment (14) are among the least
- 14. Results • HMD (102) and Handheld (100) almost equal • 2010: Handheld overtook NumberofPapers 30 25 20 15 10 5 0 Publication Year 2014 2013 2012 2011 2010 2009 2008 2007 2006 2005 … Spatial Desktop HMD Handheld Displays Used
- 15. Results • Primarily lab-based / controlled NumberofPapers 60 50 40 30 20 10 0 Publication Year 2014 2013 2012 2011 2010 2009 2008 2007 2006 2005 Heuristic Evaluation Pilot Test Field Study Formal User Study Experiment Type
- 16. Results • Tourism and Collaboration most cited • Medical had most authors per paper on average • Most collected both quantitative and qualitative data
- 17. Results • Majority indoor studies • Within-subjects studies are most common • Overall participant median 16 (12-28) • Young, male, and university-based participants -> results generelizable?
- 18. Results perception depth display evaluation user visual optical outdoor cues devices interface mobile see- through study text virtual x-ray contrast distancedrawing feedback handheld head-mounted information interaction legibility presence styles vision blur calibration character color computing design empirical environment focus hand- held haptic immersion label mr presentation rendering saliency sensitivity user-perspective view accommodation activity acuity attention aug autostereoscopy awareness background bci bone consistency context depth- of-field discrimination disparity eeg embodied exocentric expectations experience geo human illusion image layout measures mented methodology metric mirror navigation online peripheral placement polarity predictive projected proprio-perception quantitative realism rubber saltation sensation simulation situation stereoscopic stiffness subtle systems three-dimensional video Perception user museum human interface virtual gesture guide interaction mobile study activity audio aware behavioral collaborative computing context detection environment expectations experience following human-robot identification immersion information investigation learning lunar modeling multi-user ontologies presence public recognition requirements robot rule-based science screen sensory services social surface systems tabletop tour-guide tracking ubiquitous usability visiting Tourism mobile navigation displays studies user visual indoor virtual acceptance camera computing devices driver interface maps audio driving field handheld interaction localization outdoor phones processes rendering spatial activity-based aging ambience arrow artificial authoring aware band binaural camera-based controls cues design differences digital disorientation distraction dynamic editing electronic evaluation gap geospatial gps graphic guided heads-up highlighting human inattention individual input intelligent interac legged locomotion magic management maneuver manipulation movement object occlusion output path peephole perceptual presence roads screen search selection sensory slam small super- imposed system tags tangible techniques technology tion topometric transportation trust turn ultar-wide urban user-centered view visibility wayfinding wearable windshield-based Navigation rehabilitation surgery medical simulation training virtual feedback laparoscopy navigation computer control phobia device endoscopy force gait guidance laparoscopic learning motor needle visual animal biopsy box ct cues education exposure game hapticimage medicine mixed neurosurgery none patient phone physical registration small stroke systems therapy tracking trainers treatmentuser acrophobia augmented-reality closed-loop computer-assisted exercise fmri image- guided index insertion intravenous load mobile model mri nasa objective parkinson placement pretend radio- frequency sinus skull soft therapeutic tissue tomography variability Medical interaction mobile user displaydevice interface techniques camera computing environments gesture lens selection tracking virtual magic spatial study tangible visual design evaluation experience field handheld manipulation multimodal phone projection attention based communicative head head- mounted menus modeling mounted none view wearable acquisition active annotation application conflict context depth dynamic end eye finger fitts force gaze hand human human-computer illusion information input intent law map natural object online orientation oz peephole point prototype pseudo-haptics recognition see- through service survey system target touch usability wizard Interaction user computing guidance interaction interfaces maintenance mobile studies virtual awareness device model order picking task assembly building display engineering ergonomics industrial information instruction multimedia none real-time repair service tracked aided algorithms analysis applications architecture attention audio- haptic augmentedreality compass computer-assisted contex data design environments everyday expectations fabrication facility geometries geospatial gis gps handheld haptic hardware head- mounted heart-rate helmet hmd hypermedia interfaces--input large learning life logistics long-term management methodology micro-projection military mixed monocular mounted non-visual optical participation planning prototypes readability scenarios screen see-through sensor signal sketching spatial strategies styles surveying systems systems--artificial techniques techniques--interaction text tree urban video vision visualization wearable workpiece Industrial games mobile study handheld play co- located collaboration experiences interactive interface phone social sound virtual ar audio audio-augmented auditory avoidance based board capture cd competitive computer computer-supported cooperative cross-media cscp cscw depth devices displays drama empirical engagement environments exploratory eyes-free field garden imaginary interference intergenerational lens linear location magic mixed motion music narratives orientation peephole perception performance physical presence probabilistic qualitative spatial static tabletop task trial user work Gaming learning interactive user computing education environments interface systems teaching collaborative evaluation experience information mobile multimedia training academic human tangible virtual acceptance achievement applications areas attitude authoring cal classroom cognitive cooperative design elementary game guideline human-computer improving method motivation none rehabilitation sensor simulations subject task technology ubiquitous visualization Education collaborative communication interaction video-mediated computer mobile cscw depth human telepresence analysis ar around artificial automotive awareness camera design device games gesture handheld interfaces interpretation markerless mediated mixed mr out-of-body person phones presence recognition referencing remote sensors social spatial streaming study support system tangible telecollaboration touch tracking user view virtual visual vr Collaboration
- 19. • Earlier Surveys • Methodology • Results • Key Insights A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014
- 20. Key Insights • Nine primary application areas emerged • Primarily young, male, and university-based participants used • Handheld displays had a rapid uptake and equalled HMDs • More user studies needed in collaboration and entertainment • Median of 12-28 participants used in studies (overall median 16) • Not enough execution of pilot studies • Heuristics for AR applications need to be developed • Subjective questionnaires are most widely used instrument • Medical interfaces less focused on usability • Work in progress List of all papers reviewed are available on poster!
- 21. Arindam Dey arindam.dey@unisa.edu.au Empathic Computing Lab University of South Australia www.empathiccomputing.org