Designing Useful and Usable Augmented Reality Experiences
•Télécharger en tant que PPTX, PDF•
1 j'aime•222 vues
The UX research and design methodology for Augmented Reality experiences
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Designing Useful and Usable Augmented Reality Experiences
Similaire à Designing Useful and Usable Augmented Reality Experiences (20)
Dernier
Dernier (20)
Designing Useful and Usable Augmented Reality Experiences
- 1. Designing Useful and Usable AR User Experiences Experiences for the Future Dr. Yan Xu, 03/08/2021
- 2. 2 UX Research Scientist at Facebook Reality Labs Research Previously: ● Principal UX Researcher & Designer at Magic Leap ● Research Scientist at Intel Labs ● Earned my PhD at Georgia Institute of Technology in the Human-centered Computing program About Me
- 3. 3 My Timeline with AR 2020-: AI + AR, back to research 2005: Screen-based AR: “Blended reality” 2006 Nintendo Wii 2010 Microsoft Kinect 2007-12: Mobile AR: Shopping and Social Games 2007, iPhone 2012, Vuforia, an augmented reality software development kit (SDK) for mobile Arth, Clemens, Raphael Grasset, Lukas Gruber, Tobias Langlotz, Alessandro Mulloni, and Daniel Wagner. "The history of mobile augmented reality." arXiv preprint arXiv:1505.01319 (2015). 2013-16: Camera array: user generated creative content 2017 iPhone X: double cameras 2016-20: Head-mounted display product March, 2016, Oculus Rift CV1 March, 2016, Microsoft Hololens 1, Developer Kit April 2016, HTC Vive July 2016, Pokemon Go
- 4. 4 Topics of Discussion Today • Focus on the design thinking and methods tailored for AR. - What shall we make to create meaningful experiences for people? - How do we make it well for people? • AR is a multidisciplinary field. It’s important to keep in mind that this talk focuses on design frameworks and methods only. • The goal is to inspire you to create the AR apps and services that you care about the most, and give you a few design thinking tools and methods to work with.
- 6. 6 Design Thinking and Process Design Thinking About the co-evolution of problem and solution: “The designers start by exploring the problem space, and find, discover, and recognize a partial structure...They consider the implications of the partial structure in the solution space, use it to generate some initial ideas for the form of a design concept, and so extend and develop the partial structuring...The goal is to create a matching problem-solution pair.” - Nigel Cross, “Designerly Ways of Thinking”, 2007 • What’s design thinking? • “Design thinking is a non-linear, iterative process that teams use to understand users, challenge assumptions, redefine problems and create innovative solutions to prototype and test. ” – Interaction Design Foundation • How is my work going to benefit from design thinking? • Innovative work • Emergent technology and user population
- 7. 7 The Double Diamond Process Design Thinking The ‘Double Diamond’ process maps the divergent and convergent stages of a design process. Created by The British Design Council, it describes modes of thinking that designers use.
- 8. 8 Methods for Design and UX Research (In Product) Design Thinking Choose features • Who are the users? • What are the pain points? • What’s the competitive landscape? • What are the problems that we focus on? Concept design & alternatives • Out of different design directions we diverge onto, which ones are worth further effort? • Whether the assumptions of the design true or not? Prototypes & iterations • How to ensure the effectiveness and usability of the design? • What are the first time user and expert users feedbacks? Finalize the Design • Can we further improve the design? • Are the corner cases fully considered? • Is the design robust? Deliver • Did the design meet the design goals? • What are the future design directions? Discovery • Field Studies • Diary studies • Telemetry analysis • Customer feedback • Surveys • Literature review • Define UX goals • Jobs-to-be-done • Design pattern Prioritization & Selection • Frameworks and models • Test low fidelity prototypes, fail fast, e.g. Wizard of Oz testing, bodystorming, paper prototypes etc. • Design critiques Iterative design and testing • Iterative design • Data-driven design • Usability testing: heuristics, expert review, think aloud, cognitive walkthrough, observations etc. • Comparative studies: A/B testing, experimental designs with log data Beta & Alpha testing • Test with extreme conditions and sample groups • Test in context of the user’s natural environments, diary studies, log data, videos Evaluations • Measure adoption and retention • Review telemetry / usage patterns • Monitor and incoperate customer feedback
- 9. 9 Why AR?
- 10. 10 The Non-linear Nature of Tech History Why AR We can’t predict what technology is hot: - Tech readiness - Tech adoption For example, the well-known Engelbart demo had little immediate impact right after it was first shown to the audience in 1968 (see the quote from Andy van Dam) The first the VR system was created by Ivan Sutherland in 1968. It start to reach the general public with multiple commercial VR devices released in 2016. https://blog.siggraph.org/2018/08/vr-at-50-celebrating-ivan-sutherland.html/ About the Engelbart Demo: “Everybody was blown away and thought it was absolutely fantastic and nothing else happened. There was almost no further impact. People thought it was too far out and they were still working on their physical teletypes, hadn't even migrated to glass teletypes yet. So it sparked interest in a small vigorous research community but it didn't have impact on the computer field as a whole.” — Andy van Dam, Reflections on a Half- Century of Hypertext (29:15). Keynote at Hypertext 2019 conference
- 11. 11 The Myth about “Killer Apps” • The industry has been searching “killer apps for AR” for many years • Assume that there exists a “killer app” for AR, what are you going to do about it? • Searching for “killer apps” may blind us about the real opportunities Why AR
- 12. 12 Why AR: for Me? For Others? • User-centered Design 101: “You are NOT the users” • But many innovations are created for oneself initially • E.g. Airbnb, Instagram • If it is useful for me, is it useful for others? • Common human needs among a subset • Using methods like Jobs-to-be-done to understand the root cause. • The Why behind What: To understand what motivates people to act, you first must understand what it is they to need to get done (Clayton Christensen [1]) • WHAT can change rapidly with solution shift, but WHY remains stable largely • Fundamental human perception/capabilities (e.g. Existing theories in human needs, perception science, cognitive science, and sociology could guide us) [1] https://hbr.org/2005/12/marketing-malpractice-the-cause-and-the-cure
- 13. 13 Why AR: for My Discipline? For Other Disciplines? • What’s the problem in my discipline that can be addressed by new technologies like AR? What are the challenges in AR that can be uniquely addressed by my discipline? • e.g. using eye tracking to segment neurons in volumetric electron microscopy images. • Application of the same solution to other domains • E.g. What about using it for the data labeling in general? Thomas Templier, Kenan Bektas, and Richard H.R. Hahnloser. 2016. Eye-Trace: Segmentation of Volumetric Microscopy Images with Eyegaze. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). Association for Computing Machinery, New York, NY, USA, 5812–5823. A microscopy image from: https://en.wikipedia.org/wiki/Microscopy
- 14. 14 Why AR: For light engagement? For deep engagement? • What are the tasks that people are going to do with AR? • Is it always about high-degree of freedom 3D object manipulations? • What about the real world interactions? • Enable people to enjoy their experience with the right amount of information at the right context • ULFCAII (Ultra-low Friction Contextualized AI interface) from Michael Abrash’s Facebook Connect 2020 keynote (https://youtu.be/-cRxT32G7y4?t=4432).
- 15. 15 What are you going to make with AR? Why AR
- 16. 16 How do we design AR Experiences? SECTION TITLE
- 17. 17 User-Centered Design • User-centered Design give us a good start. But when applying it to the emergent field of AR, there are a few challenging areas. Model of the user-centered design, cited from usability.gov: https://www.usability.gov/what-and-why/user-centered-design.html How does the design improve?
- 18. 18 Research-through-Design • Design as a research method • What goes into design? • Tech opportunities and constraints • Empirical understandings and data of people • Theories and models (e.g. cognitive, perceptual, social science etc.) • What do we learn from the design and how it is used? • New technology capabilities (requirement or implementations) • New empirical data from usage (sometimes its emergent patterns are hard to predict) • Verification, modification and extension of theories “Following a research through design approach, designers produce novel integrations of HCI research in an attempt to make the right thing: a product that transforms the world from its current state to a preferred state.” — Zimmerman, John, Jodi Forlizzi, and Shelley Evenson. "Research through design as a method for interaction design research in HCI." Proceedings of the SIGCHI conference on Human factors in computing systems. 2007.
- 19. 19 Three Design Considerations for AR When the design space is less certain, not as many design examples and heuristics to follow, consider the following: • Design with the Material of AR • Data-driven design • Theory-based design
- 20. 20 Design with the Material of AR • How does the embodied experience feel? (bodily, physics, environmental and social [1]) • How does spatial interaction feel? • Design method: Bodystorming • What does it mean for a system to understand my context? • Design method: Wizard-of-Oz • How does it feel to wear something on body? • Design method: Diary study and Life-logging • What does it mean to be empathetic, being able to see in others’ perspectives? • Design method: iterative design with different types of fidelities of technology • Empathic computing: how do we develop systems that allow us to share what we are seeing, hearing and feeling with others? (Mark Billinghurst [2]) • Exploring different representations of reality: from 2D, to 360 view, to 3D dynamic geometry [1] Jacob, Robert JK, Audrey Girouard, Leanne M. Hirshfield, Michael S. Horn, Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. "Reality-based interaction: a framework for post-WIMP interfaces." In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 201-210. 2008. [2]The coming age of empathic computing | Mark Billinghurst | TEDxUOA https://www.youtube.com/watch?v=ybdcBrG0aPA
- 21. 21 Design with the Material of AR (cont.) • What is the specific AR/AI tech good vs. not so good at doing? Affordances vs. constraints? • Design method: MVP (minimal viable product); Fast iterations; Get your hands dirty • In our design research on the student projects of AR games [1], we found that the teams that start on-device prototypes earlier tend to have a better rated design quality at the delivery • Design pattern research: “Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem in such a way that you can use the solution millions time over, without ever doing it the same way twice.” - Alexander, Christopher. “A pattern language: towns, buildings, construction”. Oxford university press, 1977 [1] Barba, Evan, Yan Xu, Blair Maclntyre, and Tony Tseng. "Lessons from a class on handheld augmented reality game design." In Proceedings of the 4th international conference on foundations of digital games, pp. 2-9. 2009.
- 22. 22 Data-driven Design • We may encounter more data related questions/challenges/opportunities when designing for AR • Challenges: • Chicken and egg problem: What if the design direction need to be decided by data, yet data cannot be obtained without design? • Probabilistic output: Not a clear 0/1 signal like what a button gives • Too much data, too little time
- 23. 23 Chicken and Egg Problem: Design and Data, Which Comes First? Data-driven Design • Sometimes the design cannot be decided until we have enough data about each of the direction; but it’s not possible to make all prototypes at once with resource limitations • Design lessons: • What’s your solution space? Can we enumerate all the possibilities in the solution space and collect data accordingly? • Bootstrap data with minimal design • Can you reduce the solution space to be as small as possible? • Listen to data –especially when it does not agree with your first design intuition. • Iterate between design and data constantly • Can we decide design direction computationally?
- 24. 24 Probabilistic Data Data-driven Design • Outputs from many machine learning models and computer vision algorithms are likely to be probabilistic • False positive and false negative errors • Precision and recall • Confidence level • Confusion matrix … • Design lessons: • Confidence if your friend; always consider the confidence first • The human threshold of what’s usable • The cost of false positive and false negatives differs with tasks and contexts • The trade-offs between different data sources
- 25. 25 Too Many Data, Too Little Time Data-driven Design • When we need a lot of user data? • Ergonomics: diversity of anthropometry data • Social: cultural differences • Input: people move differently • …. • The design lessons for economic ways of iterative testing • Sample for diversity, different from representative composition/distribution of target users • Over sample the users where the algorithms does not seem to do well, stress test • When the economic ways of iterative testing does not apply: • Validation type of study
- 26. 26 Theory-driven Design • From perceptual science to social science, theories are big time savers for design decisions and iterations. • A few exercise scenarios: • How do we make sure texts are legible with the see-through, additive display? • What kind of feedback would you like to design for best gesture-based interactions? • How do we design social experiences where people feel the presence with each other?
- 27. 27 Theory-driven Design (cont.) • A few exercise scenarios: • How do we make sure texts are legible with the see-through, additive display? • Vision theories: stereoscopic vision, depth perception, light and contrast • What kind of feedback would you like to design for best gesture-based interactions? • Perceptual science: action-feedback loop; multimodal perception • How do we design social experiences where people feel strong presence with each other? • Sociological theories: social presence, synchronization, emotion contagion, mutual focus of attention.
- 28. 28 Recap What to make? • For yourself, and for others • For your discipline, and for disciplines • For deep engagement, and for light engagement How to make? • Research-through-Design model • AR as design material • Data-driven design • Theory-based design
- 29. 30 “The best way to predict the future is to invent it.” - Alan Kay SECTION TITLE
- 30. Thank you Q&A
Notes de l'éditeur
- I have been working on AR topics since 2005. My first AR project was with Microsoft Research, we brought in people’s movement to the digital world on a screen. People’s faces or physical objects are tracked in real time and changes the states in the game, introducing more sense of control and presence. At that time we called it ”blended reality”. 1 year later, Nintendo launched Wii, the first physical interaction controller that reaches to the mass market. There were reports about how Wii games become a popular social activity for elderly homes, reaching to many non-typical gamers out there. In 2010, Microsoft Kinect was launched, this time incorporating the full body movements into the game world. Their marketing campaign ”You are the controller”. In 2007-12, I worked on mobile AR experiences. The goal is to encourage more social behaviors by integrating the physical and digital interactions. I also designed and evaluated Augmented Reality mobile shopping experiences during my internship with Nokia Research. In 2007, the launch of iPhone largely accelerated smart phone hardware improvements. QUALCOMM and Nvidia start to make the chips for high-quality graphics on mobile devices, making it feasible to run 3D graphics and computer vision on mobile devices efficiently without draining the battery. When I started making mobile AR games in 2007, the framerate was 12 fps; in 2012, the framerate increased to 50-60 fps. The development of AR also becomes a lot easier with Software development kit like Vuforia and Unity. In 2013, I joined Intel Labs to work on computational photography and camera array. My work was to enable non-expert users to make creative contents by leveraging the capabilities of camera array and computational photography. For example, there have been a lot of sophisticated camera effects and image effects used in movies, but how do we enable non-expert users to easily use these techniques to create their own stories? I designed and developed a series of effects and interfaces based on the scene geometry, temporal manipulation, and dynamic camera properties. While there were a few other multiple camera mobile devices launched, Apple launching the iPhone X with two cameras marks the camera array becoming a norm in the smart phone market. In 2016, I joined Magic Leap to work on Head mounted AR. At Magic Leap, I was the design lead for several products launched with the HMD, including the out of box experience, several core apps related to eyes, such as eye calibration, fitting, Iris login, and display calibration. Collaborating closely with many different teams to deliver these products, such as computer vision, deep learning, optics, vision science, etc. It was exciting work with hardware in its infant stages, and create user experiences along with the growth of the technology through numerous hardware, software, and design iterations. 2016 is an exciting year for AR and VR, many consumer products are released in VR, such as Oculus Rift and HTC Vive. Microsoft released Hololens in March and the quality of tracking blew my mind! In the July, Pokemon Go launched it’s AR mobile app, quickly gathering the attention. I talked about the AR gaming experience at NPR. In 2020, I decided to join Facebook Reality Labs to work on the intersection between AI and AR. This is a very exciting area that provides a whole new set of opportunities for us to create meaningful experiences. That’s why my talk is centered around creating useful and usable AR experiences. Now that the components of the technology are almost here, what do you want to make? And how are you going to make it? I hope today’s talk is going to give you some inspirations. Studierstube, mobile SDK Mobile AR: Graz university of technology Iphone X, One is the standard wide-angle lens that captures a wide view of the scene. The other is a telephoto lens The advances in technology means that we can design very different kinds of user experiences ULFCAI (Ultra-Low-Friction-Context-Aware-AI, ref: Michael Abrash’s Talk at Facebook Connect) (AI + AR)
- First I will give a high level view of designing thinking and process; later I break it into the two sections, first about answering the question of usefulness… I am really glad to see how your AR/VR program is created for multidisciplinary work.
- First of all what’s design thinking? According to interaction Design foundation, Design thinking is a non-linear, iterative process that teams use to understand users, challenge assumptions, redefine problems and create innovative solutions to prototype and test. The core of design thinking is the co-evolution between problem and solution. This is especially true for innovative work where finding the right problem to solve is half of the battle, if not more. Here I cited one of my favorite design research scholars, Nigel Cross, from his book, Designerly ways of thinking. His research method is based on the observation of design teams and their process. This quote summarizes some of the most important learnings from these observations. What I like about it is the acknowledgement of the partial structure – taking the partial understanding of the problem, and use it as inspirations to generate ideas for solutions. Neither the problem nor the solution is fixated until this back-and-forth iterative process if fully explored. The goal is not to find the perfect problem to solve, but rather to find the matching pair between problem and solution. Taking augmented reality as an example here, it could be treated as both a problem or a solution. It’s a problem because the technology and interactions with AR are still being researched and developed. AR itself needs many improvements. Out of all the potential challenges to improve AR, which one you should focus on? AR can also be seen as a solution. With the contextualized information becomes so easy to access, we can solve so many interesting problems with it. But the question is, which problem you’d like to solve with AR? I will get back to these questions again and again later in this talk. I will focus on sharing the design frameworks and methods that can help answering these questions. Now you may ask the question, ”I am not a designer. How is my work benefiting from design thinking?” First of all, design thinking is not for designers only. It’s for innovators. Stanford design school and MIT management school teach people from all sorts of disciplines to use design thinking as a process to do innovative work in their own domain and disciplines. By design thinking, we can become more aware of our own biases and assumptions. Working with technologies like AR and VR provides unique sets of challenges. The technology and the user populations are both emerging. Some of the existing the assumptions will get in the way of innovating in this space.
- The iterative nature of both diamonds Definition: The problem that is prioritized to be focused on; and also defining the criteria of success
- Here is a summary of the design process, the key questions asked in each phase of the diamond, and the corresponding User Experience research methods I am not going to have enough time today to go through all the methods. The purpose of this slide is to give you a summary view about the breadth of the work that is done in design and UX. You can refer back to this diagram later if you hear about certain methods. The diagram helps you to situation each method with the kind of questions it can answer for you.
- I’d like you to pause for a second here and answer this question, why are you interested in AR? Is it because you see the opportunities of applying AR to solve a previously challenging problem? Is it because AR technology has received so much attention in the recent years, and carries the promise of the new era for human-oriented computing? Or is it because you see that applying your discipline to AR could improve the technology greatly? Or all the above?
- … It’s great that AR/VR starts to reach the mass market. But it does not mean that we can predict the timing. Historically, it is difficult to predict what technology will become mainstream because tech readiness and adoption of the market is hard to predict. Even for the very famous and successful Douglous Englebart Demo, which is called ”mother of all demos”, showing almost all the fundamental elements of modern personal computing: windows, hypertext, graphics, efficient navigation and command input, video conferencing, the computer mouse, word processing, dynamic file linking, revision control, and a collaborative real-time editor (collaborative work). For such a demo that profoundly changed the history computing. It were not able to make immediate impact because it was too far out. Similaryly, VR was first invented in 1968 by Ivan Sutherland and his team. It starts to reach the general public around 2016 with multiple commercial VR devices released. In siggraph 2018, I was very excited to sit in the session where Ivan Sutherland talked about how he invented VR in 1968. He acknowledged the psychological studies that he learned from Bell helicopter company, which confirms that people felt the sense of presence when the rendering camera follows the users’ head movement. When I decided to do my thesis work on AR, I had no idea that it will be widely adopted one day. But I was drawn to the fact that it blurs the boundary between physical and digital world, and all the human-cented questions we need to answer for that mixed reality future. As researchers, I’d like us to forget whether a technology is hot or novel. Instead, focusing our eyes on what human experiences that can be enabled by certain technology or medium.
- …I’d like to talk about another myth of emergent fields like AR, which is the myth about “killer apps” Although the industry has been searching the “killer apps for AR” for many years, it’s not conclusive what the killer app is for AR Assume that there exists a “killer app” for AR, what are you going to do about it? For example, AR has improved the field of industrial training in the past decade. Are you also going to work in the field of training because of it?
- In the field of AR, there are many many opportunities and challenges for us to explore together. What are the design thinking and methods that can help us finding the right problem to solve? I would first encourage you to be more observant about what your own needs, wants and wishes are, but do not stop there, instead, expand that observation to a larger population. This seems to be going against the basic rule from interaction design. For anyone who has taken a course or read a book about user-centered design, we learned that it’s important to understand others’ needs and avoid using our own pre-notions in the user requirement phase because “You are NOT the users”. However, for a nascent field like AR where new paradigms of computing and usage may emerge, the user could be anyone. For example, when Wii was released, it quickly became popular among elderly homes. This demographics were not previously considered as gamers. If the design only considers the traditional demographics of a gamer, Wii couldn’t have reached to the same level of popularity. There are many examples in the tech industry that, a service/application made for oneself initially can be widely adopted by others. Airbnb started from the two founders renting their apartment in SF to others by night. Instagram started as a class project for computational photography inspired by a vacation trip. If it is useful for one person, the chances are, it may be useful for a group of users. But how do we know? There is a method called Jobs-to-be-done, which originates from marketing research. The inventor of this method is Clayton Christensen, who was a professor from Harvard business school and known for his work in disruptive innovations. He created this jobs-tob-be-done method to understand the reasons behind what people choose to buy and use. For example, he talked about using this method to understand why people buy a milkshake from a fast food company. They identified their target milkshake-slurping demographic and sent researchers to analyze this audience's milkshake preferences. Unfortunately, once the fast food company began making new, evidence-backed and "better" milkshakes based on the research findings, they discovered milkshake sales didn't improve, at all. Prof. Christensen asks a very different question than the target user demographic approach, instead, he asks, “ what kind of jobs that people try to get done when they ”hire” a milkshake?” The researchers would talk to the users in this framing of hiring milkshake for jobs, and ask about what are the other things they may hire to get a similar job done. People may have hired a banana, a donut or a bagel to do the same job, but milkshake seem to work out better because it fills people up during a boring morning drive, and they fit well with the cupholder and does not leave messy crumbs. What’s intriguing about the jobs is that, it is the cause behind many technical solutions. As we all know, the solutions may change over time, the fundamental jobs remain stable. Marslow’s 7 layers of need hierarchy holds very well against numerous tools and services we see these days. For example, on the social need level, the social norms and etiquettes remain largely stable, guiding how people perform with all sorts of communication tools, allowing us as researchers to look at the problem space not empirically but also from by the guidance of existing theories. What’s also interesting about jobs to be done is that, although the jobs are stable, the combination and prioritization of the jobs may vary greatly. Take the Airbnb example, one big reason of its wide adoption, in my opinion, is that it incorporates a new job, enabling hosts to gain additional income. Historically, the take off of the Airbnb overlaps with the timing of the 2008 economic crisis. The priority of the job became more important.
- Another inspiration of choosing what to work on is the disciplinary knowledge that each one of you have. What’s the problem in my discipline that can be addressed by new technologies like AR? What are the challenges in AR that can be uniquely addressed by my discipline? There was a paper in CHI 2016 that illustrates the usefulness of thinking in the lens of the discipline. The authors created an eye-tracking based tool to segment volumetric microscopy neuron cells. Sorry that I can’t directly attach the image from the paper because I don’t have the copyright to the images. Instead, I put a picture from Wikipedia about a microscopy image for some cells in human body. I don’t know much about what these cells are. But imagine your goal is to segment each of these cells in this image by a mouse and keyboard, or touchscreen, and have to do this for hundreds of frames, it’s probably very repetitive and tedious. But with eye tracking, they leverage human eyes superb capability of keeping track of objects overtime to reduce the burden greatly. I couldn’t come up with such as great usage for eye tracking myself in a million years. The deep domain knowledge that each of you have in your discipline is a big source of inspiration when it comes to AR innovations. Moreover, these knowledge does not have to be limited to one particular domain. For example, the eye tracking based segmentation and labeling can be more useful beyond microscopy data; it could be useful to all sorts of data labeling practice in general.
- Another dimension of inspiration comes from the work in Facebook reality labs. In Michael Abrash’s Facebook Connect 2020 keynote, he talked about what an ideal AR interface would be like, it will be truly personalized and contextualized, that opens the door to an interface that is proactive rather than reactive, that understands our intent, and acts almost before we know it. With interface like this, the interaction is ultra-low friction, the distance between thought and action is minimalized, and people can stay present in their everyday life. For example, people are used to taking photos and videos of their everyday life these days. But sometimes we take photos just to keep track of things, and sometimes we take photos to keep valuable memories. For the first category, can we minimize the interaction that’s required to take the photos, because these photos does not have much emotional value; for the second category, can we automatically capture these valuable enjoyable memories so that we can fully be present with the people that we share these moments with. We all know that it has been the focus of many existing research that focuses on high degrees of freedom interactions with 3D objects, and it still have many important and interesting questions to solve in that space. But on the other side of the physical-digital reality spectrum, it is equally important, if not more, to design interfaces that does not take people out of their moment. We will work on the innovations across the spectrum of light-weight engagement and deep engagement with the computing technology.
- Now that we’ve gone through several design thinking and methods for the first diamond, which is about discovering and defining the problem space. And I hope that this would inspire and support you to think about what you are going to make with AR, by being more observant of your own needs or the needs from people around you, but not limited to the small group; by thinking about the domain knowledge and challenges in your discipline; and by expanding the notion about AR innovations across the spectrum of light and deep engagement of computing.
- Now I am going to move on to the next stage of the diamond model, about creating and iterating the solutions. How do we design for usable AR Experiences?
- User-centered Design
- Designers know their design materials. If we are painters, then we need to understand how different paint, like oil, watercolor, acrylic works. Similarly, when making new things with AR or for AR, we need to understand AR as a kind of design materials. What makes AR unique and exciting is that it enables embodied experience. Embodied experience refers to the type of experience that a person can leverage their existing… skills Bodystorming: in contrast to brainstorming, during a bodystorming, the person imagine what it would be like if the product existed, and act as though it exists, ideally in the place it would be used. You could use low fidelity tools such as post-its and markers to simulate where the interface may be attached. AR systems are equipped with a lot of sensors and AI algorithms, therefore these systems will have the capability to understand context very well. Instead of just imagining the future where the context recognition is accurate, we can use wizard-of-oz method to try to simulate the algorithm. Wizard-of-oz uses a human proctor instead of some intelligent algorithm to test an experience without fully implementing the system. For example, we can use wizard-of-oz to test different levels of smart assistance. This method can help us forecast the experiential outcomes before the algorithm is made. AR systems are oftentimes wearable and mobile. How does it feel to wear something on body? We can use methods like diary study or life logging to collect people’s feedback about a wearable system through out the day. In these methods, the participants are prompted to record episodes of their everyday life in text, video, or surveys. XR systems enables us to experience life in other’s perspective. The empathic computing lab lead by Prof Mark Billinghurst has been conducting the leading edge research on this topic. Their method is to iterate through different types of fidelities of technology. In their experiments, the task remain largely the same. A remote expert is helping a person onsite by seeing what they see and communicating through annotation, speech and gesture. One challenge is how the remote expert experience the onsite person’s space. They iterated from 2D displays, to 360 view videos, and then to 3D dynamic geometry in the world. With each iteration, they increase immersion, enabling more embodied spatial experience more and more so for the remote user.
- The approach that Prof. Billinghurst’s lab has done with the iterative design goes deep into each and every one of these representations of reality they have tried. There are nuanced differences between specific AI/AR algorithms. Fast iterations that focus on the core minimal viable product will enable the team to experience the outcome of the design and adjust quickly. In short, get hands dirty with the material of AR. From 2008 to 2011, I helped to teach the course of mobile AR games. Through design research, we found that the teams that start on-device prototypes earlier tend to have a better rated design quality at the delivery; confirming the importance of fast iterations with the specific AR or AI algorithms. Last but not least, I want to mention design pattern research method. You probably have heard of the software design patterns. It was originated by an architect, Christopher Alexander, who summarized over 200 patterns from architecture design. A design pattern is about repeated solutions for reoccurring problems. As the field of XR develops fast with the research and developer community, more and more design examples will emerge. Ssynthesizing the design patterns will inform both the problem space and solution space.
- AR is built on sensor data. These data not only helps to track the human body in the physical world, but also provides understandings about the world and the user. Therefore, we may encounter new data challenges
- Usually this wouldn’t be a problem. Design comes first and data is gathered when using the system that’s designed. However, this assumption may not be true if the design space is huge. For example, when I was designing an app that allows users to fit HMD efficiently, I faced a huge solution space: the person can adjust the headset by rotations from yaw, pitch, roll axis; they can tighten the headband or swap the nose piece or forehead piece. The first draft of the design sketch is ridiculously long because of all the possible adjustments. I realize that the solution space is too large to make an effective and efficient fitting exprience. So the first thing I did was to collect data so that I can triage the design space. I collected data with users that go through all possible combinations of headset adjustment. I recorded their eye positions in relation to the headset when a user goes through these conditions. With the data, I was able to triage more than half of the adjustment dimensions and prioritize the most important ones: the pitch of the device (meaning how high or low the headset sits on the back of the users’ head); the nose pad size and also the forehead pad. Once the solution space is reduced, my team were able to put together a quick MVPrototype with basic graphics but fully interactive. A user could adjust the device and see a cross that moves in real time. With the initial prototype, our design was to try to fit the user’s eyes to an 3D volume we calculated from the display and camera configurations. In this volume, we anticipate the users eyes can view the display comfortably and eye tracking works well. We thought this approach could work. However, it didn’t. Confirmed by the data from testing the prototype with ~50 users. When the data tells a different story than we expected, we listened to data. We complete re-designed the prototype using different data source and different decision flow. And we continued run the tight iteration between design and testing for about 8 rounds. In the end of the day, we were able to make a fitting app that the majority of people can finish in 2 minutes on average while the KPI was 7 minutes. On the hindsight, an alternative of this approach is to solve the problem computationally? Can we collect enough personalized data of the user during the purchase process so that every device is configured perfectly for individuals. I believe it is possible with machine learning approaches. That would be the ultimate for of data changing design.
- If you are a designer who work with AR or AI, chances are, you will need to encounter probabilistic data Never use the output of the algorithm without checking its confidence level. We also need to empirically test the human threshold of what’s usable. For example, the auto-correction algorithms only become usable when it reaches over 98% accuracy. For example, in the eye tracking pipeline, there are outputs at each step in the pipeline. You can get the 2D position of the pupil in an image space from an early step of the pipeline or the 3D position of the eyeball center from a later step. There are pros and cons of the data at different steps. For example, the 3D eyeball center is more stable, not affected by the rotation of the eye. The 2D position of the pupil is less stable but it is more reliable, has less dependency with other parts of the pipeline, such as glint detection. As a designer, you need to weigh in the trade-offs between these different data. More importantly, think about what part that design can help through interaction and interfaces, and which part is less influenced by the design.
- Precision and recall
- Precision and recall
- Precision and recall
- Precision and recall
- Precision and recall
- I am going to conclude my presentation with AR is an exciting field that full of new challenges and opportunities. What are you going to make?
- Tech.fb.com AR/VR Facebook Connect Talks on youtube