Clark Dodsworth’s AREvent talk, Santa Clara, CA June 3, 2010: "Context is King: AR, AI, Saience, and the Constant Next Scenario" Mostly about smartphone AR as a gateway to context aware computing becoming indispensible.

1. Context is King AR,-Salience and the Constant Next-Scenario

2. Process of tool design Empathic pragmatismBroad-based tool knowledge / pattern recognition / intuitive Perspective“Product as tool as interface” Tool should fit you. WorkUser-centered product strategy, dev, design.

3. “Ambient Intelligence” strategy Philips, ‘98 all photos: Philips © 2002 Elmo Diederiks

4. Hands-up displays Commercial helmet-mounted displays AR Timeline Efficientpower usage Dedicated tools begin: Commercial aviation 1st car HUD:Cutlass Supreme ARToolkitUW HITLab Rear-view mirrors Useful smart- phone AR Sailboat tell-tales US DoD air-combat testing “Augmentedenvironments,” sensor nets 1stjetfighter heads-up display Nokia’s MARA Now 20?? ~1900 1958 1970s 1974-8 1988 1992+ 1999 2006 Prehistory,soon after invention of the sail. Horse carriages The goal Tom Caudell &David Mizell “UbiComp”:Mark Weiser Sutherland/Sproull, ‘68: 1st HMDs

5. AR Phase One current status Smartphone ecosystem. Hardware & software platforms exist. Infrastructure issues: huge. Power use issues: huge. Privacy/security issues: gigantic. Interoperability issues: roadblocks. ButPhase One AR is just part of transition to mobile broadband as our primary platform.

6. AR Phase Two Smartphone ecosystem: ------The platform for ) ( Pervasive computing Context awareness Semantic Web Ubicomp & sensornets Ambient Intelligence Things that Think ...at last.

7. Alfred North Whitehead-(1861-1947)First augmented reality theorist 1: “Civilization advances by extending the number of important operations which we can perform without thinking of them.” ----------------------------...AR shouldn’t increase cognitive load 2: “The art of progress is to preserve order amid change and to preserve change amid order.” ...AR: contextually dynamic data management & display 3: We think in generalities, but we live in detail.” ---------------------------...AR tasks: managing detail

9. The available data streams must be far better managed. ...It will just happen first on smartphones.

10. AR Phase Two Adaptively deliver info, highly personalized via n dimensions of user states, with high space/time precision, without spiking your cognitive load. Augmented Context + Information Salience = Value

11. AR Phase Two Mobility • Diversity of use cases / user. • User attention = scarce resource. Tool should learn the user. The learning is contextual: context becomes king. • Collect & analyze user’s paths through location, time, tasks, people, queries, interests, decisions... • Accretes • Deliver predictive, scenario-driven support. Result: Adaptive context analysis manages data delivery: What, when, how much, what modality, what priority. -------------------------------------------------------Especially when not-to.

12. Phase Two Context Is King Realtime, context-driven mobile broadband services. “The software layer that learns you.” • Realtime data delivery (live monitoring of processes & states). • Constant predictive analytics, contextual filtering/salience eval. • Semantic Web:better,-partly-automated-search. • Hyperpersonalized database creation, web services. • Digital assistants (Sense Networks, Siri) learn you over time.

13. Phase Two Software • Allostatic control of data delivery: only the data needed, when needed. • Constant autodiscovery of data feeds, sensor feeds. • Constant markerless 2D & 3D feature I.D. and object recognition. • Motion analysis + evaluation linked to the object recognition. • Indoor position mapping: multiple methods. • Nuanced gestural interpretation + integration with voice. • Constant audio + visual awareness with flagging & interpretation.

15. Constant, wide-field visual recording + auto-tagging

16. Constant markerless-object I.D. in 2D-&-3D-+-auto-tagging

17. Constant audio recording + auto-tagging

18. Galileo-grade location (+ dead reckoning, incl. gyros)

19. Ultrasonic& IR-sensors for position sensing, spatialanalysis,modeling, mapping, contextualizing

20. Laser projector + 3D object geometry awareness: -------------“shape-adaptive projection”-for-object-augmentation

21. Phase Two Example Market Entertainment First rule of theme parks: You bring the fun in with you - your friends-& family. • The attractions you pay for aren’t the experience. • Now, a new type of friend. For as long as you want. c. Lynne LaCascia, all rights reserved All images © Chimera Design 2007-2008

22. Phase Two Example Market Entertainment Any licensed character: a useful buddy, a delivery method on top of context-aware, locational, hyper-personalized services. The character behavior, persona, & animation: Character Skin.

23. Dubailand AR project ’07-’08 • Convert concept to smartphone AR + lots of hidden sensors & projectors.• Live & scripted personalized entertainment via licensed characters, + services for hotel, leisure, retail, restaurant, transit,-and-follow visitors home. © Chimera Design 2007-2008 Marvel Superheroes park Part of Dubailand model: c. Lynne LaCascia, all rights reserved

25. Extension of attractions

26. Personal guide/friend

27. Contextuallycustomizedexperiences

28. Interaction with park elements

29. Wayfinding

30. Drill-down for information

31. Social networking: local & global

32. Database: each user’s interests

33. Game play on device & with others

34. Marketing: retail opportunities

35. Long-Term Relationship: “follow the visitor home”All images © Chimera Design 2007-2008

36. Phase Two Biological Model - Consciousness Key functions of consciousness: 1) Constant, integrated analysis & evaluation of surroundings 2) Dynamically judge salience of each aspect of your current situation (state) in context of your goals/needs 3) Constantly construct scenarios for next step Parallel function of AR2: assess, select & deliver salient data • “Contextual evaluation drives contextual augmentation” • “Software ‘awareness’ augments user’s awareness” History + current state = basis of decision-making for next state

37. Phase Two Biological Model The brain’s Default Mode Network • One of the hardest-working systems in the brain • Active only when the brain is notfocused on a particular task • 2 linked brain areas: Empathy re: intent of others,-+ your own state awareness Personal memories + visualization of future scenarios State & intent awareness Scenario modeling

38. Default Mode Network of the brain Posterior cingulate cortex, with the precuneus: accessing personal memories, visualizing oneself in scenarios. Medial prefrontal cortex: Imagining & modeling-the mindset of others. > Who you are, what’s relevant now, and very-near-future scenarios to choose from Brain photo: Omikron/photoresearchers Network overlay: Olaf Sporns/Indiana Univ. (modified by J. Korenblat)

39. Phase Two Implementation Brain Default Mode Network: constantly considers what to do next > Observe states/intents, visualize-scenarios/outcomes > Scenario creation supports each next decision Phase Two mobile broadband services assess, prioritize, & select data ...to support decisions • Realtime data mgmt. helps preserve-focus • Inverse Content Management System (CMS) ...the filter part • Adaptive B2B & B2C service optimization ...the revenue part

40. Facets of Phase Two AR Efficiency & revenue from contextual salience Location Based-Services Constant personal scenario support Autodiscovery:net services &sensor data Integrated sensor,gesture, & voice Hyper-personalization,relationship mgmt. Predictive analytics

41. Facets of Phase Two AR Efficiency & revenue from contextual salience Normal Everyday Life Data interchange standards. Artificial intelligence permeates web svcs. Context awareness. User control of personal data. Constant personal scenario support Autodiscovery:net services &sensor data Integrated sensor,gesture, & voice Hyper-personalization,relationship mgmt. Predictive analytics Better interfaces matter. Probabilistic data delivery. Activity awareness & modeling. Adaptive filtering of realtime data. Resource conservation. Manage the info overload. The opposite of advertising. Sensor datastream analytics. Qualcomm My6sense Aloqua, Xiam Your Name Here Canesta, Apple/Siri Softkinetic, Nuance Corventis Pachube Sense Networks © Clark Dodsworth clark@dodsworth.com

43. clark@dodsworth.com end OA logo Contact for .pptx & speaker notes.

45. Schilit, Bill.; Adams, Norman; Want, Roy. Context-Aware Computing Applications. 1st International Workshop on Mobile Computing Systems and Applications. (1994) 85-90

47. Photo of sail’s tell-tales courtesy Bill Gracey, cc

48. Modern helmet-mounted see-through display courtesy Rockwell-Collins: "SIM EYE SR100-A”

49. Theme park illustrations © Chimera Design, 2007-2009

50. Photo of Dubailand model © Lynne LaCascia, 2009

51. Photo of Alfred North Whitehead from Wikipedia.org