Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Reality As Your Next Build Target, Mobile AR, and the Future of Authoring

397 vues

Publié le

https://youtu.be/YQlcwrTvDOU
Learn how any creator with an iPhone X will soon be able to leverage the built-in camera to animate their characters' faces for use in any project on any platform. Also, get insight into authoring philosophy and the fun, nitty gritty details of developing digital experiences using real-world data from mobile devices, APIs, and even new wearable AR devices.

Speakers:
Stella Cannefax (Unity Technologies)
Amy DiGiovanni (Unity Technologies)
Jono Forbes (Unity Technologies)
Matthew Schoen (Unity Technologies)
Timoni West (Unity Technologies)


https://unite.unity.com/2018/berlin/

Publié dans : Logiciels
  • Login to see the comments

Reality As Your Next Build Target, Mobile AR, and the Future of Authoring

  1. 1. Reality As Your Next Build Target Mobile AR and the Future of Authoring Authoring Tools Group, Unity Labs Timoni West Matt Schoen Amy DiGiovanni Stella Cannefax Jono Forbes
  2. 2. Unity Labs Animation Augmented and Virtual Reality Graphics Research Future of Game Creation Machine Learning Today we are focusing on the Authoring Tools Group, which has been investigating how Unity will both be used to make the future of spatial computing, and what Unity will look like in that future.
  3. 3. Authoring Tools Group AnimateVR ARView Carte Blanche EditorXR MARS XR Foundational Toolkit
  4. 4. Project MARS
  5. 5. All the world’s a stage, but you don’t know what’s on it • The way you author for augmented reality is uncharted territory • In completely digitals worlds, you know everything that will happen • In partially digital worlds, you can only control the digital • Your experience must be as robust, flexible, and responsive as possible
  6. 6. SUPER robust, flexible, responsive • You need to be able to test, test, test • Test against unusual, inaccessible, or varied environments • Must have all the information about the world to edit directly • Machine learning can help, but is not widely available in a way to do what we need—yet
  7. 7. The challenge with world data • Usually only available in apps on the device after shipping • We need to flip this • Computer vision providers need to have their tech work on many kinds of devices • ML is often tied to specific hardware now—needs to become more ubiquitous and consistent
  8. 8. Jono Forbes UX Lead - MARS Unity Labs
  9. 9. Designing for an unpredictable real world
  10. 10. Use cases • Sick table jump • Essentially the demo you saw • Zeldify the world • Floor -> Water // Tables -> Grass // Walls -> Cliffs • Character enters a room • Semantically understanding a door and a chair • How do I food? • Great semantics & object tracking
  11. 11. Resonai
  12. 12. Building up from nothing • Start with the base layers (floor -> water..) • Design simple queries (big surfaces, high surfaces..) • Then more complex / rare (relationships to define a couch..)
  13. 13. Building up from nothing • Start with the base layers (floor -> water..) • Design simple queries (big surfaces, high surfaces..) • Then more complex / rare (relationships to define a couch..) • Finally, very context specific / trait-based • Analytics will be a big deal for AR devs
  14. 14. Building up from nothing • Start with the base layers (floor -> water..) • Design simple queries (big surfaces, high surfaces..) • Then more complex / rare (relationships to define a couch..) • Finally, very context specific / trait-based
  15. 15. Building up from nothing • Start with the base layers (floor -> water..) • Design simple queries (big surfaces, high surfaces..) • Then more complex / rare (relationships to define a couch..) • Finally, very context specific / trait-based • Analytics will be a big deal for AR devs
  16. 16. Markers
  17. 17. 3D Markers / Geofences
  18. 18. Fallbacks
  19. 19. Procedural content
  20. 20. Procedural content
  21. 21. Amy DiGiovanni Software Engineer Unity Labs
  22. 22. Conditions • Check against real world data • Flexible • Adaptable • Author around the real world Conditions specify what a MARSEntity requires to perform some kind of function
  23. 23. Spatial conditions in the scene view
  24. 24. Miniature worlds
  25. 25. Scale the camera parent “XR Cameras” GDC talk by Matt Schoen
  26. 26. World scale in AR
  27. 27. World scale in the scene view
  28. 28. It’s all relative • MR authoring =/= traditional 3D authoring • The scene is an abstract setup of conditions about the real world • World scaling is necessary to support certain use cases - it must be clear what scale your content is relative to real objects • Positioning of entities is not relevant at runtime, but in editor is meant to convey how the content is spatially related
  29. 29. Stella Cannefax Software Engineer Unity Labs
  30. 30. Contextual Authoring Instead of explicitly designing scenes, think about what context you need in the real world.
  31. 31. Data-Driven Authoring
  32. 32. Data-Driven Authoring Each entity defines only what it depends on.
  33. 33. Data-Driven Authoring Two more big advantages: 1.Easier multi-platform support 1.Simulation & testing
  34. 34. Data-Driven Authoring
  35. 35. Queries (editor)
  36. 36. Performance Modern mobile devices experience performance drops due to heat and processor throttling. Graph is from our Mobile Performance Handbook: http://on.unity.com/2Di8Hl7
  37. 37. Performance MARS strives to be efficient in several ways: • The behind-the-scenes work is distributed across time • Built-in module to run processing tasks on an interval • Managed memory is allocated only when absolutely necessary
  38. 38. Matt Schoen Integrations Lead - MARS Unity Labs
  39. 39. Hardware Camera Pose Surfaces Hit Tests Meshing Faces Markers Relocalization 3D Markers Object recognition Light Estimation ARKit devices X X X (X) X X X X ARCore devices X X X (X) X (X) Tango (defunct) X X X X Hololens X X X Magic Leap X X X X ? X X ? ? X Vive Pro X X X X X Windows MR X Mirage Solo X Santa Cruz X ? ? ? ? ? ? ? ? ? Vive X Rift X Oculus Go (X) GearVR (X)
  40. 40. Software PC Mobile Camera Pose Surfaces Hit Tests Meshing Faces Markers Relocalization 3D Markers Obj rec Light Est Body Tracking Hand Tracking Vuforia X X X X X X X X X 6d.ai X X Placenote X X Selerio X X X ULsee X X X Visage X X Google Mobile Vision X (X) X Apple Vision X X (X) X Wrnch.ai X X X Leap Motion* X X X OpenCV X X (X) (X) X X X X dlib X X somewhere? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
  41. 41. • Faces • Landmarks from (ARKit) face mesh • Expressions from blendshapes • 2D -> 3D landmark poses • Expressions from landmark positions • Surfaces • “Meta-surface” • Elevation / floor • Rotation / alignment • Overlap test • Pause button Room for improvement
  42. 42. Functionality Injection
  43. 43. Functionality Injection
  44. 44. Functionality Injection
  45. 45. MARS Provider Types (so far) • CameraImage • CameraIntrinsics • CameraOffset • CameraPose • CameraPreview • CompassHeading • FaceTracking • FacialExpressions • FunctionalityInjection* • LightEstimation • MarkerTracking • PlaneFinding • PointCloud • ReferencePoints • WorldLocation
  46. 46. Editor Providers • Must run in edit mode • PC / mobile parity • 3D face pose • Markerless tracking • Surface detection • Remoting and Recording • Local testing / debug • Field recording • Multi-user recording • Generated Rooms • ISimulatable and runInEditMode
  47. 47. Reasoning APIs Fill in the missing pieces • Which surface is the floor? • Markers for relocalization • Data correlation • Which face is which? • Which object is which? • More to come
  48. 48. Thank you!
  49. 49. Hacked

×