In this session, Google engineers will walk through ARCore's Cloud Anchors feature, which will soon allow world-scale persistent augmented reality experiences. Cloud Anchors gives developers the ability to create AR experiences that are shared by multiple users both simultaneously and across a large time frame. With the new ARCore Extensions for AR Foundation, it is easier than ever to create shared AR experiences between users, regardless of what mobile device they are using. This presentation will include a special preview from our partners, Sybo and iDreamSky, about their new experience using persistent Cloud Anchors.
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
Persistent world-scale AR experiences with ARCore Cloud Anchors and AR Foundation- Unite Copenhagen 2019
1. Persistent world-scale AR
experiences with ARCore Cloud
Anchors and AR Foundation
John Ullman - Google
Jeff Lyndon - Mark AR
Murari Vasudevan - Mark AR
Ted Bisson - Google
Unite Copenhagen Sep 25, 2019
2. Google @ Unite Copenhagen
Google’s ARCore Cloud Anchors
Persistent world-scale AR Experiences
ARCore Extensions for AR Foundation
Overview
8. Google @ Unite Copenhagen
UndAR the Sea:
Aquatic World in AR
9. Google @ Unite Copenhagen
❖ Each device has its own AR session
❖ Each device has its own Cloud Anchor
❖ Each Cloud Anchor is in the same physical
place
❖ Together they establish a shared reference
frame
❖ Place content relative to the Cloud Anchor
for a shared experience
How does it work?
Shared reference frame
10. Google @ Unite Copenhagen
1. Scan area around anchor
2. Call hostCloudAnchor() , passing the local
anchor to host
3. ARCore uploads visual data
4. ARCore returns unique Cloud Anchor ID
5. Share ID with other devices
How does it work?
Hosting an anchor
11. Google @ Unite Copenhagen
How does it work?
Resolving an anchor
1. Look at the place where the Cloud
Anchor was hosted
2. Call resolveCloudAnchor() , passing
the Cloud Anchor ID
3. ARCore uploads visual data and
attempts to match to hosted data
4. ARCore creates a new Anchor object
5. On successful resolve, Anchor object
begins tracking with a valid pose
12. Google @ Unite Copenhagen
Place content relative to Cloud Anchors for a shared experience
13. Google @ Unite Copenhagen
Cloud
computing
advantages
❖ More powerful algorithms in the cloud.
❖ Less compute, battery consumption on device.
❖ Improvements without app / ARCore updates.
❖ Cross-platform.
14. Google @ Unite Copenhagen
Best Practices
❖ AR in General
❖ Hosting
❖ Resolving
❖ Placement of Cloud Anchors
15. Google @ Unite Copenhagen
Best Practices
AR in General
Sufficient light
Feature rich
Wait for tracking to stabilize
Enough motion
Moving too fast
Reflective surfaces
16. Google @ Unite Copenhagen
Best Practices
AR in General
Sufficient light
Feature rich
Wait for tracking to stabilize
Enough motion
Moving too fast
Reflective surfaces
17. Google @ Unite Copenhagen
Best Practices
AR in General
Sufficient light
Feature rich
Wait for tracking to stabilize
Enough motion
Moving too fast
Reflective surfaces
18. Google @ Unite Copenhagen
Best Practices
AR in General
Sufficient light
Feature rich
Wait for tracking to stabilize
Enough motion
Moving too fast
Reflective surfaces
19. Google @ Unite Copenhagen
Best Practices
AR in General
Sufficient light
Feature rich
Wait for tracking to stabilize
Enough motion
Moving too fast
Reflective surfaces
20. Google @ Unite Copenhagen
Best Practices
AR in General
Sufficient light
Feature rich
Wait for tracking to stabilize
Enough motion
Moving too fast
Reflective surfaces
21. Google @ Unite Copenhagen
Place anchors near visual features
Move around to see the anchor from
different perspectives
Cover areas you want to resolve from
Call host() soon after scanning
Avoid precisely repeating patterns in
the environment
Best Practices
Hosting
22. Google @ Unite Copenhagen
Best Practices
Hosting
Move around to see the anchor from
different perspectives
Cover areas you want to resolve from
Place anchors near visual features
Call host() soon after scanning
Avoid precisely repeating patterns
in the environment
23. Google @ Unite Copenhagen
Best Practices
Resolving
Look at anchor from similar position to
where hosted from
Move around to see from different
perspectives
24. Google @ Unite Copenhagen
Best Practices
Placement of Cloud Anchors
Use single Cloud Anchor for content
within a small area
Use multiple Cloud Anchors for pieces
of content that are far apart
5°
25. Google @ Unite Copenhagen
❖ Hosting
➢ Buffer 10 seconds -> 30 seconds
(without increasing data size)
➢ Much lower latency
Improving the Cloud Anchors user experience
Quality improvements launched in ARCore 1.12 this month
❖ Resolving
➢ Continually refining pose
➢ Resolve up to 20 anchors at a time efficiently
➢ Accuracy improvements (ongoing...)
➢ More robust to lighting changes
26. Google @ Unite Copenhagen
Unlocking new user experiences with persistent Cloud Anchors
“Save button” for AR
29. ❌ Social -> Single User
❌ Persistent -> One Time Use
❌ Accurate-> GPS
WHAT WAS AR LIKE BEFORE?
30. WHAT IS MARK AR?
As one of the world’s first social AR platform
Mission: Create the second layer of the world through
letting users create and share their own AR Art anywhere
in cities
31. ✓ AR Save Button
✓ Sharing
✓ Collaboration
MARK AR + SOCIAL
VIA PERSISTENT CLOUD ANCHORS
35. Scanning stage for 2 purposes:
1. Detecting AR surfaces
2. Improving hosting and relocalisation
Tapping on a desired surface:
1. Creates a canvas at that plane anchor
2. Hosts the Cloud Anchor
BASIC FLOW:
CREATION / HOSTING
IMPORTANT: When you host a Cloud Anchor, ARCore
uses the last 30 seconds of feature points for
relocalisation purposes.
36. To find art in the world, there are 2 discovery
methods:
● Map View - shows you the most
relevant sprays in that location
● Feed View - shows you a scrolling list
of sprays, orderable by distance,
trending and time of creation
Based on your distance from it, you get
walking directions or switch to AR view for
relocalisation Map view Feed view
BASIC FLOW:
DISCOVERY
37. AR View:
Relocalisation approaches:
● Past: Milk Carton approach: Thumbnail images of
the real world background as “hints”
● Present: Video capture of the area, so users can
find it easier
● Future: Localize from a larger area, to guide users
to the content in AR
BASIC FLOW:
RESOLVING
38. For the actual 3d painting, we modified an Asset Store
package called Paintcraft Pro:
● Using the camera object as the raycast source for
the brush
● Where the raycast intersects the 3D canvas, brush
images and noise textures are blitted.
● Spline interpolation is performed between the line
drawing points
● Node-based pipeline for creating different brushes /
cap types
VIRTUAL SPRAYPAINTING
IN UNITY 3D
39. Custom rendering with Overlay blend shader
To get the spray to blend into the background, and bring
out textural detail of the surface underneath
● Overlay shader is a combination of both Multiply and
Screen blending.
● Darkens the dark areas and brightens the bright
areas
RENDERING
40. ● Creating content in AR is hard, so we wanted to
make it easier for users, but still fun.
● We chose to implement Stencils
○ User can position, rotate and scale stencils
○ Multi-layered stencils for more complex pieces
● Users can create custom stencils, from pictures &
selfies and express personal style
● Text stencils for quick messages, with different fonts
AR CREATION
SIMPLIFIED
41. 1. Expansion on painting canvas/surface
2. Advanced creation tooling
a. Custom stencil creation (pictures, selfies, etc.,)
b. Typography / Lettering tools
c. Effects, Animations, 3D
3. Offline creation
WHAT NEXT?
42. Proprietary + ConfidentialGoogle @ Unite Copenhagen
Ted Bisson
Google
Apply to collaborate on persistent Cloud Anchors: goo.gle/2kvnjGt
44. Google @ Unite Copenhagen
ARCore SDK for Unity and AR Foundation
C# Extensions
Adding ARCore Extensions to Your Project
Using Cloud Anchors
Agenda
45. Google @ Unite Copenhagen
ARCore SDK
for Unity
Over a year of development and support.
Vibrant community of developers and apps.
developers.google.com/ar
46. Google @ Unite Copenhagen
AR Foundation
AR Foundation
com.unity.xr.arfoundation
47. Google @ Unite Copenhagen
AR Foundation
AR Foundation
com.unity.xr.arfoundation
AR Subsystems
com.unity.xr.arsubsystems
48. Google @ Unite Copenhagen
AR Foundation
AR Foundation
com.unity.xr.arfoundation
AR Subsystems
com.unity.xr.arsubsystems
ARCore XR Plugin
com.unity.xr.arcore
ARCore
49. Google @ Unite Copenhagen
AR Foundation
AR Foundation
com.unity.xr.arfoundation
AR Subsystems
com.unity.xr.arsubsystems
ARCore XR Plugin
com.unity.xr.arcore
ARCore
ARCore Extensions
50. Google @ Unite Copenhagen
C# Extensions
A language feature that allows you to add
new methods to existing classes, and use
them as if they were built-in methods.
52. C# Extensions Example, Reverse a String
public static class StringExtensions
{
// Method to reverse a string.
public static string Reverse(this string str)
{
53. C# Extensions Example, Reverse a String
public static class StringExtensions
{
// Method to reverse a string.
public static string Reverse(this string str)
{
// .gnirts eht esreveR
return /* reversed */;
}
}
54. C# Extensions Example, Reverse a String
public static class StringExtensions
{
// Method to reverse a string.
public static string Reverse(this string str)
{
// .gnirts eht esreveR
return /* reversed */;
}
}
// Elsewhere in code.
string myString = "Hello World";
string mirror = myString.Reverse();
55. Google @ Unite Copenhagen
Google’s ARCore
Cloud Anchors
Shared frame of references leads
to shared experiences.
56. Google @ Unite Copenhagen
ARCore
Extensions /
Your Project
1. Unity 2019.2 or later
2. Access to Google's GitHub repository
github.com/google-ar/arcore-unity-extensions
57. Google @ Unite Copenhagen
ARCore
Extensions /
Your Project
58. Google @ Unite Copenhagen
ARCore
Extensions /
Your Project
59. Google @ Unite Copenhagen
ARCore
Extensions /
Your Project
60. Google @ Unite Copenhagen
ARCore
Extensions /
Your Project
1. Add the ARCore Extensions object
2. Create ARCore Extensions Config
3. Add references to AR Foundation
61. Google @ Unite Copenhagen
ARCore
Extensions /
Your Project
62. Google @ Unite Copenhagen
ARCore
Extensions /
Your Project
63. Google @ Unite Copenhagen
ARCore
Extensions /
Your Project
64. Google @ Unite Copenhagen
ARCore
Extensions /
Your Project
65. Google @ Unite Copenhagen
Clever
Terminology
AR Foundation ARCore SDK
Reference Point Anchor
66. Google @ Unite Copenhagen
Clever
Terminology
AR Foundation ARCore SDK
Reference Point Anchor
Cloud Reference Point Cloud Anchor
67. Google @ Unite Copenhagen
Clever
Terminology
AR Foundation ARCore SDK
Reference Point Anchor
Cloud Reference Point Cloud Anchor
Cloud Reference Id Cloud Anchor Id
68. Google @ Unite Copenhagen
Adding a Cloud
Reference Point
1. Create a local Reference Point
2. Create a Cloud Reference Point
3. Wait for the Cloud Reference point to be ready
69. Adding a New Cloud Reference Point
// 1. Create a reference point to some location.
ARReferencePoint referencePoint = ReferencePointManager.AddReferencePoint(pose);
70. Adding a New Cloud Reference Point
// 1. Create a reference point to some location.
ARReferencePoint referencePoint = ReferencePointManager.AddReferencePoint(pose);
// 2. Create a Cloud Reference Point.
m_cloudReferencePoint = ReferencePointManager.AddCloudReferencePoint(referencePoint);
if (m_cloudReferencePoint == null)
{
// Error
}
71. Adding a New Cloud Reference Point
void Update()
{
if (m_cloudReferencePoint != null)
{
CloudReferenceState state = m_cloudReferencePoint.cloudReferenceState;
72. Adding a New Cloud Reference Point
void Update()
{
if (m_cloudReferencePoint != null)
{
CloudReferenceState state = m_cloudReferencePoint.cloudReferenceState;
if (state == CloudReferenceState.Success)
{
// 3. Hurray, the Cloud Reference Point is ready to use!
myGameObject.transform.SetParent(m_cloudReferencePoint.transform, false);
}
73. Adding a New Cloud Reference Point
void Update()
{
if (m_cloudReferencePoint != null)
{
CloudReferenceState state = m_cloudReferencePoint.cloudReferenceState;
if (state == CloudReferenceState.Success)
{
// 3. Hurray, the Cloud Reference Point is ready to use!
myGameObject.transform.SetParent(m_cloudReferencePoint.transform, false);
}
else if (state != CloudReferenceState.TaskInProgress)
{
// An error has occurred.
}
}
}
74. Google @ Unite Copenhagen
Resolving a
Cloud
Reference Id
4. Create a Cloud Reference Point from a Cloud
Reference Id
5. Wait for the Cloud Reference Point to be ready
75. Resolving a Cloud Reference Id
// 4. Start with a Cloud Reference Id from another client.
m_cloudReferencePoint =
ReferencePointManager.ResolveCloudReferenceId(m_CloudReferenceId);
if (m_cloudReferencePoint == null)
{
// Error
}
76. Resolving a Cloud Reference Id
void Update()
{
if (m_cloudReferencePoint != null)
{
CloudReferenceState state = m_cloudReferencePoint.cloudReferenceState;
if (state == CloudReferenceState.Success)
{
// 5. Hurray, the Cloud Reference Point is ready to use!
myGameObject.transform.SetParent(m_cloudReferencePoint.transform, false);
}
else if (state != CloudReferenceState.TaskInProgress)
{
// An error has occurred.
}
}
}
78. Proprietary + ConfidentialGoogle @ Unite Copenhagen
Codelab Tomorrow!
Apply to collaborate on persistent Cloud Anchors: goo.gle/2kvnjGt
For more information about Google's ARCore: developers.google.com/ar
Google's ARCore Extensions: github.com/google-ar/arcore-unity-extensions