Future of Mobile Augmented Reality - Zenitum's internal view point and the ultimate goal

1. March 2012
Albert Heekwan Kim
Founder & CEO
mail : albertk@zenitum.com
twitter : @hkalbert

2. Market - Vision
AR Market will be Gigantic
predicted by market research firms around the world
ABI Research said We say...
Gazillion dollar Market!!
$350 million in 2014
Juniper Research said
$732 million in 2014
ARC Chart said
$2.2 billion in 2015*
It could be worth
1,000 time s more!
* Mobile AR application market size in US alone 2

3. Market - Vision
Evolution of Computers
Computing paradigm evolved to process more information
More computing power is needed to process the era of
highly realistic visual data for pleasing human eyes Visual Computing
Most power goes to
• Highly realistic graphics
• precision vision tracking
• visual representation of data
3

4. Market - Vision
AR naturally mixes Real & Virtual
thus people intuitively understand the world digitally
Tablet Devices
AR Goggles
All you need to do is to look at the objects!
Bionic eyes for everyone!!
4

5. Market - Vision
people play games inside of Mixed Worlds
where the virtual & the real co-exist seamlessly
Coca Cola Commercial (2009)
Real & virtual are all mixed in the Real World (within its physical frames)
5

6. Market - Enabling the Vision
the ultimate AR tracking technology
must compute the 3D coordinates from the real world
Mixed Reality on to the real world real world 3D coordinates & objects detection Street seen by naked eyes
Application
Life-size virtual Processor
Ninja on the street!
Visual information
SCREEN
CAMERA
Virtual Wireless Contextual Information
Contents Sensors
Users (location, environment, etc)
Real/Physical World (Street of Kyoto, Japan)
Mixed Reality System
Mixed Realities via the ultimate visual tracking technology
6

7. Market - Status
are we there yet?
NO! but somewhat stable technology is available now
GPS-only Tracking NFT1 - only Tracking Panoramic Scene with GPS
Small Target Tracking (Vision)
Large Location Tracking (GPS)
• Tracking error rate : 50~90m • Precise visual tracking on targets • GPS for absolute position on the Earth
• AR objects never align with real scene • Only tracks objects near by • Specific target tracking with vision
• Revisit rate is extremely low! • Targets must be processed prior to use • Combine tracking for large & small
• Tracking the viewer, NOT the targets • Targets only. Can’t track 3D space • only works from specific view points
No practical uses! Commercially available! R&D phases, yet
1. Natural Feature Tracking (NFT) : Realtime visual tracking of registered texture patterns on planar surfaces (i.e. magazines, photos, etc) 7

8. Market - Problems
limitations on current NFTs1
1. they always need “pre-processing2” of targets
Target Pre-Processing on NFT Natural Feature Tracking
1. Select a target image
Best to use graphical images (i.e. photos)
Preferably jpg or png file
Takes
2. Extract feature points
30-40 min.
Extracting feature points based on
to generate proprietary algorithm
a target info
Target homography
344, 2334, 34,2223, 45, 45 3. Calculate a “target info”
343434, 4454, 332, 444, 5
4533, 22, 1,2 33 Calculating images from many different
angles, and out put “target information”
And then, AR App Development
• Precise visual tracking on targets
Develop apps with “target info” • Only tracks objects near by
With the “target info,” developers now • Targets must be processed prior to use
able to code AR apps with “target info”
1. Zenitum, Qualcomm and Metaio provides NFT technologies. They all share similar problems.
2. Pre-processing is required to create a target information out of physical objects. (Training or pre-define targets also used in this document as the same meaning) 8

9. Market - Problems
limitations on current NFTs
2. No more than TEN targets per application
Targets per applications
Up to 10 targets
Up to 7 targets
Up to 7 targets
ARAD Client App (2011) - Only able to track up to Seven targets simultaneously
Business Models with
1) FEW targets simultaneously
• AR games (with less than 10 cards)
• AR Advertisements (with less 10 targets)
2) Thousands targets simultaneously
• Massive Ads on entire product lines (i.e. P&G products)
• Interactive Ads & games (i.e. Games on entire Kellogg's cereal boxes)
• AR books for entire publishings (i.e. Video lecture on entire Penguin Classics)
• Massive AR search (i.e. Dietary video info. on all products in supermarkets)
• AR knowledge search (i.e. AR contents on books in libraries) the user clicks the virtual button, then related on a book shelf, relation lines pop up when
• and more web pages pops up next to the book pulling out a book.
9

10. Market - Steps into Mixed Realty
overcoming techno-barriers
then evolve into new paradigm : Mixed Reality
NOW Solutions
Target Pre-processing1 Allowing only Few Targets2
(~2011) but Problems • annoyance to developers & users • annoyance to developers & users
• limiting business models around AR • limiting business models around AR
TO BE Interim
Large number of Instant Target Generation
(2011~) Solution • Instant target generation by either developers or users
• allowing vast number of targets to cover many business models
GOAL Ultimate
Natural Scene Tracking
(2013~) Solution • understands the physical space & generating 3D coordinates in real-time
• mixing virtual data on the physical space for mixed reality interaction
Mixed Reality - Visually indistinguishable mix of Real & Virtual
1. refer p. 7 for more on “Target Pre-processing” 2. refer p. 8 for more on “Tracking few targets” 10

11. Market - Macro AR Technology Trends
To Reach AR Event Horizon
Two Major Infrastructure Must be completed
Massive AR Content Creation & Distribution
Instant (on-line) Target Generation
Massive Target Search (preferably video streaming search)
Natural Scene Tracking
Hardware Innovation (GPGPU, AR Core, Multi-Cameras)
COTS Devices (AR Goggles)
11

12. Market - At the end of the day
AR market will be explode!
as the technology matured More!!
Natural Scene Tracking
AR as the ultimate HCI!!
1st Wave 2nd Wave
GPS based tracking NFT with target training
Early adopters & hobbyists Real business applications
3rd Wave
Instant Target Generation
Tracking many objects
Market Size
New applications (i.e. SNS)
2008 2010 2011 2012
12

13. Zenitum, NOW - Technology
AR Tracking Engine #1 : zFT
Pre-assigned targets tracking with NFT
Performance : 15 ~ 30 frames per second
Min. CPU requirement : ARM11 / 700 MHz or higher
Pre-assigning a target takes about 30 minutes
Max. number of targets per application : 10 (Recommends 7)
Platforms supported : iOS, Android, Win7, MacOS
Zenitum Feature Tracker (zFT) is a natural feature tracking Multiple target combination available
engine that tracks 3D movement of planar targets(i.e. Some portion of zFT cab be embeddable to microchips
books, magazines, product packages, etc) (tested on Altera Stratix II FPGA DSP)
zFT is commercially available NOW.
Available Now!
13

14. Zenitum, NOW - Technology
AR Tracking Engine #2 : AR Goo
Instant target generation and tracking
untrained object (a ceramic coaster) Placing AR content instantly!!!
click!
TM
AR Googeneration AR Tracker
Instant Target
2nd generation NFT engine with no target pre-assigning
Targets can be generated instantly on user
Max. number of targets per an application : 5
Engine available for iOS, Android and PC platform
Some portion of ARGoo, zarCORE is available for microchips
(tested on Altera Stratix II FPGA DSP)
solving problem #1 14

15. Zenitum, NOW - Technology
Massive Target Search Engine : zARTIS
fast target image search and find
stream of images AR Contents
Content
AR Streamer DB
Service
Server zARTIS
Zenitum zARTIS
1) search result AR Target DB
(target info.) Image Search
2) AR contents
TM
zARTISimage search engine
massive target
Works with either zFT or ARGoo tracking engines
zARTIS support for unlimited targets
(up to 15,000 targets per channel)
Each application can have unlimited number of channels
Content builders easily create targets and AR contents
through web interface
solving problem #2 15

16. Zenitum, NOW - Technology
AR Tracking Engine : D-Track
1st attempt for the space & depth tracking
Initial depth calculation by users’ cam movement A planar 3D filed is generated for 3D contents mix
D-Track I LE D!
FAD-Track
TM
Space and Depth tracking #1
How to initialize R&D from May 2008 ~ Sept. 2010
1) Run the App with D-Track Oxford’s “PTAM” licensed algorithm based
2) Swing the device Mixture of PTAM tracker and zFT Detector
- left & right Implemented on Samsung Omnia II with Windows Mobile
- back & forth Every initialization result is differ from one another
3) Generating 3D planar coordinate
30% of initialization failed miserably!!
Heavily depended on the initialization process Not ROBUST enough to be commercialized
16

17. Zenitum, NOW - Technology
AR Tracking Engine #3 : SceneTrack
finally, mother of all tracking for Mixed Reality
Tracking starts with one object 3D coordinate extended 3D coordinate extended (3D coordinate) Gradually includes neighboring space
TM
SceneTrack
Robust Space & Depth tracking
R&D since Sept. 2010 ~
Mixture of AR Goo engine & SfM (Structure from Motion)
How to initialize Start with one object (i.e. magazine, photo, or product pkg)
1) Run the App with SceneTrack The tracking space gets larger as the camera moves
2) Choose an image as an initial position
3) From the image, move the camera to For max. performance, users can add images in the space
generate 3D coordinate of larger space as 3D coordinate stepping stones
4) More images can be selected & work as
stepping stones for larger space tracking
Every user has exact same virtual coordinate the interim solution 17

18. Zenitum, NOW - Technology
AR Tracking Engine #4 : zarCORE
direct hardwired engine with steroid
3rd Party APP APP APP APP APP APP
Applications
AR Goo Algorithm
ARGoo SDK
Application APIs
Selected
algorithms 3D Renderer Game Engine
AR Goo
zarCORE Android iOS Windows Phone, etc
Application
Processors Direct Access to zarDrive
Hardware side Software side
TM Selected high speed computation algorithms from ARGoo
zarCORE Directly hardwired onto application processors
Boosting whole performance in 150 folds
IP Core for Application Processors - 150 times faster calculation
- Very precise tracking
Leaving CPU for other computationally rich tasks
- Realistic 3D Graphics Rendering
- Scene Tracking (if developers use SceneTrack)
18

19. Zenitum, NOW - Technology
AR Tracking Engine Evolution
perspiration, despair and triumph
ARGoo zarCORE SceneTrack
2.0~ zARTIS 1.0~ 2.0~
1.0~
2012
zarCORE SceneTrack
1.0
2011 zFT ARGoo
4.0 1.0 zARTIS
SceneTrack
Core inheritance
2010 ARGoo
Non-core inheritance
D-Track
1.0
Depth Tracking (Open source based)
2009 zFT
2.0~3.0 D-Track Open Source Tracking engine
licensed
2008 ARCORE Natural Scene Tracking
1.0 miniPTAM
zFT NFT (Instantly created target)
1.0
ARCORE PTAM
2007 NFT (pre-processed target)
zFT
Marker based Tracker
2006 Frozen
MobEYE
Abandoned
2005
Marker based Markerless Instant Target Generation AR IP Core Natural Scene Tracking
19

20. Zenitum, FUTURE - Evolution
Gradual Shift to B2C Market
Current AR technology is not for the mass users
lems s olved!!
ob
PrTechnology (to be)
AR Technology (now) AR
Tracks only tiny target space Tracks the real 3D spaces
(i.e. magazine pages, street billboards) (i.e. desktops, rooms, buildings, streets)
Tracks less than 10 targets Tracks many targets at any given time
(i.e. 10 cards, 10 pages of Magazine) (i.e. AR contents on products on aisles of super markets)
Every target must be pre-processed Generate targets in real-time
(average target processing time : 30 min./target) (pointing objects to generate targets in real-time)
Novelties & Promotional Mass user ready!
Sample demo AR applications Zenitum already solved problem #3
Simple mobile AR games with less than 10 targets and partly solved problem #2 and #1
Mobile AR advertisements with less than 10 targets
20

21. Zenitum, FUTURE - Geozet
Geozet -Wold Decorator
Massive User Generated AR Contents Sharing Service
Geozet
All free, user generated AR content sharing
Creating generated AR With Geozet client, users AR decorate real world
Search targets for AR
Users collect targets, iPhone/Android - select a target
then add AR contents Watching targets with - add AR contents (thus an AR combo)
- store it on Geozet server for later search (by zARTIS)
- 3D files
- Audio/Video
Geozet Client Geozet & zARTIS - share it with friends
answers
- Texts/Images - detection results
- current POIs - tracking information Unlimited number of channels per users
zARTIS - AR contents - 15,000 targets per channel
- either collected one at a time or all targets can be dumped at once
- shared with friends (Facebook, Twitter and Geozet account)
Hosting Premium Content Partner(PCP) Channels
Users authors AR Geozet Server - Inviting premium contents providers for higher user repeat rate
contents over the map - Potential partners : Large media, Consumer product manufacturers,
- 3D files Restaurant Chains, Museums, Governments/NGO
- Audio/Video
- Texts/Images
- POIs
Geozet.com Unfair Advantages
- ARGoo : Instant AR content creation
- zARTIS : Massive # of Targets created & shared
Creating generated AR - Geozet API : Interactive AR contents are easily built
Geozet Web - Geozet Billing : Revenue Generation for 3rd party
- It’s All FREE!!
21

22. Zenitum, FUTURE - Geozet
ario
s
Geozet -Wold Decorator
e n 3D AR Contents on licensed products
e Sc
Usag
22

23. Zenitum, FUTURE - Geozet
ario
s
Geozet -Wold Decorator
e n Massive AR contents on Supermarket aisles
e Sc
Usag
23

24. Zenitum, FUTURE - Geozet
ario
s
Geozet -Wold Decorator
e n Merchandizes information & promotion on the fly
e Sc
Usag
24

25. Zenitum, FUTURE - Geozet
ario
s
Geozet -Wold Decorator
e n Style & location guide
e Sc
Usag
25

26. Zenitum, FUTURE - Products (from 2013)
MMOMRG Platform Service
Massive Multiplayers Online Mixed Reality Game Service
MMOG MMOMRG
Massive Multiplayers Online Games Massive Multiplayers Online Mixed Reality Games
a multiplayer video game which is capable of supporting a multiplayer mixed reality game which is capable of
hundreds or thousands of players simultaneously. supporting thousands of players simultaneously.
played on the Internet played both on the Internet and the Real world
feature at least one persistent world feature multiple layers of persistent world on the Real world
played on PC, consoles and mobile devices played on any devices with cameras for real world tracking
Merging two worlds and Gaming everywhere
26

27. Zenitum, FUTURE - Products (from 2013)
MMOMRG : project Nabokov
plunge your self into Mixed Worlds
Real world Tracking Gestural User Interface
Real world Tracking Creates 3D coordinate Hands Tracking Gesture Tracking
SceneTrack from Zenitum EgoMO from Zenitum
Wearable Displays MMOG Platforms
Project Nabokov
• started July, 2010 (to replace D-Track engine)
• SceneTrack ver. 1.0 released July, 2011
Ultra Light HMD Contact Lens Display • ARGoo + 20K object search/client will be released in Dec. 2011 MMOG Server Platform
• Beta version of Nabokov launched in the early 2013
HMDs from Brother, Vuzix, NTT DoCoMo (able to track up to 711 ft2 ,or 66 m2 indoor space first) MMOG Platform from existing 3D games
27

28. Zenitum, FUTURE - Products (from 2013)
MMOMRG : project Nabokov
escape & venture to the virtual worlds on earth
Mixed Reality Games
project Nabokov City wide virtual layers of game worlds
Real spaces (i.e. rooms, buildings & cities) become game worlds
Computer generated virtual worlds overlaid onto the real world
The physical world becomes background frames for MR games
MR designers replace real ads. with virtual ads.
Shop owners can offers free/paid virtual goods within their shops
Ordinary street billboards can be game targets for more engagement
Once logged in, a user scans her room to extract 3D coordinates
MRG system stores the coordinates as virtual worlds
The user can download spatially matching games for free/paid
worm holes
between layers
Mixed Reality #n : Design your own fantasy worlds
Mixed Reality #3 : Dystopian future action games
Mixed Reality #2 : Zombie Street with FPS games
Mixed Reality #1 : Medieval role playing games
Real & Physical World
28

29. Zenitum, FUTURE - Products (from 2013)
MMOMRG : project Nabokov
a huge eco-system for contents developers & users
Purchasing Goggles/HMD
the contents
3D Contents MMOMRG Platform
Providers
Nabokov Server Life-size 3D contents
Freemium Developer API
Game
Developers
Contents World MMO Nabokov
Client Real & Physical Space
Education Server Server Server
Contents
Tablets/Smart phones
Adult Virtual Currency Platform
Contents Real & Physical Space
Virtual currency
& more
Virtual Currency to Real Currencies Life-size 3D contents
Nabokov
Interface to other payment services (i.e. PayPal) Client
Content Developers MMOMRG Platform MMOMRG Users (selecting game worlds)
29

30. Zenitum, FUTURE - New Strategy
Addressing Massive Users Directly
tapping into large users with right AR technology
B2C (Freemium)
MMOMRG Platform - Nabokov
Nabokov Developer API (zMART w/ SceneTrack)
Geozet.com (DIY AR + Location based)
B2B (Licensing)
ARAD Platform (w/ ARGoo + zARTIS)
zMART SDK (w/ zFT) zMART SDK (w/ ARGoo)
zarCORE v. 1.0 (w/ ARGoo for chip makers) zarCORE v. 2.0 (w/ SceneTrack for chip makers)
2011 2012 2013
zFT ARGoo SceneTrack
• Requires Pre-processing • No Pre-processing • Tracks real 3D space
• Tracking small targets • Tracking small targets • Instant target generation (w/ ARGoo)
AR Technologies
• 5 targets/app • 5 targets/app zARTIS (v. 1.0) • Massive targets and Scene tracking zARTIS (v. 2.0)
• Massive target processing (w/ zARTIS ver. 2.0) • Massive target processing
• Real-time target recognition • Real-time target recognition
• ~10,000 targets /channel • ~60,000 targets/channel
zarCORE (v.1.0) zarCORE (v.2.0)
• ARGoo based IP Core • SceneTrack based IP Core
• AR features on microprocessors • AR features on microprocessors
• ~100x better than SW only • 3D space tracking on chip level
2011 2012 2013
30

31. Zenitum, FUTURE - Target Markets
MMOMRG Platform targets
global mobile and online game market
$44 Billion in 2014
50% of Global Game Market (est. $87 b.)
MMOMRG Platform
(project Nabokov)
Light & Casual Gamers Heavy & Hardcore Gamers
Professional Productivity Tools Educational Contents Object/Location based Knowledge Management
Mobile commerce 3D Adult contents Tourism Religious Contents Immersive Interfaces 3D Advertisement
& other unexpected uses...
31

32. Zenitum, FUTURE - Business Strategy
Cross Platform Reinforcement
Helping SW-HW support each other to create more values
Providing based algorithm core
Software Algorithm Hardware Algorithm
SW Tracking Engine Engine IP Core
New & Improved Algorithm
Offering AR features on mobile devices
New requirements & opportunities
Cross platform AR Developer tools
Application Processor Sales
project Nabokov
MMOMRG Platform
New AR/MR applications run on the AR devices
3rd Party Developers New AR Devices
zMART SDK OEM SDK
More AR optimized devices to be used for the applications
32

33. Appendix
Technology-Market Relation
addressable market grow bigger with better solutions
Mobile AR Games or Ads
Mobile SNS or SNG
AR object tracking & search
AR Education (AR on books)
AR User interface on devices
AR on automobiles
Nabokov Platform
• up 100,000 targets/client
• No pre-processing of targets zarCORE
• 500 times faster tracking
• practical AR based services NEW AR devices
Market Size
Mobile AR Games or Ads NEW entertainment
Mobile SNS or SNG NEW education
AR object tracking & search Life-size AR on streets
AR Education (AR on books) Real space tracking
Industrial applications
Mobile AR Games or Ads
• up 20,000 targets/client
• No pre-processing of targets ARAD Platform (construction & factory sites)
Mobile SNS or SNG
• up 10 targets/app
• practical AR based services zarCORE • up 500,000 targets/client
• tracking the physical space
• No pre-processing of targets • 2,500 times faster tracking
• AR based SNS/SNG • life-long illusion of merging
virtual & real in real-time
Mobile AR Games or Ads - virtual lovers, friends & pets
• up 10 targets/app
• pre-processing is MUST ARAD Platform - immersive education
• Simple but novelty app only
Instant target generation
& other NFT Technologies
2010 2011 2012 2013 33