Workshop taught by Mark Billinghurst at the ICIDM 2013 conference about using the Junaio platform for designing mobile AR applications. Presented on December 2nd 2013. Provides material about using Metaio Creator, and Junaio coding for developing marker based and GPS based mobile AR applications.
6. Mobile Phone AR
Mobile Phones
camera
processor
display
AR on Mobile Phones
Simple graphics
Optimized computer vision
Collaborative Interaction
7. 2005: Collaborative AR
AR Tennis
Shared AR content
Two user game
Audio + haptic feedback
Bluetooth networking
10. Evolution of Mobile AR
Camera phone
Wearable
Computers
Camera phone
- Thin client AR
Wearable AR
Handheld
AR Displays
Camera phone
- Self contained AR
PDAs
-Thin client AR
PDAs
-Self contained AR
1995
1997
2001
2003
2004
12. Handheld AR Display - Tethered
1995, 1996 Handheld AR
ARPad, Cameleon
Rekimoto’s NaviCam, Transvision
Tethered LCD
PC Processing and Tracking
13. Mobile AR: Touring Machine (1997)
University of Columbia
Feiner, MacIntyre, Höllerer, Webster
Combines
See through head mounted display
GPS tracking
Orientation sensor
Backpack PC (custom)
Tablet input
14. MARS View
Virtual tags overlaid on the real world
“Information in place”
15. Backpack/Wearable AR
1997 Backpack AR
Feiner’s Touring Machine
AR Quake (Thomas)
Tinmith (Piekarski)
MCAR (Reitmayr)
Bulky, HMD based
16. Mobile AR - Hardware
RTK correction Antenna
GPS
Antenna
HMD
Controller
Example self-built working
solution with PCI-based 3D graphics
PCI 3D Graphics Board
Tracker
Controller
PC104 Sound Card
DC to DC
Converter
Wearable
Computer
CPU
PC104 PCMCIA
Battery
GPS
RTK
correction
Radio
Hard Drive
Serial
Ports
Columbia Touring Machine
17. HIT Lab NZ Wearable AR (2004)
Highly accurate outdoor AR
tracking system
GPS, Inertial, RTK system
HMD
First prototype
Laptop based
Video see-through HMD
2-3 cm tracking accuracy
23. Mobile Phone AR – Thin Client
2003 ARphone (Univ. of Sydney)
Transfer images via Bluetooth (slow – 30 sec/image)
Remote processing – AR Server
24. Early Phone Computer Vision Apps
2003 – Mozzies Game - Best mobile game
Optical motion flow detecting phone orientation
Siemens SX1 – Symbian, 120Mhz, VGA Camera
2005 – Marble Revolution (Bit-Side GmbH)
Winner of Nokia's Series 60 Challenge 2005
2005 – SymBall (VTT)
25. Handheld AR – Self Contained
2003 PDA-based AR
ARToolKit port to PDA
Studierstube ported to PDA
AR Kanji Educational App.
Mr Virtuoso AR character
Wagner’s Invisible Train
- Collaborative AR
26. Mobile Phone AR – Self Contained
2004 Mobile Phone AR
Moehring, Bimber
Henrysson (ARToolKit)
Camera, processor, display together
27. AR Advertising
Txt message to download AR application (200K)
See virtual content popping out of real paper advert
Tested May 2007 by Saatchi and Saatchi
29. Real World Information Overlay
Tag real world locations
GPS + Compass input
Overlay graphics data on live video
Applications
Travel guide, Advertising, etc
Eg: Mobilizy Wikitude (www.mobilizy.com)
Android/iOS based, Public API released
Other companies
Layar, AcrossAir, Tochnidot, RobotVision, etc
31. 2013 State of the Art
Handheld Hardware available
PDA, mobile phones, external cameras
Sensors: GPS, accelerometer, compass
Software Tools are Available
Tracking: ARToolKitPlus, stbTracker, Vuforia
Graphics: OpenGL ES
Authoring: Layar, Wikitude, Metaio Creator
What is needed:
High level authoring tools
Content development tools
Novel interaction techniques
User evaluation and usability
32. Mobile AR Companies
Mobile AR
GPS + compass
Many Companies
Layar
Wikitude
Acrossair
PressLite
Yelp
Robot vision
Etc..
38. AR Browsers
AR equivalent of web browser
Request and serve up content
Commercial outdoor AR applications
Nokia, Junaio, Layar, Wikitude, etc
All have their own language specifications
Wikitude – ARML
Junaio – XML, AREL
41. Key Features
Content provided in information channels
Over 2,000 channels available
Two types of AR channels
GLUE channels – visual tracking
Location based channels – GPS, compass tracking
Simple to use interface with multiple views
List, map, AR (live) view
Point of Interest (POI) based
POIs are geo-located content
42.
43.
44. Try it Yourself
Download Junaio (app store, android market)
Search for Junaio
Run Junaio
To try Glue channels
Download the Junaio Demo Book from
www.junaio.com
Hit scan button, point at QR codes
To try Location channels
Hit search icon in Junaio
Click popular tab, pick channel (eg “Flickr”)
65. Key Steps
1. Download and install Junaio application
From iOS and Android app stores
2. Create a developer account on Junaio.com
3. Download and install Metaio Creator
From http://www.metaio.com/creator/
4. Build sample AR scene
5. Test scene on desktop
6. Publish mobile AR channel
72. Creator Features
Drag and drop AR scene creation
Multiple types of content (2D, 3D, text, video, etc)
Multiple tracking techniques (image, model, point cloud)
Upload to Metaio server space
Offline preview mode
81. Other Demos
Adding other trackable images
Adding other content (video, 3D models, etc)
Adding buttons and interactions
Using the User Interface designer
83. Other Types of Trackables
Image, object, environment tracking
Use Metaio Toolbox for data capture
84. Good Tracking Patterns
pattern that is highly structured
lot of visual hints with different colors
high contrasts and sharp edges
pattern in a "common" format,
Square or rectangle format in 3:2 or 4:3 or similar
not too dark and no reflection points
shortest side of the image 150 – 200 pixels
86. Bad Tracking Patterns
Reference Image not flat and blurry
Shadows create false contrasts
Angled reference images create false
reference orientation
Pattern too bright or dark
Angled with surrounding information
88. Adding More Content
3D models
Position
Adjust properties
- Shadow, occlusion, etc
- Trigger animation
Video content
In-page video, full-screen video
Websites, audio
Page triggered
89. Adding Buttons
Drag button icon into scene
Scale and position
Right click to add behaviours
Open website, play video, etc
90. Using the User Interface Designer
Drag and drop 2D UI elements
Buttons, images, etc
Runs in screen aligned mode
91. Making the Chanel Public
Click to change
channel status
Becomes available for anyone in the world
Takes 1-2 days for review by Metaio
Need a paid Creator License ($530 USD)
95. You will Need
A Junaio developer account
Create at dev.junaio.com
A web server where content can be uploaded
Eg free server from http://www.000webhost.com/
GPS Location of POI
POI content
Text, 2D image, etc
97. Finding the POI location
Use Google maps, right click the POI location
Copy Lat/Long information
98. Server Content
Download “Hello World” template content
http://www.junaio.com/develop/quickstart/
Edit on local machine
Edit index.php to add POI information
Use own POI icon
Upload to web server
102. Channel Creation
Use any name and channel description
Channel Type: Location Based Channel
Callback URL is most important
Path to the index.php file on your server
http://www.junaiotest.comze.com/JunaioTest/1HelloWorld/?path=
Note ?path= at end of URL, you may not need this
Once channel is saved then it is added to My Channels list
Next Validate the Channel
103. Channel Validation
Runs a number of tests to see if server path is
correct, if content is there, returned code correct
106. Loading the Channel
If the Channel is validated you can run it
Either login into Junaio and enter developer mode
Channel appears under ‘My Favourites’
Or use QR code from My Channel page
108. XML Parameters
Many XML Parameters can be set
See http://www.junaio.com/develop/docs/arel-xml-schemaxml-parameters/
109. Adding More POI – edit index.php
<results>
<object id="1”>
//Define POI One
<location>
<lat>-43.536743</lat>
<lon>172.587705</lon>
<alt>0</alt>
</location>
</object>
<object id="2">
//Define POI Two
<location>
<lat>-43.536743</lat>
<lon>172.587000</lon>
<alt>0</alt>
</location>
</object>
</results>";
POI One Location
POI Two Location
111. Limitations of Plain XML
No interactivity
Only simple pop-ups
No user interface Customizations
Can only use Junaio GUI elements
No local interactivity
Always needs remote server connection
113. AREL
Augmented Reality Environment Language
Overcomes limitations of XML by itself
Based on web technologies; XML, HTML5, JavaScript
Core Components
1. AREL XML: Static file, specifies scene content
2. AREL JavaScript: Handles all interactions and animation.
Any user interaction send an event to AREL JS
3. AREL HTML5: GUI Elements. Buttons, icons, etc
Advantages
Scripting on device, more functionality, GUI customization
114.
115.
116.
117. Example 2: Customizing your POI
Using AREL HTML5 to develop custom interface
Download Tutorial 2
http://www.junaio.com/develop/quickstart/customizingyour-pois-images-sounds-videos-and-more/
In Example 1 edited index.php, now use search.php
124. Example 3: Loading a 3D Model
Position a model relative to the user position
if(!empty($_GET['l']))
$position = explode(",", $_GET['l']);
//calculate the position of T-Rex based on the position of the request. An offset is added to
the latitude value.
$tRexLocation = $position;
$tRexLocation[0] += 0.00004;
Use createLocationBasedModel3D to load 3D
model
128. 3D Models
Junaio supports two model formats:
MD2: Animated models, simple textures
OBJ: Static models, high quality textures
Use OBJ for high quality static models, MD2 for animated
Making Models
Make Models using Blender or similar tools
May need file conversion tools
Limit size to 500 – 1000 polygons/model
See http://www.junaio.com/develop/docs/3d-models/
130. Basic Interactivity
Add a button on screen to move virtual character
Use the following
HTML: button specification
Javascript: Interaction
PHP/XML: 3D model
Junaio Tutorial 5
http://www.junaio.com/develop/quickstart/advancedinteractions-and-location-based-model-3ds/
134. Logic_LBS5.js - JavaScript
Create an event listener
setEventListener();
Add functionality to model object
Load model from scene
Adding model behaviours
Add functionality to GUI objects
Define the event listener
Bind model behaviours to GUI objects
145. User Experience
Truly Wearable Computing
Less than 46 ounces
Hands-free Information Access
Voice interaction, Ego-vision camera
Intuitive User Interface
Touch, Gesture, Speech, Head Motion
Access to all Google Services
Map, Search, Location, Messaging, Email, etc
146. Junaio on Google Glass
Junaio Mirage - Junaio Browser on Glass
Adapt UI to Glass (see through, touch gesture)
Backend server unchanged
156. Conclusion
Junaio provides easy way to create mobile AR
Cross platform, client/server based, customisable
Metaio Creator allows non-programmers to
create mobile AR applications
Drag and drop visual interface
Junaio supports HTML, XML, Javascript
AREL format
Support for wearable devices/computers
Google Glass, etc
157. More Information
Mark Billinghurst
mark.billinghurst@hitlabnz.org
HIT Lab NZ
http://www.hitlabnz.org/