Lecture 4 from the COSC 426 graduate class on Augmented Reality. Taught by Mark Billinghurst from the HIT Lab NZ at the University of Canterbury. August 1st 2012

1. COSC 426: Augmented Reality
Mark Billinghurst
mark.billinghurst@hitlabnz.org
August 1st 2012
Lecture 4: AR Developer Tools

2. Low Level AR Libraries

3. Low Level AR Libraries
 ARToolKit
 Marker based tracking
 FLARToolKit
 Flash version of ARToolKit
 SSTT
 Simple Spatial Template Tracking
 Opira
 Robust Natural Feature Tracking

4. What is ARToolKit?
 Marker Tracking Library for AR applications
 Open Source, Multi-platform (Linux, Windows, MacOS)
 Overlays 3D virtual objects on real markers
 Uses single tracking marker
 Determines camera pose information (6 DOF)
 ARToolKit Websites
http://www.hitl.washington.edu/artoolkit/
http://artoolkit.sourceforge.net/

5. How it Works

6. ARToolKit Software
 ARToolKit version: 2.65 or later
 Currently two license models
 Open Source (GPL): ARToolKit 2.72
 Commercial (ARToolWorks): ARToolKit 4.0
 OS: Linux, Windows, MacOS X, iPhone/Android
 Programming language: C
 Related software
 ARToolKit Professional: Commercial version
 ARToolKitPlus: Advanced version
 NyARToolkit: Java and C# version
 FLARToolKit: Flash version

7. ARToolKit Family
ARToolKit NFT
ARToolKit
ARToolKit Plus
JARToolKit (Java)
ARToolKit (Symbian)
FLARToolKit (Flash)
NyToolKit
- Java, C#,
- Android, WM
FLARManager (Flash)

8. ARToolKit Structure
ARvideo.lib
DirectShow
 Three key libraries:
 AR32.lib – ARToolKit image processing functions
 ARgsub32.lib – ARToolKit graphics functions
 ARvideo.lib – DirectShow video capture class

9. Additional Software
 ARToolKit just provides tracking
 For an AR application you’ll need more software
 High level rendering library
 Open VRML, Open Inventor, osgART, etc
 Audio Library
 Fmod, etc
 Peripheral support

10. What does ARToolKit Calculate?
 Position of makers in the camera coordinates
 Pose of markers in the camera coordinates
 Output format
 3x4 matrix format to represent the
transformation matrix from the marker
coordinates to the camera coordinates

11. Coordinate Systems

12. From Marker To Camera
 Rotation & Translation
TCM : 4x4 transformation matrix
from marker coord. to camera coord.

13. AR Application Development

14. An ARToolKit Application
 Initialization
 Load camera and pattern parameters
 Main Loop
 Step1. Image capture and display
 Step2. Marker detection
 Step3. Marker identification
 Step4. Getting pose information
 Step5. Object Interactions/Simulation
 Step6. Display virtual objects
 End Application
 Camera shut down

15. Sample ARToolKit Applications
 Ex. 1: Simple video display
 Ex. 2: Detecting a marker
 Ex. 3: Using pattern
 Ex. 4: Getting a 3D information
 Ex. 5: Virtual object overlay

16. Ex 1: Simple Video Display
 Program : sample1.c
 Key points
 Loop structure
 Video image handling
 Camera parameter handling
 Window setup
 Mouse and keyboard handling

17. Sample1.c – video initialization
 Configure the video input
vconf = <video configuration string>
 Start video capture
arVideoCapStart();
 In init(), open the video
arVideoOpen( vconf );
arVideoInqSize(&xsize, &ysize);
 When finished, close the video path
arVideoCapStop();
arVideoClose();

18. Changing Image Size
 For input capture "
vconf = “videoWidth=320,videoHeight=240";
Note – the camera must support this image size
 For display
argInit( &cparam, 1.5, 0, 0, 0, 0 );
The second parameter means zoom ratio for display
image size related to input image.

19. Graphics handling: libARgsub
 Set up and clean up the graphics window
void argInit( ARParam *cparam, double zoom,
int fullFlag, int xwin, int ywin,
int hmd_flag );
void argCleanup( void );
cparam: camera parameter
zoom: zoom ratio
fullFlag: 0: normal, 1: full screen mode
Xwin, ywin: create small window for debug
hmd_flag: 0: normal, 1: optical see-through mode

20. Sample1.c Main Function
main()!
{!
!init();!
!argMainLoop( mouseEvent, !!
!keyEvent, mainLoop); !
}!

21. Graphics handling: libARgsub
 Go into the iterative cycle
void argMainLoop(
void (*mouseFunc)(int btn,int state,int x,int y),
void (*keyFunc)(unsigned char key, int x, int y),
void (*mainFunc)(void)
);
 Swap buffers
void argSwapBuffers( void );

22. Sample1.c - mainLoop Function
if( dataPtr = (ARUint8 *)
arVideoGetImage()) == NULL ) {
arUtilSleep(2);
return;
}
argDrawMode2D();
argDispImage(dataPtr, 0, 0 );
arVideoCapNext();
argSwapBuffers();

23. Image capture: libARvideo
 Return the pointer for captured image
 ARUint8 *arVideoGetImage( void );
 Pixel format and byte size are defined in config.h
 #define AR_PIX_FORMAT_BGR
 #define AR_PIX_SIZE 3

24. Graphics handling: libARgsub
 Set the window for 2D drawing
void argDrawMode2D( void );
 Set the window for 3D drawing
void argDrawMode3D( void );
void argDraw3dCamera( int xwin, int ywin );
 Display image
void argDispImage( ARUint8 *image,
int xwin, int ywin );

25. Ex. 2: Detecting a Marker
 Program : sample2.c
 Key points
 Threshold value
 Important external variables
 arDebug – keep thresholded image
 arImage – pointer for thresholded image
 arImageProcMode – use 50% image for image
processing
- AR_IMAGE_PROC_IN_FULL
- AR_IMAGE_PROC_IN_HALF

26. Sample2.c – marker detection
/* detect the markers in the video frame */
if(arDetectMarker(dataPtr, thresh,
&marker_info, &marker_num) < 0 ) {
cleanup();
exit(0);
}
for( i = 0; i < marker_num; i++ ) {
argDrawSquare(marker_info[i].vertex,0,0);
}

27. Sample2.c – marker_info structure
typedef struct {
int area;
int id;
int dir;
double cf;
double pos[2];
double line[4][3];
double vertex[4][2];
} ARMarkerInfo; !

28. Ex. 3: Using a Pattern
 Program : sample3.c
 Key points
 Pattern files loading
 Structure of marker
information
- Region features
- Pattern Id, direction
- Certainty factor
 Marker identification

29. Making a pattern template
 Use of utility program:
mk_patt.exe
 Show the pattern
 Put the corner of red line
segments on the left-top
vertex of the marker
 Pattern stored as a
template in a file
 1:2:1 ratio determines the
pattern region used

30. Sample3.c – Pattern File Loading
int patt_id;
char *patt_name = “Data/kanjiPatt”
/* load pattern file */
if(patt_id=arLoadPatt (patt_name) < 0)
{
printf ("Pattern file load error !! n");
exit(0);
}

31. Checking for known patterns
/* check for known patterns */
k = -1;
for( i = 0; i < marker_num; i++ ) {
if( marker_info[i].id == patt_id) {
/* you've found a pattern */
printf("Found pattern: %d n",patt_id);
if( k == -1 ) k = i;
else
/* make sure you have the best pattern
(highest confidence factor) */
if( marker_info[k].cf < marker_info[i].cf )
k = i;
}
}

32. Ex. 4 – Getting 3D information
 Program : sample4.c
 Key points
 Definition of a real marker
 Transformation matrix
- Rotation component
- Translation component

33. Sample4.c – Transformation matrix
double marker_center[2] = {0.0, 0.0};
double marker_width = 80.0;
double marker_trans[3][4];
arGetTransMat(&marker_info[i],
marker_center, marker_width,
marker_trans);

34. Finding the Camera Position
This function sets transformation matrix from marker
to camera into marker_trans[3][4]."
arGetTransMat(&marker_info[k], marker_center,
marker_width, marker_trans);
You can see the position information in the values of
marker_trans[3][4]."
" Xpos = marker_trans[0][3];
Ypos = marker_trans[1][3];
Zpos = marker_trans[2][3];

35. ARToolKit Coordinate Frame

36. Ex. 5- Virtual Object Display
 Program : sample5.c
 Key points
 OpenGL parameter setting
 Setup of projection matrix
 Setup of modelview matrix

37. Appending your own OpenGL code
Set the camera parameters to OpenGL Projection matrix.
argDrawMode3D();
argDraw3dCamera( 0, 0 );
Set the transformation matrix from the marker to the camera to
the OpenGL ModelView matrix.
argConvGlpara(marker_trans, gl_para);
glMatrixMode(GL_MODELVIEW);
glLoadMatrixd( gl_para );
After calling these functions, your OpenGL objects are
drawn in the real marker coordinates.

38. 3D CG Model Rendering
 ARToolKit does not have a function to handle
3D CG models.
 3rd party CG rendering software should be
employed.
 OpenVRML
 OpenSceneGraph
 etc

39. Loading Multiple Patterns
 Sample File: LoadMulti.c
 Uses object.c to load
 Object Structure
typedef struct {
char name[256];
int id;
int visible;
double marker_coord[4][2];
double trans[3][4];
double marker_width;
double marker_center[2];
} ObjectData_T;

40. Finding Multiple Transforms
 Create object list
ObjectData_T *object;
 Read in objects - in init( )
read_ObjData( char *name, int *objectnum );
 Find Transform – in mainLoop( )
for( i = 0; i < objectnum; i++ ) {
..Check patterns
..Find transforms for each marker
}

41. Drawing Multiple Objects
 Send the object list to the draw function
draw( object, objectnum );
 Draw each object individually
for( i = 0; i < objectnum; i++ ) {
if( object[i].visible == 0 ) continue;
argConvGlpara(object[i].trans, gl_para);
draw_object( object[i].id, gl_para);
}

42. Limitations of ARToolKit
 Partial occlusions cause tracking failure
 Affected by lighting and shadows
 Tracking range depends on marker size
 Performance depends on number of markers
 cf artTag, ARToolKitPlus
 Pose accuracy depends on distance to marker
 Pose accuracy depends on angle to marker

43. ARToolKit in the World
 Hundreds of projects
 Large research community

44. FLARToolKit
 Flash AS3 Version of the ARToolKit
(was ported from NyARToolkit the Java Version of the ARToolkit)
 enables augmented reality in the Browser
 uses Papervision3D for as 3D Engine
 available at http://saqoosha.net/
 dual license, GPL and commercial license

45. AR Application Components
Adobe Flash
Papervision 3D
FLARToolkit

46. private function mainEnter(e:Event):void {
/* Capture video frame*/
capture.draw(vid);
/* Detect marker */
if (detector.detectMarkerLite(raster, 80) && detector.getConfidence() > 0.5)
{
//Get the transfomration matrix for the current marker position
detector.getTransformMatrix(trans);
//Translates and rotates the mainContainer so it looks right
mainContainer.setTransformMatrix(trans);
//Render the papervision scene
renderer.render();
}
}

47. FLARToolKit Examples

48. Papervision 3D
 http://www.papervision3d.org/
 Flash-based 3D-Engine
 Supports
 import of 3D Models
 texturing
 animation
 scene graph
 alternatives: Away3d, Sandy,…

49. Source Packages
 „Original“ FLARToolkit (Libspark, Saqoosha) (
http://www.libspark.org/svn/as3/FLARToolKit/trunk/ )
 Start-up-guides
 Saqoosha (http://saqoosha.net/en/flartoolkit/start-up-guide/ )
 Miko Haapoja (http://www.mikkoh.com/blog/?p=182 )
 „Frameworks“
 Squidder MultipleMarker – Example (
http://www.squidder.com/2009/03/06/flar-how-to-multiple-instances-of-multiple-
markers/ )
 FLARManager (http://words.transmote.com/wp/flarmanager/ )

50. Other Languages
 NyARToolKit
 http://nyatla.jp/nyartoolkit/wp/
 AS3, Java, C#, Processing, Unity, etc
 openFrameworks
 http://www.openframeworks.cc/
 https://sites.google.com/site/ofauckland/examples/8-artoolkit-example
 Support for other libraries
- Kinect, Audio, Physics, etc

51. void testApp::update(){ //capture video and detect markers
mov.update();
if (mov.isFrameNew()) {
img.setFromPixels(mov.getPixels(), ofGetWidth(), ofGetHeight());
gray = img;
tracker.setFromPixels(gray.getPixels());
}
}
//--------------------------------------------------------------
void testApp::draw(){ //draw AR objects
ofSetColor(0xffffff); mov.draw(0, 0);
for (int i=0; i<tracker.markers.size(); i++) {
ARMarkerInfo &m = tracker.markers[i];
tracker.loadMarkerModelViewMatrix(m);
ofSetColor(255, 255, 0, 100); ofCircle(0,0,25); ofSetColor(0);
ofDrawBitmapString(ofToString(m.id),0,0);
}
}

52. AR Tools

53. Building Compelling AR Experiences
experiences
applications
tools Authoring
components Tracking, Display
Sony CSL © 2004

54. AR Authoring
 Software Libraries
 osgART, Studierstube, MXRToolKit
 Plug-ins to existing software
 DART (Macromedia Director), mARx, Unity,
 Stand Alone
 AMIRE, BuildAR, etc
 Next Generation
 iaTAR (Tangible AR)

55. mARx Plug-in
 3D Studio Max Plug-in
 Can model and view AR content at the same time

56. BuildAR
 http://www.buildar.co.nz/
 Stand alone application
 Visual interface for AR model viewing application
 Enables non-programmers to build AR scenes

57. Plug-ins for 3D authoring tools
 Unity - Game development tool (www.unity3d.com)
 Esperient Creator – General interactive 3D authoring
(www.esperient.com)

58. Metaio Design
 Complete commercial authoring platform
 http://www.metaio.com/software/design/
 Offers viewer and editor tools
 Drag and drop tools
 3D model import

59. Total Immersion D’Fusion Studio
 Complete commercial authoring platform
 http://www.t-immersion.com/
 Multi-platform
 Markerless tracking
 Scripting
 Face tracking
 Finger tracking
 Kinect support

60. Others
 AR-Media
 http://www.inglobetechnologies.com/
 Google sketch-up plug-in
 LinceoVR
 http://linceovr.seac02.it/
 AR/VR authoring package
 Libraries
 JARToolKit, MXRToolKit, ARLib, Goblin XNA

61. OSGART Programming Library
 Integration of ARToolKit with a High-Level
Rendering Engine (OpenSceneGraph)
OSGART= OpenSceneGraph + ARToolKit
 Supporting Geometric + Photometric Registration

62. osgART:Features
 C++ (but also Python, Lua, etc).
 Multiple Video Input supports:
 Direct (Firewire/USB Camera), Files, Network by
ARvideo, PtGrey, CVCam, VideoWrapper, etc.
 Benefits of Open Scene Graph
 Rendering Engine, Plug-ins, etc

63. Advanced Authoring: iaTAR (Lee 2004)
 Immersive AR Authoring
 Using real objects to create AR applications

64. More Information
• Mark Billinghurst
– mark.billinghurst@hitlabnz.org
• Websites
– www.hitlabnz.org