1. Open Source GearVR Framework (GVRf)
Intro, 3D Cursor, and Input Devices
Tom Flynn
Director, SW Engineering for VR – Mobile R&D Lab
Rick Lau
Director, Product Management & Engineering for VR Interaction/IOT – Mobile R&D Lab

2. 2
Agenda – Developing for the Gear VR
• Gear VR Framework
– Introduction
– How far we’ve come
– Scripting demonstration
– Planned Features
• Gear VR Framework Interaction
– 3D Cursor
– VR I/O Devices & Integrated Devices

3. 3
GVRf - Introduction
• Current development options for the Gear VR:
– Unity 3D plugin
– Unreal plugin
– Oculus Mobile SDK (C++ and OpenGL and Android NDK)
• What’s missing from the above options
– A Java based VR framework
– Something that is easy for non-game developers who still want to
develop VR apps

4. 4
GVRf - Introduction
• Java library that allows easy
development of VR applications in
Java for Android
• Built on top of Oculus Mobile SDK
and compliments existing solutions
• Allows developers to:
– Load common 3d file formats (fbx, collada,
obj, etc.)
– Perform scene management
– Attach animations to objects
– Automatically handle head tracking
– Automatically handle drawing the scene in
stereo
• Open Source; No royalties

5. 5
GVRf App - Immersivepedia

6. 6
GVRf – Hello World App
// get a handle to the scene
GVRScene scene = gvrContext.getMainScene();
// load texture
GVRTexture texture = gvrContext.loadTexture(“GearVR.jpg”);
// create a scene object
GVRSceneObject sceneObject = new GVRSceneObject(gvrContext, 4.0f, 4.0f, texture);
// set the object position
sceneObject.getTransform().setPosition(0.0f, 0.0f, -3.0f);
// add the scene object to the scene graph
scene.addSceneObject(sceneObject);

7. 7
New GVRf Features – Standard Geometric Shapes
Quad Cube Text
Sphere Cylinder Cone

8. 8
New GVRf Features – Scene Objects
WebView
Video
Camera

9. 9
New GVRf Features – Miscellaneous
 Frustum culling
 State sorting
 Transparency sorting
 Level of Detail
 Full Assimp support
 Skinning and skeleton animations
 Alpha to Coverage

10. 10
New GVRf Features – Miscellaneous (cont)
 Display and interact with Android Views
 Android Studio support
 Monoscopic support on non-Samsung phones
 IO subsystem for VR Input
 Scripting support for Lua and JavaScript
 Support for multiple lights,
– lightmapping,
– and shadows
 Contributions from KolorCompany, Niusounds, and many others.
 Much much more

11. 11
GVRf Scripting Features – Easy to Do
• Support for either Lua or JavaScript
• Can write full program logic via scripting.
• Can attach scripts to scene objects (“script bundles”)
– See gvr-javascript sample.
• Connect over network and perform scripting commands
– gvrContext.startDebugServer();
• Remote scripting support:
– Compile and run the gvr-remote-scripting demo on
https://github.com/gearvrf/GearVRf-Demos/
– adb forward tcp:1645 tcp:1645
– telnet localhost 1645 (on Windows, use Window’s telnet program or cygwin).

12. 12
Scripting – Displaying a 360 Photosphere
// https://github.com/gearvrf/GearVRf-Demos/blob/scripting/
// gvr-remote-scripting/assets/photo.js
importPackage(org.gearvrf);
importPackage(org.gearvrf.scene_objects);
var url = new java.net.URL("https://raw.githubusercontent.com/gearvrf/GearVRf-
Demos/master/gvrf_minimal360photo/res/raw/photosphere.jpg");
var resource = new GVRAndroidResource(gvrf, url);
var texture = gvrf.loadFutureTexture(resource);
var photosphere = new GVRSphereSceneObject(gvrf, false, texture);
photosphere.getTransform().setScale(20, 20, 20);
photosphere.setName(“photosphere”);
var scene = gvrf.getMainScene();
scene.addSceneObject(photosphere);

13. 13
Scripting – Loading and Playing a 360 Video
// https://github.com/gearvrf/GearVRf-Demos/blob/scripting/
// gvr-remote-scripting/assets/video.js
importPackage(org.gearvrf);
importPackage(org.gearvrf.scene_objects);
var sphere = new GVRSphereSceneObject(gvrf, false);
var mesh = sphere.getRenderData().getMesh();
var mediaPlayer = new
org.gearvrf.utility.ImageUtils.createMediaPlayer("https://raw.githubusercontent.com/
gearvrf/GearVRf-Demos/master/gvrf_minimal360video/assets/videos_s_3.mp4")
var video = new GVRVideoSceneObject(gvrf, mesh, mediaPlayer,
GVRVideoSceneObject.GVRVideoType.MONO );
video.setName(“video”);
var scene = gvrf.getMainScene();
scene.addSceneObject(video);
mediaPlayer.start();

14. 14
Scripting – Loading Object Over Network and Animating It
// https://github.com/gearvrf/GearVRf-Demos/blob/scripting/
// gvr-remote-scripting/assets/trex.js
importPackage(org.gearvrf);
var scene = gvrf.getMainScene();
var trex_url = new java.net.URL("https://github.com/gearvrf/GearVRf-Demos/raw/
master/gvr-meshanimationsample/assets/TRex_NoGround.fbx");
var trex = gvrf.loadModelFromURL(trex_url);
scene.addSceneObject(trex);
trex.getTransform().setPosition(0.0, -5.0, -10.0);
trex.getTransform().setRotationByAxis(90.0, 1.0, 0.0, 0.0);
trex.getTransform().setRotationByAxis(40.0, 0.0, 1.0, 0.0);
trex.getTransform().setScale(1.5, 1.5, 1.5);
trex.setName(“trex”);
var engine = gvrf.getAnimationEngine();
var animation = trex.getAnimations().get(0);
animation.setRepeatMode(1).setRepeatCount(-1);
animation.start(engine);

15. 15
GVRf – Planned Features
• Support for OVR_Multiview
• Support for Vulkan
• Support for X3D
• Better support for physics and spatial audio
– Already have samples for jbullet and cardboard-audio
• Debugger and profiling information
• Lots of optimizations still to come

16. 16
GVRf – Available Now!
• Available as Open Source!
• Apache 2.0 license
• www.gearvrf.org
• https://github.com/Samsung/GearVRf
• https://github.com/gearvrf/GearVRf-Demos/
• Mailing lists: http://lists.gearvrf.org/

17. 3D Cursor – New Interaction Model for VR
& Partner Input Devices
Rick Lau

18. 18
Challenge: How to improve the VR Interaction experience.
• Team was tasked to improve the VR interaction experience.
• Today, gaze cursor is the primary input device.
• Expensive and complicated.
– Too much coding, too little results.
– 3rd party HW took a lot of time to integrate, per application.
– App developers had to understand and develop to specific HW.

19. 19
Solution - 3D Cursor!
 A cursor for VR - extension of the PC
mouse/cursor.
 Framework level implementation.
 Standardizes interaction for VR apps
and input type integration.
 Behavior
 Same API as the PC cursor but with a
Z-axis component.
 An unlimited number of cursors can be
instantiated.
 Cursors can be dynamically instantiated
and destroyed.
 Cursors can be customized.
3D Cursor
Input Device
GVRF
Scene Graph
&
Objects
Application

20. 20
1) Laser Cursor
– onCollision events for all objects in path of “laser”.
2) Point Cursor
– Mass and shape of object can be user defined.
– Intersection with other objects are more specific.
– Objects can be exlcuded from onCollision events.
2 Types of 3D Cursor
point cursor
object
intersect event/volume
optional: application
calculates intersected volume

21. 21
Point Cursor Example - Cannonball as a Cursor
cursor/cannonball’s parabolic (-like...)
trajectory can be calculated
onCollision detection of object,
explosion animation can be rendered
2
3
1 cursor is instantiated

22. 22
Point Cursor Example - Sword as a Point Cursor
onCollision Event
Cursor
onCollision animation
As volume of intersection
increases, shape of pumpkin can
change to reflect slicing action.

23. 23
Point Cursor Example - Hand as a Cursor Input Device
center of palm used for primary
(x,y,z) cursor placement
Each finger can be a point cursor:
• onCollision event can be
determined by all any fingertip
touching object.
• onButtonClick event can be
determined by a pinch gesture.

24. 24
Samsung GearVRf Partners for Input & General Support
3D Cursor
Input Device
GVRF
Scene Graph
&
Objects
Application
Gear 2/S2
Hand Recognition
Eye Tracking
6DOF Controllers
VR Gloves
Mouse, Gamepad, GearVR Touchpad
2 Cursor Types
Laser Cursor Examples
Point Cursor Examples
Integrated to GVRF!

25. 25
Samsung GVRf I/O Partners & Live Demo
See and experience these cools VR device input demos at the GVRf booth!

26. 26
Touchless hand & eye tracking example using point & laser cursors.
Laser Cursor
(The Eye Tribe)
Point Cursors
(Gestigon)

27. 27
3D Cursor API Overview
3D Cursor
Input Device
GVRF
Scene Graph
&
Objects
Application
Application Interfaces
 Cursor Manager
 Event Listener for Objects
 Cursor Event: OnCollision
 Button Events: up, down, pressed
 Customization/Settings
 read, modify, save
 Gestures
I/O Device Interfaces
 (x,y,z) position
 Num buttons &button click events
 Gestures (tbd)
 OK, cancel
 up, down, left, right
 scroll up, scroll down,
 others...
mouse
gamepad
gaze
built in support

28. 28
Point Cursor Application State Machine Example
rest clicked
behind collision

29. 29
Customization & Settings
XML based customization

30. 30
Release Plans for 3D Cursor
• Basic features to date – baseline APIs
– More complex cursor behaviors in the second release.
• Gather feedback from this Developer’s Conference
• Will push to gearvrf.org open source website – May/June.
Raffle giveaway by XImmerse at the GVRf booth!
Get chance to win their next gen 6DOF VR controllers.

31. Q & A
and THANK YOU for your time.
Tom Flynn and Rick Lau
tom.flynn@samsung.com
rick.lau@samsung.com
www.SDC2016.com
© 2016 Samsung Developer Conference. All rights reserved.