The development of Augmented Reality (AR) applications is still driven by development at source-code level. Although recent approaches focus to standardize AR functionality, to our knowledge a declarative and model-driven design (MDD) has not been applied for AR development so far. MDD approaches have been successfully applied to model user interfaces for a wide spectrum of modes (such as speech-command interfaces, or remote controls) and media such as HTML, 3D, and smart phone interfaces for instance. In this paper we propose a MDD approach for modeling seamless interaction between Web and AR interfaces. Therefore we implemented a prototype of a web furniture shop that supports arranging furniture in an augmented reality to prove our approach. We then present our approach of modeling a reality spanning Drag-and-Drop interaction between a 2D browser and an AR scene. Finally, we discuss the issues that we were confronted with to support reality spanning interaction like switching interaction modes and coordinate systems and present limitations that we have experienced with using other control modes, like a Wii-Controller or glove-based gesture detection.
1. 1. Juni 2011 Model-based Design of Interactions that can bridge RealitiesThe Augmented “Drag-and-Drop”Sebastian Feuerstack, Allan C. M. de Oliveira, Regina B. Araujo Universidade Federal de São CarlosDepartamento de Computacão
6. The Augmented “Drag-and-Drop 6 1. Juni 2011 Why modeling? One specification that is declarative and exact to discuss & execute Is located between Tool and Code can be easily adapted or enhanced to different forms of interaction
7. The Augmented “Drag-and-Drop 7 1. Juni 2011 Use Case: Augmented Drag and Drop Buying Furniture Shopping Cart Drag and Drop Into my environment
8. Video: Connecting the Web with AR The Augmented “Drag-and-Drop 8 1. Juni 2011 VIDEO Mouse-based Drag-and-Drop
9. The Augmented “Drag-and-Drop 9 1. Juni 2011 How to model? Models Interactors(Abstract & Concrete Media, Mode) Static: class diagramm, Behavior: state-chart Mappings Mode-To-Media Synchronization Custom Notation
11. The Augmented “Drag-and-Drop 11 1. Juni 2011 Abstract Media Model Single Choice Aggregates a set of Entities from that only one can be chosen at a time Examples: Direction (leftorright), Shopping Cart Further Properties: Output tothe User Container -> Aggregation, Discrete, thatare Inputs Contains Single Choice Elements
15. The Augmented “Drag-and-Drop 15 1. Juni 2011 Conclusions & Future Work What’s the advantage? Detailed, declarative Modeling of Interactions (Media, Mode) We can combine Modes & Media and bridge Realities What`s still missing is a tool What`s the advantage??? Lets see a demo with different Modes
16. Gesture-based Drag and Drop Control The Augmented “Drag-and-Drop 16 1. Juni 2011 Cam Jessica Pointing Cam Command Monitor
17. Gesture-based Drag and Drop Control Design of Multimodal Interaction 17 1. Juni 2011 VIDEO Gesture-based Drag-and-Drop
19. The Augmented “Drag-and-Drop 19 1. Juni 2011 Abstract Model Reference – Object or Property of an Object to unambigously identify it Examples: Name, Color, Number, Shape, Coordinate Further Properties: Discrete -> Distinguishable Input -> bythe User
