Petrinect: Modeling gestural interactions with executable Petri nets
1. PetriNect : Modeling gestural interactions with executable Petri nets
PetriNect : Modeling gestural interactions with
executable Petri nets
Romuald Deshayes1 Philippe Palanque2 Tom Mens1
Université de Mons – UMONS, Belgium
firstname.name@umons.ac.be
Université de Toulouse, France
name@irit.fr
4 september 2012
Romuald Deshayes – UMONS 1 / 31
2. PetriNect : Modeling gestural interactions with executable Petri nets
Table of contents
1 Introduction
Goal
Case Study
Tools and formalisms used
2 Petshop and ICO
3 Modeling of the case study
4 Discussion
5 Future Work and improvements
6 Conclusion
Romuald Deshayes – UMONS 2 / 31
3. PetriNect : Modeling gestural interactions with executable Petri nets
Introduction
Introduction
Introduction
Goal
Case Study
Tools and formalisms used
Romuald Deshayes – UMONS 3 / 31
4. PetriNect : Modeling gestural interactions with executable Petri nets
Introduction
What is it about ?
(Informal) Games
Formal specification
techniques
VS
Romuald Deshayes – UMONS 4 / 31
5. PetriNect : Modeling gestural interactions with executable Petri nets
Introduction
Goal
Goal of the talk
Why and how to use formal specification techniques for interaction
with games ?
Why ?
Because it is not more How ?
complicated than By using tools
coding allowing to execute
And it is safer ! Games formal models
also have the right to See what’s next !
security
Romuald Deshayes – UMONS 5 / 31
6. PetriNect : Modeling gestural interactions with executable Petri nets
Introduction
Case Study
Case study
Pong Game
Almost like the classic game
Uses Kinect
With free hands only (gestural interaction)
⇒ Very simple to play
Move the hand to control the paddle
Close it to grab the ball
Interaction modeled with executable Petri-nets
Romuald Deshayes – UMONS 6 / 31
7. PetriNect : Modeling gestural interactions with executable Petri nets
Introduction
Case Study
Gestural interaction
Gestural Interaction ?
In a general way : control a computer with the hands
In our case study : move or manipulate virtual objects
displayed on the screen with gestures
Possible Gestures
FlyOver
Grab
Release
Drag
Zoom (in/out)
Romuald Deshayes – UMONS 7 / 31
8. PetriNect : Modeling gestural interactions with executable Petri nets
Introduction
Case Study
Kinect
Kinect : what and why ?
3D sensor
Equipped with powerful real-time tracking algorithms
Detects hand position in metric space (point in space)
Homemade algorithm to detect if hand is open/closed
For our case study, we will use the position and status of the hands
and the head
Romuald Deshayes – UMONS 8 / 31
9. PetriNect : Modeling gestural interactions with executable Petri nets
Introduction
Tools and formalisms used
Tools and formalisms used
Work done in IRIT Lab
Petri-net rules
Formalism called ICO based on Petri-nets
Tool called Petshop to create ICO’s and execute them
Romuald Deshayes – UMONS 9 / 31
10. PetriNect : Modeling gestural interactions with executable Petri nets
Petshop and ICO
Petshop and ICO
Petshop and ICO
Romuald Deshayes – UMONS 10 / 31
11. PetriNect : Modeling gestural interactions with executable Petri nets
Petshop and ICO
Petri-net
Directed bipartite graph
Node correspond to Places and Transitions
Directed arcs connect places and transitions
Tokens travel from place to place
State of the Petri net is represented by the position of the
tokens in the places
Non-deterministic execution
locality in transitions
Romuald Deshayes – UMONS 11 / 31
12. PetriNect : Modeling gestural interactions with executable Petri nets
Petshop and ICO
ICO
ICO : main key concepts
Defined as a formal description technique designed to the
specification, modeling and implementation of interactive
systems
Uses high-level Petri nets (HLPN) to describe interactive
systems’ behavioural aspects
In HLPN, tokens can carry informations
In ICO, tokens can store Java objects (e.g. Kinect data)
In ICO, transitions can contain Java code (some processing on
Kinect data)
Romuald Deshayes – UMONS 12 / 31
13. PetriNect : Modeling gestural interactions with executable Petri nets
Petshop and ICO
ICO example
Romuald Deshayes – UMONS 13 / 31
14. PetriNect : Modeling gestural interactions with executable Petri nets
Petshop and ICO
Petshop
Petshop
Tool suite dedicated to the engineering of interactive systems.
Supports ICO
Allows the editing, simulation and dynamic execution of
models.
Use of petshop to create our models, link them together, execute
them, and interact with Java code
Romuald Deshayes – UMONS 14 / 31
15. PetriNect : Modeling gestural interactions with executable Petri nets
Petshop and ICO
Petshop
Romuald Deshayes – UMONS 15 / 31
16. PetriNect : Modeling gestural interactions with executable Petri nets
Modeling of the case study
Modeling of the case study
Modeling of the case study
Romuald Deshayes – UMONS 16 / 31
17. PetriNect : Modeling gestural interactions with executable Petri nets
Modeling of the case study
Modeling of the case study
Models used to process kinect raw data
Transform data into abstract events
Events are interpreted by virtual objects models
Romuald Deshayes – UMONS 17 / 31
18. PetriNect : Modeling gestural interactions with executable Petri nets
Modeling of the case study
Modeling of the case study
Use Petshop to create a 3 layered ICO model
Encapsulate the data of hands and head of each player into
tokens (position and state)
Models are created to be independent of the number of players
(ID for each player)
Only one instance of each layer whatever the amount of virtual
objects to control
Each virtual object can be modeled using petshop (4th layer)
Romuald Deshayes – UMONS 18 / 31
19. PetriNect : Modeling gestural interactions with executable Petri nets
Modeling of the case study
Raw data to positions
Layer 1 : Raw data to positions
Receives positions of hands and head of each player at regular
time intervals
3 tokens are generated at each kinect frame refresh (1 for the
head, 1 for each hand)
Informations about head and hand positions are stored in the
tokens
ICO model joins data from head and hand and create a new
token for each hand
Tokens are sent to the next layer
Romuald Deshayes – UMONS 19 / 31
20. PetriNect : Modeling gestural interactions with executable Petri nets
Modeling of the case study
Raw data to positions
Romuald Deshayes – UMONS 20 / 31
21. PetriNect : Modeling gestural interactions with executable Petri nets
Modeling of the case study
Positions to gestures
Layer 2 : Positions to gestures
Receives absolute positions of hands (1 token per hand per
player)
Uses 2 consecutive tokens to calculate relative movement
Detects state changes of the hands (open/closed)
Sends open/close/move events to the 3rd layer
Romuald Deshayes – UMONS 21 / 31
22. PetriNect : Modeling gestural interactions with executable Petri nets
Modeling of the case study
Positions to gestures
Romuald Deshayes – UMONS 22 / 31
23. PetriNect : Modeling gestural interactions with executable Petri nets
Modeling of the case study
Gestures to events
Layer 3 : Gestures to events
Receives move/open/close events from layer 2
Stores the state of each player in a 4 places sub Petri net
Joins information about the player and events from layer 2 to
generate events
Generates 5 different events
Grab
Release
Flyover
Drag
Zoom (in/out)
Sends events to modeled objects
Romuald Deshayes – UMONS 23 / 31
24. PetriNect : Modeling gestural interactions with executable Petri nets
Modeling of the case study
Gestures to events
Romuald Deshayes – UMONS 24 / 31
25. PetriNect : Modeling gestural interactions with executable Petri nets
Discussion
Discussion
Discussion
Romuald Deshayes – UMONS 25 / 31
26. PetriNect : Modeling gestural interactions with executable Petri nets
Discussion
Discussion
Modeling has a lot of advantages :
Reusable models, easy to maintain or extend (dynamicity)
Models hide some technical complexity
Amenable to formal analysis
Can deal with Multi-user
Easy to integrate with many different virtual objects
Goal : toolkit for gestural interaction
Romuald Deshayes – UMONS 26 / 31
27. PetriNect : Modeling gestural interactions with executable Petri nets
Future Work and improvements
Future Work and improvements
Future Work and improvements
Romuald Deshayes – UMONS 27 / 31
28. PetriNect : Modeling gestural interactions with executable Petri nets
Future Work and improvements
Future work and improvements
Future Work and improvements :
Specify the game (rules and graphics) with Petri nets
Performance issue
Threads tweaking (constant token flow)
Compile for deployment
Extend the models to deal with more gestures
Create a DSL over ICO to simplify the specification of virtual
objects behaviour
Romuald Deshayes – UMONS 28 / 31
29. PetriNect : Modeling gestural interactions with executable Petri nets
Conclusion
Conclusion
Conclusion
Romuald Deshayes – UMONS 29 / 31
30. PetriNect : Modeling gestural interactions with executable Petri nets
Conclusion
Conclusion
Wrap-up
Possible to specify gestural interaction of a game with formal
methods
Formal methods might help for safety issues in gaming too
Lot of other advantages to resort to executable modeling
instead of coding
Dynamicity lead to productivity increase
Hides some technical details
Easy to read and extend
Petshop is a cool tool suite, USE IT !
Romuald Deshayes – UMONS 30 / 31
31. PetriNect : Modeling gestural interactions with executable Petri nets
Conclusion
Thank you
Thank you for your attention !
Questions ?
Romuald Deshayes – UMONS 31 / 31