Trabajo Regular presentado en CoSECiVi'14.
Resumen:
This paper describes a serious game intended to teach singing to children. The system shows the students a virtual world they can explore, and the evolution of the world is based on their performance.
Automatic content generation techniques are used to present the player with different exercises, and their singing is evaluated using state of the art techniques.
Influencing policy (training slides from Fast Track Impact)
CoSECiVi'14 - E-Cecilia. Implementation of a music game
1. D’ IMATGE I
GRÀFICS!
TECNOLOGIE
S!
D’ IMATGE I GRÀFICS!
TECNOLOGIES!
http://gilab.udg.edu
E-cecilia:
implementation
of a music game
R. García1,5, I. Barbancho2, L. Tardón2,
J. Arambarri3, M. Magdics1,4, M. Sbert1
1University of Girona
2University of Malaga
3Virtualware
4Budapest University of Technology and Economics
5iMinds / University of Hasselt
2. D’ IMATGE I
GRÀFICS!
TECNOLOGIE
S!
• Introduction
• Game description
• Architecture, client-server model
• Server
• Music composition & analysis
• Client
• Graphics, non-photorealistic rendering
• Natural interaction
• Immersive devices
Contents
3. D’ IMATGE I
GRÀFICS!
TECNOLOGIE
S!
Introduc-on
• Learning to sing requires long practising hours
• The early stages use very simple songs → boring
• Good singers start very young, so they tire more easily
• Solution: Make the practising into a game
• Related work:
• SingStar: full songs, too complex for starting
young children
• Other games:
• focused on keyboard / mouse interaction
• don't do audio analysis
4. D’ IMATGE I
GRÀFICS!
TECNOLOGIE
S!
• Serious game to practice singing and rhythm
• The user is presented with a small village to explore
• The houses are small, and can be enhanced by the user:
• After clicking a house, a song is presented
• The user sings to the best of his abilities
• The house evolves in response to the user singing
• The user can continue exploration and singing until he
has practiced enough.
Game
descrip-on
6. D’ IMATGE I
GRÀFICS!
TECNOLOGIE
S!
System
architecture
• Three main components:
• Game client (Unity)
• Assets (on a web server)
• Recommendation and Evaluation server
• Audio and rhythm evaluation (matlab)
• Recommendation system (C++ library)
• Interconnection:
• TCP sockets (game-server)
• HTML (game-webserver)
7. D’ IMATGE I
GRÀFICS!
TECNOLOGIE
S!
Game
client
• Object oriented, asynchronous architecture using the Unity
game engine. Object behaviour uses C# scripts.
• The main objects are:
• World Creation (terrain, house creation and evolution)
• World View: Camera, NPR effects, Dome transforms,
GUI
• World control: Kinect, mouse interaction
• Asset downloading, communication with web server.
• Microphone: song recording
• ServerConnect: Communication with the server,
evolving the GameState
8. D’ IMATGE I
GRÀFICS!
TECNOLOGIE
S!
Music
server
• The game server is responsible for the automatic generation
of musical excercises (songs) and the assessment of the user
performance
Server
9. D’ IMATGE I
GRÀFICS!
TECNOLOGIE
S!
Music
synthesis
• A set of training songs is used and analyzed
• Created songs replicate the selected style and complexity of
that of the corresponding exemplars
Training Generation
11. D’ IMATGE I
GRÀFICS!
TECNOLOGIE
S!
Graphics,
Non-‐photorealis-c
Rendering
(NPR)
• Post processing stylization
– State-of-the-art image processing (on the color and depth)
• Flexible, a wide range of effects
• Commercial Unity package under finalization
Original rendering
(“Photo-realistic”)
Color Comic:
Image simplification + edges
13. D’ IMATGE I
GRÀFICS!
TECNOLOGIE
S!
• Replacement of the shadowed pixels by another color
NPR:
Shadow
effects
Complementary shadows:
Shadow color is the complementary color
of the shadowed surface
14. D’ IMATGE I
GRÀFICS!
TECNOLOGIE
S!
• Increasing the level of abstraction by depth
NPR:
depth
sensa-on
Depth-based desaturation Depth-based simplification:
Texture details are removed
in the background
16. D’ IMATGE I
GRÀFICS!
TECNOLOGIE
S!
• Augmented reality: mobile device, augment camera
image with controls (e.g. buttons)
• Kinect: exergame (imitating real movements, like
walking or turning) or control by hand movements
Natural
interac-on
AR button Exergame: walk
17. D’ IMATGE I
GRÀFICS!
TECNOLOGIE
S!
• Different projection/camera settings
– E.g. stereo: 2 cameras, dome: 360º projection with
cube-map
Stereo, top-bottom Stereo, side-by-side
Immersive
visualiza-on
19. D’ IMATGE I
GRÀFICS!
TECNOLOGIE
S!
Immersive
visualiza-on
• NPR+immersive: NPR on the final image
– Regular 2D image processing
• No added computational complexity for using immersive
displays w.r.t. NPR
Dome+color comic Immersive+B&W comic
20. D’ IMATGE I
GRÀFICS!
TECNOLOGIE
S!
Conclusions
• Music teaching game
• State-of-the-art automatic music synthesis and
singing assessment
• Controllable graphics style (NPR) as post-
processing
• Support for natural interaction, immersive
visualization