The main objective of this thesis is to provide tools for an expressive and real-time synthesis of sounds resulting from physical interactions of various objects in a 3D virtual environment. Indeed, these sounds, such as collisions sounds or sounds from continuous interaction between surfaces, are difficult to create in a pre-production process since they are highly dynamic and vary drastically depending on the interaction and objects. To achieve this goal, two approaches are proposed; the first one is based on simulation of physical phenomena responsible for sound production, the second one is based on the processing of a recordings database.

1. t
Expressive Sound Synthesis
For Animation
Cécile Picard-Limpens
University of Nice/Sophia-Antipolis
École Doctorale STIC
REVES INRIA Sophia-Antipolis, France
Advisors: George Drettakis, INRIA Sophia Antipolis (Reves)
François Faure, INRIA Rhône-Alpes (Evasion)
Nicolas Tsingos, DOLBY Laboratories, CA, USA
Defense for Ph.D. in Computer Science
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
1

2. t
Outline
1 Sound and Virtuality
2 Physics-Based Sound Synthesis
Contact Modeling
Resonator Modeling
3 Example-Based Synthesis
Flexible Sound Synthesis
4 Perspectives on a Hybrid Model
Motivation and Application
5 Conclusion and Discussion
Contributions
Extensions and Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
2

3. t
Sound Rendering
Sound and
Virtuality
General Background
for Virtual Reality and Games
Motivation
Physics-Based
Synthesis
Example-Based Interactive Audio Rendering
Synthesis
Perspectives on
a Hybrid Model
Conclusion and
Discussion
(R. Vantielcke - WipeoutHD on Playstation 3)
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
3

4. t
Sound Rendering
Sound and
Virtuality
General Background
for Virtual Reality and Games
Motivation
Physics-Based
Synthesis
Example-Based Interactive Audio Rendering
Synthesis
Perspectives on
a Hybrid Model
Conclusion and
Discussion
(R. Vantielcke - WipeoutHD on Playstation 3)
Traditional Approach
Pre-Recordings Triggered
+ : Easy to implement
– : Repetitive audio, discrepancies, lack of flexibility
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
3

5. t
From Playback of Samples
Sound and
Virtuality
General Background
to Synthesis
Motivation
Physics-Based
Synthesis
Digital Sound Synthesis
Example-Based
Synthesis Source modeling ←
Perspectives on Sound propagation, Sound reception
a Hybrid Model
Conclusion and Techniques
Discussion
Rigid body simulation
Finite Element Method (FEM)
(ArtiSynth)
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
4

6. t
From Playback of Samples
Sound and
Virtuality
General Background
to Synthesis
Motivation
Physics-Based
Synthesis
Digital Sound Synthesis
Example-Based
Synthesis Source modeling ←
Perspectives on Sound propagation, Sound reception
a Hybrid Model
Conclusion and Techniques
Discussion
Rigid body simulation
Finite Element Method (FEM)
(ArtiSynth)
Physical Sound Simulation
+ : Physical approach, easy parametrization,
Low memory usage
– : Preprocess computation,
Interface between physics and sound system
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
4

7. t
Controlling the Sound Simulation
Sound and
Virtuality Challenges
General Background
Motivation
Physics-Based
Synthesis
Example-Based
Synthesis
Perspectives on
Sound Coherent With Visuals
a Hybrid Model
Conclusion and
Unpredictable character of sounds
Discussion
Real-time sound synthesis
Parametrization and Expressiveness
Control and interactivity
Authoring
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
5

8. t
Our Contribution
Sound and
Virtuality Three Research Axes
General Background
Motivation
Physics-Based
Synthesis
Example-Based
Synthesis
Physics-Based Sound synthesis
Perspectives on
a Hybrid Model Contact modeling
Conclusion and Resonator modeling
Discussion
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
6

9. t
Our Contribution
Sound and
Virtuality Three Research Axes
General Background
Motivation
Physics-Based
Synthesis
Example-Based
Synthesis
Physics-Based Sound synthesis
Perspectives on
a Hybrid Model Contact modeling
Conclusion and Resonator modeling
Discussion
Example-Based Sound Synthesis
Automatic analysis of pre-recordings
Flexible synthesis for physics-driven animation
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
6

10. t
Our Contribution
Sound and
Virtuality Three Research Axes
General Background
Motivation
Physics-Based
Synthesis
Example-Based
Synthesis
Physics-Based Sound synthesis
Perspectives on
a Hybrid Model Contact modeling
Conclusion and Resonator modeling
Discussion
Example-Based Sound Synthesis
Automatic analysis of pre-recordings
Flexible synthesis for physics-driven animation
Perspectives on a Hybrid Model
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
6

11. t
Overview
Sound and
Virtuality
Physics-Based
Synthesis 1 Sound and Virtuality
Contact Modeling
Audio Texture
Synthesis For Complex
Contacts
2 Physics-Based Sound Synthesis
Resonator Modeling Contact Modeling
A Robust and
Multi-Scale Modal
Analysis
Resonator Modeling
Example-Based
Synthesis 3 Example-Based Synthesis
Perspectives on Flexible Sound Synthesis
a Hybrid Model
Conclusion and 4 Perspectives on a Hybrid Model
Discussion
Motivation and Application
5 Conclusion and Discussion
Contributions
Extensions and Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
7

12. t
Sound from Contacts
Sound and
Virtuality
Physics-Based
Synthesis
Contact Modeling
Audio Texture
Synthesis For Complex
Dichotomy
Contacts
Resonator Modeling Impacts
A Robust and
Multi-Scale Modal
Analysis
Continuous contacts
Example-Based
Synthesis Two Schemes for Contact Force Modelling
Perspectives on
a Hybrid Model
Feed-forward scheme
Conclusion and [van den Doel et al. 01]
Discussion
Additive synthesis
Direct computation of contact forces
[Avanzini et al. 02]
Bristle model
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
8

13. t
Contact Modeling
Sound and
Virtuality
Physics-Based
Synthesis
Contact Modeling
What Are The Current Limitations
Audio Texture
Synthesis For Complex
for Continuous Contacts?
Contacts
Resonator Modeling
Rate for physics engine report
A Robust and
Multi-Scale Modal
Analysis No geometric details when using visual textures
Example-Based
Synthesis
Authoring and control are challenging
Perspectives on
a Hybrid Model
Conclusion and
Discussion
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
9

14. t
Contact Modeling
Sound and
Virtuality
Physics-Based
Synthesis
Contact Modeling
What Are The Current Limitations
Audio Texture
Synthesis For Complex
for Continuous Contacts?
Contacts
Resonator Modeling
Rate for physics engine report
A Robust and
Multi-Scale Modal
Analysis No geometric details when using visual textures
Example-Based
Synthesis
Authoring and control are challenging
Perspectives on
a Hybrid Model
HOW Can We Solve Them?
Conclusion and
Discussion By extracting
Excitation profiles from visual textures
with
Adaptive resolution
[Picard et al., VRIPHYS 08]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
9

15. t
Method for Impact Sounds
Sound and
Virtuality
Physics-Based
Synthesis
Contact Modeling
Audio Texture
Synthesis For Complex
Contacts
Resonator Modeling
A Robust and
Multi-Scale Modal
Analysis
Example-Based
Synthesis
Perspectives on
a Hybrid Model
Conclusion and
Discussion
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
10

16. t
Method for Continuous Contact Sounds
Sound and
Virtuality Extraction of Excitation Profiles
Physics-Based
Synthesis
Contact Modeling
Audio Texture
Synthesis For Complex
Contacts
Resonator Modeling
A Robust and
Multi-Scale Modal
Analysis
Example-Based
Synthesis
Perspectives on
a Hybrid Model
Conclusion and
Discussion
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
11

17. t
Synthesis of Excitation Profiles
Sound and
Virtuality For the Audio Force Modelling
Physics-Based
Synthesis
Contact Modeling
Audio Texture
Synthesis For Complex
Contacts Technique
Resonator Modeling
A Robust and
Multi-Scale Modal
Analysis
Extraction from the visual texture image
Example-Based Re-sampling along the trajectory
Synthesis of the contact interaction (60Hz vs 44kHz)
Perspectives on
a Hybrid Model
Conclusion and
Based on the Complexity of the Histogram
Discussion
Simple texture image:
Gradient of the image intensity
Complex texture image:
Isocurves of constant brightness (isophotes)
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
12

18. t
Complex Textures
Sound and
Virtuality Coding the Excitation Profiles
Physics-Based
Synthesis
Contact Modeling
Audio Texture
Synthesis For Complex
Isophotes = Large amount of data
Contacts
Resonator Modeling How Can We Lighten the Info?
A Robust and
Multi-Scale Modal
Analysis
Example-Based
By Coding the Excitation Profiles
Synthesis = Main Features + Noise Part
Perspectives on
a Hybrid Model
Conclusion and
Discussion
= +
Noise Part: Statistical approximation
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
13

19. t
Real-Time Audio Management
Sound and
Virtuality A Flexible Audio Pipeline
Physics-Based
Synthesis
Contact Modeling
Audio Texture
Synthesis For Complex
Contacts
Resonator Modeling Simulations Driven by Ageia’s PhysX (now NVIDIA)
A Robust and
Multi-Scale Modal
Analysis
Example-Based
Synthesis
Perspectives on
a Hybrid Model
Conclusion and
Discussion
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
14

20. t
Audio Texture Synthesis
Sound and
Virtuality A Solution for Interactive Simulations
Physics-Based
Synthesis
Contact Modeling
Audio Texture
Synthesis For Complex
Contacts
Resonator Modeling
A Robust and
Multi-Scale Modal
Analysis
Example-Based
A Sound in Coherence with Visuals
Synthesis
Perspectives on
a Hybrid Model
Flexible Resolution
Conclusion and
Discussion Adapted to Procedural Generation
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
15

21. t
Overview
Sound and
Virtuality
Physics-Based
Synthesis 1 Sound and Virtuality
Contact Modeling
Audio Texture
Synthesis For Complex
Contacts
2 Physics-Based Sound Synthesis
Resonator Modeling Contact Modeling
A Robust and
Multi-Scale Modal
Analysis
Resonator Modeling
Example-Based
Synthesis 3 Example-Based Synthesis
Perspectives on Flexible Sound Synthesis
a Hybrid Model
Conclusion and 4 Perspectives on a Hybrid Model
Discussion
Motivation and Application
5 Conclusion and Discussion
Contributions
Extensions and Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
16

22. t
Vibration Models
Sound and
Virtuality Modal Analysis
Physics-Based
Synthesis
Contact Modeling
Audio Texture
Synthesis For Complex
Contacts Generating Sounds Based on Physics Simulation
Resonator Modeling
A Robust and
Multi-Scale Modal
In computer musics
Analysis
[Iovino et al. 97, Cook 02]
Example-Based
Synthesis In computer graphics
Perspectives on [Van Den Doel 01, O Brien et al. 02]
a Hybrid Model
Conclusion and
Discussion Improvements for Interactive Sound Rendering
Modal parameter tracking
[Maxwell et al. 07]
Frequency content sparsity
[Bonneel et al. 08]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
17

23. t
Vibration Models
Sound and
Virtuality Modal Analysis
Physics-Based
Synthesis
Contact Modeling
Audio Texture
1 Get a Sounding Object and its Geometry
Synthesis For Complex
Contacts
Resonator Modeling
A Robust and
Multi-Scale Modal
Analysis
Example-Based
Synthesis 2 Construct the FEM (ex: Tetrahedral Mesh)
Perspectives on
a Hybrid Model
3 Apply Newton Second Law to DOF
Conclusion and
Discussion ¨ ˙
M d + C d + Kd = f (1)
4 Eigendecomposition ⇒ Modal Parameters
M = LL−T ; L−1 KL−T = V ΛV T (2)
where V = matrix of eigenvectors
Λ = diagonal matrix of eigenvalues
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
18

24. t
Vibration Models
Sound and
Virtuality Modal Analysis
Physics-Based
Synthesis
Contact Modeling
Audio Texture
Synthesis For Complex
Contacts
In Real-time:
Resonator Modeling
A Robust and
Modal synthesis
Multi-Scale Modal
Analysis
Example-Based
Synthesis
Perspectives on
a Hybrid Model
Conclusion and
Discussion
1
s(t) = ai sin(wi t)e −di t (3)
n
Control for vibration models
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
19

25. t
Vibration Models
Sound and
Virtuality Modal Analysis
Physics-Based
Synthesis
Contact Modeling
Audio Texture
What Are
Synthesis For Complex
Contacts The Current Limitations?
Resonator Modeling
A Robust and
Multi-Scale Modal Meshing is difficult
Analysis
Example-Based
No real control on the FEM resolution
Synthesis
No clear interface between physics and audio
Perspectives on
a Hybrid Model
Conclusion and
Discussion
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
20

26. t
Vibration Models
Sound and
Virtuality Modal Analysis
Physics-Based
Synthesis
Contact Modeling
Audio Texture
What Are
Synthesis For Complex
Contacts The Current Limitations?
Resonator Modeling
A Robust and
Multi-Scale Modal Meshing is difficult
Analysis
Example-Based
No real control on the FEM resolution
Synthesis
No clear interface between physics and audio
Perspectives on
a Hybrid Model
Conclusion and
Discussion
HOW Can We Solve Them?
By proposing
A robust and multi-scale modal analysis
which is
Coherent with the physics simulation
[Picard et al., DAFx 09]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
20

27. t
Our Deformation Model
Sound and
Virtuality
Physics-Based
Synthesis Inspired from Work by Nesme et al.
Contact Modeling
Audio Texture
[Nesme et al. 06]
Synthesis For Complex
Contacts
Resonator Modeling
A Robust and Technique
Multi-Scale Modal
Analysis Merged voxels used as Hexahedral Finite Elements
Example-Based
Synthesis
Perspectives on
a Hybrid Model
Conclusion and
Discussion
Implementation with the Sofa Framework
Validation of the Model
Tests on a metal cube
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
21

28. t
Robustness
Sound and
Virtuality
Physics-Based
Synthesis
Contact Modeling
Audio Texture Robust Even for Non-Manifold Geometries
Synthesis For Complex
Contacts
Resonator Modeling
A Robust and
Multi-Scale Modal
Analysis
Example-Based
Synthesis
Perspectives on
a Hybrid Model
Conclusion and
Discussion
Material: Aluminium
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
22

29. t
Multi-Scale for Efficient Memory Usage
Sound and
Virtuality
Physics-Based
A Squirrel in Pine Wood
Synthesis
Contact Modeling
Audio Texture
Synthesis For Complex
Contacts
Resonator Modeling
A Robust and
Multi-Scale Modal
Analysis
Example-Based
Synthesis
Perspectives on
a Hybrid Model
Conclusion and
Discussion
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
23

30. t
Multi-Scale for Efficient Memory Usage
Sound and
Virtuality
Physics-Based
A Squirrel in Pine Wood: Different FE resolutions
Synthesis
Contact Modeling
Audio Texture
3x3x3 4x4x4 8x8x8 9x9x9
Synthesis For Complex
Contacts
Resonator Modeling
A Robust and
Multi-Scale Modal
Analysis
Example-Based
Synthesis
Perspectives on
a Hybrid Model
Frequency Content = f (Hexahedral FE Resolution)
Conclusion and
Discussion
Higher resolution models
Frequency centroid shift
Convergence of the frequency content
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
24

31. t
Comparison with Classical Approach
Sound and
Virtuality
Physics-Based Sounding Bowl - Material: Aluminium
Synthesis
Contact Modeling
Audio Texture Classical Approach Our Approach
Synthesis For Complex
Contacts (816 modes) (75 modes)
Resonator Modeling
A Robust and
Multi-Scale Modal
Analysis
Example-Based
Synthesis
Perspectives on
a Hybrid Model
Conclusion and
Discussion
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
25

32. t
A Robust and Multi-Scale Modal Analysis
Sound and
Virtuality A Solution for Sound Synthesis
Physics-Based
Synthesis
Contact Modeling
Audio Texture
Synthesis For Complex
Contacts
Resonator Modeling
A Robust and
Multi-Scale Modal
Analysis
Example-Based Realistic
Synthesis
Perspectives on
a Hybrid Model Adapted to Non-Manifold Geometries
Conclusion and
Discussion Resources Flexibility
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
26

33. t
Overview
Sound and
Virtuality
Physics-Based
Synthesis 1 Sound and Virtuality
Example-Based
Synthesis 2 Physics-Based Sound Synthesis
Flexible Sound
Synthesis Contact Modeling
Retargetting Example
Sounds
Resonator Modeling
Perspectives on
a Hybrid Model
3 Example-Based Synthesis
Conclusion and
Discussion Flexible Sound Synthesis
4 Perspectives on a Hybrid Model
Motivation and Application
5 Conclusion and Discussion
Contributions
Extensions and Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
27

34. t
Implementation of Signal-Based Models
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based
Synthesis
Flexible Sound
Concatenative Synthesis
Synthesis
Retargetting Example
[Roads 91, Schwarz 06]
Sounds
Perspectives on Sound Textures Based on Physics
a Hybrid Model
[Cook 99]
Conclusion and
Discussion [Dobashi et al. 03, Zheng et al. 09] Dobashi et al. 03
Authoring and Interactive Control
[Cook 02]
Cook 99
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
28

35. t
Implementation of Signal-Based Models
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based
Synthesis
What Are
Flexible Sound
Synthesis
The Current Limitations?
Retargetting Example
Sounds
Processing is not generic
Perspectives on
a Hybrid Model Parametrizing is difficult
Conclusion and
Discussion
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
29

36. t
Implementation of Signal-Based Models
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based
Synthesis
What Are
Flexible Sound
Synthesis
The Current Limitations?
Retargetting Example
Sounds
Processing is not generic
Perspectives on
a Hybrid Model Parametrizing is difficult
Conclusion and
Discussion
HOW Can We Solve Them?
By
Retargetting example sounds
To physics-driven animation
[Picard et al., AES 09]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
29

37. t
Our Approach
Sound and
Virtuality
Physics-Based
Synthesis SINUSOIDAL
AUDIO
RECORDING
Example-Based TRANSIENT
Synthesis OBJECT GEOMETRY
VIRTUAL ENVIRONMENT
Flexible Sound
1 DICTIONARY OF AUDIO GRAINS
Synthesis
Impulsive / Continuous
Retargetting Example BUILD COLLISION
Sounds STRUCTURES
Perspectives on 2 CORRELATION PATTERNS
a Hybrid Model DEFINE PROCEDURES
PREPROCESSING
INTERACTIVE
Conclusion and RETARGETTING RIGID-BODY
Discussion TO ANIMATION SIMULATION
AUDIO RENDERER VIDEO RENDERER
ANIMATION WITH AUDIO
Amplitude
Time
Our Contributions
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
30

38. t
Preprocess: A Generic Analysis
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based
Synthesis
Flexible Sound
Synthesis Impulsive and Continuous Contacts
Retargetting Example
Sounds
Spectral Modeling Synthesis (SMS) [Serra 97]
Perspectives on
a Hybrid Model
Conclusion and
Automatic Extraction of Audio Grains
Discussion Dictionary: Impulsive/Continuous
Generation of Correlation Patterns
between original recordings and audio grains
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
31

39. t
On-Line: Flexible Sound Synthesis
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based
Synthesis Resynthesis of the Original Recordings
Flexible Sound
Synthesis Candidate grains: max. correlation amplitude
Retargetting Example
Sounds
Perspectives on
a Hybrid Model
Conclusion and
Discussion
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
32

40. t
On-Line: Flexible Sound Synthesis
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based
Synthesis Resynthesis of the Original Recordings
Flexible Sound
Synthesis Candidate grains: max. correlation amplitude
Retargetting Example
Sounds
Perspectives on Interactive Physics-Driven Animations
a Hybrid Model
Physics Info for Retargetting
Conclusion and
Discussion Contact type: impulsive or continuous?
Penetration force and relative velocity
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
32

41. t
On-Line: Flexible Sound Synthesis
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based
Synthesis Resynthesis of the Original Recordings
Flexible Sound
Synthesis Candidate grains: max. correlation amplitude
Retargetting Example
Sounds
Perspectives on Interactive Physics-Driven Animations
a Hybrid Model
Physics Info for Retargetting
Conclusion and
Discussion Contact type: impulsive or continuous?
Penetration force and relative velocity
Flexible Audio Shading Approach
Additional, User-defined Resynthesis Schemes
Spectral domain adaptation/modification
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
32

42. t
Resynthesis of the Original Recordings
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based
Synthesis
Flexible Sound
94 recordings (14.6Mb)
Synthesis ≈ 5000 grains + 94 Correlation Patterns (20% Gain)
Retargetting Example
Sounds
Perspectives on Breaking Glass
a Hybrid Model
Conclusion and Shooting Gun
Discussion
Rolling
Additional Material:
http://www-sop.inria.fr/members/Cecile.Picard/
"‘Supplemental AES"’
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
33

43. t
Flexible Audio Shading Approach
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based
Synthesis
Easy Implementation of Time-Scaling
Flexible Sound
Synthesis
Retargetting Example
Faster Rolling
Sounds
Perspectives on Slower Breaking
a Hybrid Model
Conclusion and
Discussion Synthesis of An Infinity Similar Audio Events
by varying the audio content
Rythmic pattern from Breaking Stone
New material content: stone and gun
Rythmic pattern from Breaking Glass
New material content: ceramic
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
34

44. t
Interactive Physics-Driven Animations
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based
Synthesis
Flexible Sound
Simulations Driven by Sofa Framework
Synthesis
Retargetting Example
Sounds
Perspectives on
a Hybrid Model
Conclusion and
Discussion
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
35

45. t
Retargetting Example Sounds
Sound and
Virtuality A Solution for Interactive Simulations
Physics-Based
Synthesis
Example-Based
Synthesis
Flexible Sound
Synthesis
Retargetting Example
Variety
Sounds
Perspectives on
a Hybrid Model Adapted to Scenarios
Conclusion and
Discussion
Small Memory Footprint
Real-Time Rendering
An attractive solution for industrial applications
(Eden Games, an ATARI game studio)
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
36

46. t
Overview
Sound and
Virtuality
Physics-Based
Synthesis 1 Sound and Virtuality
Example-Based
Synthesis 2 Physics-Based Sound Synthesis
Perspectives on
a Hybrid Model
Contact Modeling
Motivation Resonator Modeling
A Hybrid Model for
Fracture Events
Conclusion and
3 Example-Based Synthesis
Discussion
Flexible Sound Synthesis
4 Perspectives on a Hybrid Model
Motivation and Application
5 Conclusion and Discussion
Contributions
Extensions and Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
37

47. t
Sound Modeling
Sound and
Virtuality When Nonlinearity Occurs
Physics-Based
Synthesis
Example-Based
Synthesis
Perspectives on
a Hybrid Model
Motivation
Problems of Single Models
A Hybrid Model for
Fracture Events Vibration models assume linearity
Conclusion and
Discussion
Example-based sounds are hard to parametrize
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
38

48. t
Sound Modeling
Sound and
Virtuality When Nonlinearity Occurs
Physics-Based
Synthesis
Example-Based
Synthesis
Perspectives on
a Hybrid Model
Motivation
Problems of Single Models
A Hybrid Model for
Fracture Events Vibration models assume linearity
Conclusion and
Discussion
Example-based sounds are hard to parametrize
Previous Work
Modeling nonlinearities
[O Brien et al. 01, Chadwick et al. 09]
[Cook 02]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
38

49. t
Fracture Events
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based Background
Synthesis
Frequently occur in virtual environments
Perspectives on
a Hybrid Model
Motivation Visual rendering
A Hybrid Model for
Fracture Events [O Brien et al. 99, 02]
Conclusion and [Parker and O Brien. 09]
Discussion
Sound rendering: Little research
[Warren et al. 84] [Rath et al. 03]
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
39

50. t
Fracture Events
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based Background
Synthesis
Frequently occur in virtual environments
Perspectives on
a Hybrid Model
Motivation Visual rendering
A Hybrid Model for
Fracture Events [O Brien et al. 99, 02]
Conclusion and [Parker and O Brien. 09]
Discussion
Sound rendering: Little research
[Warren et al. 84] [Rath et al. 03]
Challenges
Event depends on the material involved
Differents phases emerge from fracture event
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
39

51. t
Parametrization of Our Hybrid Model
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based
Synthesis Selection Criteria
Perspectives on Hybrid model applied when nonlinearity occurs
a Hybrid Model
Motivation
A Hybrid Model for
Fracture Events Techniques
Conclusion and
Discussion
FM synthesis
Audio grains
FM synthesis
Parametrization
Smooth transition with vibration model
Coherence inside the hybrid model
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
40

52. t
Discussion
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based
Synthesis
Perspectives on
a Hybrid Model
Motivation Prospective model
A Hybrid Model for
Fracture Events
Conclusion and Possible problem: report from the physics engine
Discussion
Simplicity of the tools allows real-time rendering
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
41

53. t
Overview
Sound and
Virtuality
Physics-Based
Synthesis 1 Sound and Virtuality
Example-Based
Synthesis 2 Physics-Based Sound Synthesis
Perspectives on
a Hybrid Model
Contact Modeling
Conclusion and
Resonator Modeling
Discussion
Contributions 3 Example-Based Synthesis
Extensions and
Applications
Flexible Sound Synthesis
4 Perspectives on a Hybrid Model
Motivation and Application
5 Conclusion and Discussion
Contributions
Extensions and Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
42

54. t
Synthesis of Sounds for Animation
Sound and
Virtuality Difficulties
Physics-Based
Synthesis
Example-Based
Synthesis
Perspectives on
a Hybrid Model
Conclusion and
Audio-Visual Coherence
Discussion
Contributions
Extensions and
Applications
Extremely Dynamic Character
Precision of Synthesis
Large Variety of Objects
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
43

55. t
Contributions
Sound and
Virtuality An Overview
Physics-Based
Synthesis
Example-Based Complex Contact Modeling
Synthesis
Perspectives on
2D visual textures used as roughness maps
a Hybrid Model Audible and position-dependent variations
Conclusion and Detail-layer mechanisms
Discussion
Contributions
Extensions and
Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
44

56. t
Contributions
Sound and
Virtuality An Overview
Physics-Based
Synthesis
Example-Based Complex Contact Modeling
Synthesis
Perspectives on
2D visual textures used as roughness maps
a Hybrid Model Audible and position-dependent variations
Conclusion and Detail-layer mechanisms
Discussion
Contributions
Extensions and
Applications
Improved Modal Analysis for Resonator Modeling
Complex non-manifold geometries can be handled
Multi-scale resolution
Coherence between simulation and audio
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
44

57. t
Contributions
Sound and
Virtuality An Overview
Physics-Based
Synthesis
Example-Based Complex Contact Modeling
Synthesis
Perspectives on
2D visual textures used as roughness maps
a Hybrid Model Audible and position-dependent variations
Conclusion and Detail-layer mechanisms
Discussion
Contributions
Extensions and
Applications
Improved Modal Analysis for Resonator Modeling
Complex non-manifold geometries can be handled
Multi-scale resolution
Coherence between simulation and audio
Flexibility of Sound Design
Audio grains and correlation patterns
Dynamic retargetting to events
Extended sound synthesis capabilities
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
44

58. t
Contributions
Sound and
Virtuality Perspectives
Physics-Based
Synthesis
Example-Based
Synthesis
Perspectives on
a Hybrid Model
Conclusion and
Discussion
A Prospective Hybrid Model
Contributions for Complex Physical Phenomena
Extensions and
Applications Focus on Nonlinearity
Combination of physically based
and example-based methods
Application Case: Fracture Events
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
45

59. t
Overview
Sound and
Virtuality
Physics-Based
Synthesis 1 Sound and Virtuality
Example-Based
Synthesis 2 Physics-Based Sound Synthesis
Perspectives on
a Hybrid Model
Contact Modeling
Conclusion and
Resonator Modeling
Discussion
Contributions 3 Example-Based Synthesis
Extensions and
Applications
Flexible Sound Synthesis
4 Perspectives on a Hybrid Model
Motivation and Application
5 Conclusion and Discussion
Contributions
Extensions and Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
46

60. t
Promising Directions for Future Work
Sound and
Virtuality
Physics-Based
Synthesis
Complex Contact Modeling
Example-Based
Synthesis Two interacting textures
Perspectives on Surface-based interactions
a Hybrid Model
Adequate perceptual experiments
Conclusion and
Discussion
Contributions
Extensions and
Applications
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
47

61. t
Promising Directions for Future Work
Sound and
Virtuality
Physics-Based
Synthesis
Complex Contact Modeling
Example-Based
Synthesis Two interacting textures
Perspectives on Surface-based interactions
a Hybrid Model
Adequate perceptual experiments
Conclusion and
Discussion
Contributions Improved Modal Analysis for Resonator Modeling
Extensions and
Applications Recent work from [Nesme et al. Siggraph 09]
Investigations with GPU for in-line computation
Complete integration in a virtual scene
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
47

62. t
Promising Directions for Future Work
Sound and
Virtuality
Physics-Based
Synthesis
Complex Contact Modeling
Example-Based
Synthesis Two interacting textures
Perspectives on Surface-based interactions
a Hybrid Model
Adequate perceptual experiments
Conclusion and
Discussion
Contributions Improved Modal Analysis for Resonator Modeling
Extensions and
Applications Recent work from [Nesme et al. Siggraph 09]
Investigations with GPU for in-line computation
Complete integration in a virtual scene
Example-Based Technique
Clustering of similar grains
Statistical analysis of correlation patterns
Physics engine design
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
47

63. t
Promising Directions for Future Work
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based
Synthesis
Perspectives on
a Hybrid Model
Conclusion and
Discussion Hybrid Model for Fracture Events
Contributions
Extensions and
Fracture sound simulation framework
Applications
Tracking of relevant physical data
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
48

64. t
Conclusion
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based
Synthesis
Perspectives on
a Hybrid Model New Physically Based Algorithms
Conclusion and
Discussion
for Sound Rendering
Contributions
Extensions and
Flexibility of Sound Modeling
Applications
Ideas on an Adequate Hybrid Sound Model
Additional info:
http://www-sop.inria.fr/members/Cecile.Picard/
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
49

65. t
Acknowledgements
Sound and
Virtuality
Physics-Based
Synthesis
Example-Based
Synthesis
Perspectives on
a Hybrid Model George Drettakis, François Faure,
Conclusion and and Nicolas Tsingos
Discussion
Contributions REVES Team
Extensions and
Applications Marie-Paule Cani and the Evasion Team
Paul G. Kry at the McGill University, Montréal
Eden Games, an ATARI game studio, Lyon
C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation
50