1. MPEG 3D Graphics
Recent achievements and future roadmap
GLOBAL 3D Technology Forum, October 9-12, KINTEX,
Korea
Marius Preda, MPEG 3DG Chair
Institut Mines TELECOM
http://www.slideshare.net/MariusPreda/3DgraphicsAndAR

2. 3D Graphics World
 No common representation
– Heterogeneous kind of data
• Different types of geometry, appearance and animation
models
– Always easier to specify a new data representation
format than learning an existent one
 Very different application domains
– Different requirements and use cases
Towards a standard representation: a double problem

3. 3D Graphics Chain : Authoring and Publishing
Very heterogeneous production chain:
-A 3D Asset is processed by several authoring tools, each
one with dedicated functionalities
-Each authoring tool has its own format
-Artists and creators are investing time in
learning how the tool works
Authoring

4. In BOX solutions
- Games are containing all the data on DVD
- Rendering engine friendly data format
ONLINE solutions
- ad hoc format, usually zipped version of an
authoring format
3D Graphics Chain : Authoring and Publishing
Publishing

5. FAST
TRANSPORT
TROUGHT THE
NETWORK
Size doesn’t matter
Size is of main
importance
Authoring
Size doesn’t matter
Transmission Publishing
3D Graphics Chain : Authoring and Publishing
Gap between the authoring and publishing formats

6. FAST
TRANSPORT
TROUGHT THE
NETWORK
Size doesn’t matter
Size is of main
importance
Authoring
Size doesn’t matter
Transmission
Encoder Decoder
Publishing
MPEG-4 3DG
3D Graphics Chain : Authoring and Publishing
Solving the gap: encoding layer

7. To define a standard format for compressed 3D
synthetic content. In other words to be for graphics
what MP3 and AAC are for audio, MPEG-2 and
MPEG-4 are for video and JPEG is for still images.
Additionally "MPEG 3D Graphics" aims at providing
mechanisms such as APIs to enable easy integration
and development of applications using its standard
representation tools.
3D Graphics Chain : Authoring and Publishing
MPEG 3DG Vision

8. Topics of the day
Ongoing projects: integrating real and virtual
MPEG offer in the 3D Graphics field
What is MPEG?
MPEG 3D Graphics

9. Topics of the day
MPEG offer in the 3D Graphics field
What is MPEG?
MPEG 3D Graphics
Ongoing projects: integrating real and virtual

10. What is MPEG?
Coding/compression of elementary media:
– Audio (MPEG-1, 2 and 4)
– Video (MPEG-1, 2 and 4)
– 2D/3D graphics (MPEG-4)
Storage and Transport
– MPEG-2 Transport
– File Format (MPEG-4)
– Dynamic Adaptive Streaming over HTTP (DASH)
Hybrid (natural & synthetic) scene description, user interaction (MPEG-4)
Metadata (MPEG-7)
Media management and protection (MPEG-21)
Sensors and actuators, virtual worlds (MPEG-V)
Advanced User interaction (MPEG-U)
Media-oriented middleware (MPEG-M)
More ISO/IEC standards under development for
– 3D Video, 3D Audio
– Coding and Delivery in Heterogeneous Environments
– …
A suite of ~130 ISO/IEC standards

11. A standardization activity continuing for 24 years
– Supported by several hundreds companies/organisations from ~25 countries
– ~500 experts participating in quarterly meetings
– More than 2300 active contributors
– Many thousands experts working in companies
A proven manner to organize the work to deliver useful and used standards
– Developing standards by integrating individual technologies
– Well defined procedures
– Subgroups with clear objectives
– Ad hoc groups continuing coordinated work between meetings
MPEG standards are widely referenced by industry
– 3GPP, ARIB, ATSC, DVB, DVD-Forum, BDA, EITSI, SCTE, TIA, DLNA, DECE, OIPF…
Billions of software and hardware devices built on MPEG technologies
– MP3 players, cameras, mobile handsets, PCs, DVD/Blue-Ray players, STBs, TVs,
…
What is MPEG?
Involvement, approach, deployment

12. Topics of the day
MPEG offer in the 3D Graphics field
What is MPEG?
MPEG 3D Graphics
Ongoing projects: integrating real and virtual

13. MPEG technologies related to 3D Graphics
MPEG-1/2
(AV content)
1992/4
VRML
1997
• Part 11 - BIFS:
-Binarisation of VRML
-Extensions for streaming
-Extensions for server command
-Extensions for 2D graphics
- Real time augmentation with
audio & video
• Part 2 - Visual:
- 3D Mesh compression
- Face animation
1998
• Part 2 – Visual
- Body animation
1999
MPEG-4 v.1
MPEG-4 v.2
First form of broadcast signal augmentation

14. MPEG technologies related to 3D Graphics
MPEG-4
2003
•AFX 2nd Edition:
- Animation by
morphing
- Multi-texturing
2005
• AFX 3rd Edition
- WSS for terrain
and cities
- Frame based
animation
2007
MPEG-4
MPEG-4
• Part 16 - AFX:
- A rich set of 3D
graphics tools
- Compression of
geometry,
appearance,
animation
• AFX 4th Edition
- Scalable complexity
mesh coding
2011
MPEG-4
A rich set of 3D Graphics
representation and
compression tools

15. MPEG technologies related to 3D Graphics
MPEG-4
2003
•AFX 2nd Edition:
- Animation by
morphing
- Multi-texturing
2005
• AFX 3rd Edition
- WSS for terrain
and cities
- Frame based
animation
2007
MPEG-4
MPEG-4
• Part 16 - AFX:
- A rich set of 3D
graphics tools
- Compression of
geometry,
appearance,
animation
• AFX 4th Edition
- Scalable complexity
mesh coding
2011
MPEG-4
2009
• Part 25
- Compression of
third-party XML
(X3D, COLLADA)
MPEG-4
2004
• Part 16
- X3D
Interactive
Profile
MPEG-4

16. MPEG 3DG Tools
Compression Tool Type
3D Mesh Compression – 3DMC Geometry
Scalable Complexity 3DMesh Coding – SC3DMC Geometry
Wavelet Subdivision Surface – WSS Geometry
Coordinate, Orientation and Position Interpolator – IC Animation
Bone-based Animation – BBA Animation
Frame-based Animated Mesh Compression - FAMC Animation
Octree Compression for Depth Image-based Representation Appearance
Point Texture Appearance
FAMC
IC
WSS
3DMC
BBA

17. MPEG 3DG Tools
Compression Tool Type
3D Mesh Compression – 3DMC Geometry
Scalable Complexity 3DMesh Coding – SC3DMC Geometry
Wavelet Subdivision Surface – WSS Geometry
Coordinate, Orientation and Position Interpolator – IC Animation
Bone-based Animation – BBA Animation
Frame-based Animated Mesh Compression - FAMC Animation
Octree Compression for Depth Image-based Representation Appearance
Point Texture Appearance
FAMC
IC
WSS
3DMC
BBA

18. • Establish a standardized set of Functional Units (FUs)
• Combine the FUs in a consistent decoder by describing it
and transmit the description together with the encoded
data
Reconfigurable Graphics Coding

19. Graphics Tool Library
START
Loop until the following
is available:
- codeBytes
[INPUT]:
codeBytes_i
[PROCESS]:
length
value
n,s,x,y
Loop until the following
is available:
- codeBuffer
[INPUT]:
codeBuffer_i
codeBytes < 4
else
[INPUT]:
useDecoderTable_i
Loop until the following
is available:
- useDecoderTable
useDecoderTable
=
1
[INPUT]:
decoderTableSize_i
distributionSize_i
Loop until the following
are available:
- decoderTableSize
- distributionSize
[INPUT]:
decoderTable_i
[decoderTableSize]
distribution_i [distributionSize]
lastSymbol_i
Loop until the following
are available:
- decoderTable [ decoderTableSize ]
- distribution [ distributionSize ]
- lastSymbol
[PROCESS]:
s
n
n > s + 1 [PROCESS]:
m
[PROCESS]:
n
distribution[m] > dv
else
[PROCESS]:
s
[PROCESS]:
x
else
[PROCESS]:
y
s != lastSymbol
else
[PROCESS]:
x
s
length
m
[PROCESS]:
z
[PROCESS]:
n, y
[PROCESS]:
s,x
m != s
[PROCESS]:
m
z > value
else
[PROCESS]:
value
length
else
else
[PROCESS]:
value
ac_pointer
length
length < AC_MinLength
length < AC_MinLength
else
[PROCESS]:
symbolCount [ s ]
symbolsUntilUpdate
[EMBED]:
update
[OUTPUT]:
dataOut_o = s
symbolsUntilUpdate = 0 else
[INPUT]:
tableShift_i
Loop until the following
is available:
- tableShift
[INPUT]:
dataSymbols_i
Loop until the following is available:
- dataSymbols
[INPUT]:
reset_i
Loop until the following
is available:
- reset
[PROCESS]:
bufferSize
newBuffer
codeBuffer
ac_pointer
dataModelFlag [] = 0
reset
=
1
else
[INPUT]:
dataModelIndex_i
Loop until the following
is available:
- dataModelIndex
[PROCESS]:
dv
t
dataModelFlag
[
dataModelIndex
]
=
0
dataModelFlag
[
dataModelIndex
]
=
0
else
else
dataModelFlag
[
dataModelIndex
]
=
0
else
[PROCESS]:
dataModelFlag
[dataModelIndex ]
= 1
ArithmeticDecode_AdaptiveData
dataSymbols_i
lastSymbol_i
tableBits_i
tableShift_i
distribution_i
useDecoderTable_i
decoderTable_i
tableSize_i
symbolCount_i
dataModelIndex_i
codeBytes_i
codeBuffer_i
[ AC_MaxLength ]
[ DM__LengthShift ]
[ N_DATA_MODELS ]
dataOut_o
FU Name
Algo_ParserSC3DMC
Algo_InverseQuantization1D
Algo_InverseQuantizationND
Algo_InversePrediction1D
Algo_InversePredictionND
Algo_ArithmeticCodec_StaticBit
Algo_ArithmeticCodec_AdaptiveBit
Algo_AdaptiveData_Model_GenerateAlphabet
Algo_ArithmeticDecode_AdaptiveData
Algo_ReOrderConnec1D
Algo_ReOrderConnecND
Algo_ReOrderTab1D
Algo_ReOrderTabND
Algo_ReOrderPerm1D
Algo_ReOrderPermND
Algo_InverseBinarization
Algo_Provider_1D
Algo_Provider_2D
Algo_GeometryDecoder_TFAN
Algo_Decode_Symbol
Algo_Decode_I_Frame_Quantized
Algo_Decode_P_Frame
Algo_Huffman_Decode
Algo_DCT_Dequant
Algo_IDCT_Trans
Algo_InterpolateCoord
Algo_QuatNorm
Algo_QuatConj
Algo_QuatScalar_NotNorm
Reconfigurable Graphics Coding

20. Decoding network example
Reconfigurable Graphics Coding
A sub-network of SC3DMC

21. Applications
Phone and tablet players
Web databases
(with and
without plugins - WebGL)
Games

22. Topics of the day
Ongoing projects: integrating real and virtual
MPEG offer in the 3D Graphics field
What is MPEG?
MPEG 3D Graphics

23. MPEG technologies related to real world
MPEG-V - Media
Context and Control
2011
• 2nd Edition:
- GPS
- Biosensors
- 3D Camera
201x
• Compression
of video +
depth
201x
MPEG-V
- 3D Video
• 1st Edition
- Sensors and
actuators
- Interoperability
between Virtual
Worlds
• Feature-point based
descriptors for image
recognition
201x
CDVS
MPEG-U –
Advanced
User Interface
2012
A rich set of sensors and
actuators
- 3D Audio
MPEG-H

24. MPEG-V: Sensors and Actuators

25. Bridging Virtual and Real Worlds
Real World
(Sensor Device)
Real World
(Actuators)
User
Virtual Objects and Virtual World
Augmented
Reality

26. • All AR-related data is available from MPEG standards
• Real time composition of synthetic and natural objects
• Access to
– Remotely/locally stored BIFS/compressed 2D/3D mesh
objects
– Streamed real-time BIFS/compressed 2D/3D mesh objects
• Inherent object scalability (e.g. for streaming)
• User interaction & server generated scene changes
• Physical context
– Captured by a broad range of standard sensors
– Affected by a broad range of standard actuators
Main features of MPEG AR technologies

27. MPEG vision on AR, the MPEG AR Browser
Point to a URL – no need to download new applications for each context.
The browser
– Retrieves scenario from the internet
– Starts video acquisition
– Tracks objects
– Recognizes objects from visual signatures
– Recovers camera pose
– Gets streamed 3D graphics
– Composes new scenes
– Gets inputs from various sensors
– Offers optimal AR experience by constantly adapting interaction
possibilities and objects from a remote server.
Industry
– Maximize number of customers through MPEG-compliant authoring
tools and browsers
– No need to develop a new application for each use case and device
platform

28. MPEG vision on AR
MPEG-4/MPEG-7/MPEG-21/
MPEG-U/MPEG-V
MPEG Player
Compression
Authoring Tool
Produce
Download

29. Architecture
AR Player
Media
Servers
Service
Servers
User
Local
Sensors &
Actuators
Remote
Sensors &
Actuators
AR file or
stream
Local
Real World
Environment
Remote
Real World
Environment

30. AR Player
Media
Servers
Service
Servers
User
Local / Remote Context
AR
Document
Enterprise viewpoint: global architecture and actors
Augmented Reality Reference Model
ARTC
AREC
AC
AR Tools Creator (ARTC)
AR Experience Creator (AREC)
Assets Creator (AC)
AC AA
Assets Aggregator (AA)
Device Manufacturer (DM)
Middleware/Component Provider (MCP)
DM MCP
MCP
End-User (EU)
EU
Telecommunication Operator (TO)
TO TO
TO
TO
Online Middleware/Component
Provider (OMCP)
AR Service Provider (ARSP)
OMCP ARSP

31. Information viewpoint
Augmented Reality Reference Model
AR Player
Media
Servers
Service
Servers
User
Local/Remote Context
AR
Document
Scene/Real World
• Raw image
• Sensed data
• Virtual Camera view
• Detected features
• Area of Interest/Anchors
Tracking objects
• Markers
• Marker-less
Device Context
• Device capabilities
Location of Device
• Location
• Orientation
Spatial Models
• Coordinate Ref. Sys.
• (Geol)ocation
• Projections
• Coordinate conversion
Presentation
• Augmentation
• Registration
• Styling/complexity
• Spatial Filtering, e.g.
range
User Input
• Query
• Manipulation of
Presentation
• Topics of interest
• Preferences
Digital Assets
• Presentation data
• Trigger/Event rules
• Accuracy based

32. AR Player
Media
Servers
Service
Servers
User
Local / Remote Context
AR
Document
Computational viewpoint
Augmented Reality Reference Model
1
2
3 4
5

33. AR Player
Media
Servers
Service
Servers
User
Local / Remote Context
AR
Document
Computational viewpoint
Augmented Reality Reference Model
1
2
4 5
3

34. Engineering viewpoint
Augmented Reality Reference Model
AR Player
Media
Servers
Service
Servers
User
Local/Remote Context
AR
Document
Camera Mic
Accelero
meter
Compass GPS …
Rendering
Engine
Display
(A/V/H)
Application
Engine
…

35. Glossary
Augmented Reality Reference Model

36. Use cases
Augmented Reality Reference Model

37. How to contribute?
Augmented Reality Reference Model
Use Trac!
http://wg11.sc29.org/trac/augmentedreality/

38. A set of scene graph nodes/protos as defined in MPEG-4 Part 11
– Existing nodes
– Audio, image, video, graphics, programming, communication, user
interactivity, animation
– New standard PROTOs
– Map, MapMarker, Overlay, ReferenceSignal,
ReferenceSignalLocation, CameraCalibration, AugmentedRegion
Connection to sensors as defined in MPEG-V
– Orientation, Position, Angular Velocity, Acceleration, GPS, Geomagnetic,
Altitude
– Local camera sensor
Compressed media
Three components: scene, sensors/actuators, medias
MPEG-A Part 13 ARAF

39. MPEG-A Part 13 ARAF
AR Quiz Augmented Book
MPEG Augmented Reality examples

40. MPEG Augmented Reality examples
MPEG-A Part 13 ARAF
AR Quiz Augmented Book
http://youtu.be/LXZUbAFPP-Y
http://youtu.be/la-Oez0aaHE

41. • MPEG standardized a rich set of 3D Graphics representations
and theirs compressions
• MPEG recently proposed an approach to address the complexity
of 3DG domain: Reconfigurable Graphics Coding
• Joint development of an AR Reference Model (with SC24/WG9,
ARStandards, Khronos, OGC)
– The community at large is invited to contribute
– http://wg11.sc29.org/trac/augmentedreality
• MPEG promoted at CD a first version of an integrated and
consistent solution for representing content in AR applications
and services
– Continue synchronized/harmonized development of technical
specifications with X3D, COLLADA, OGC content models
Conclusions

42. • www.MyMultimediaWorld.com
• www-artemis.it-sudparis.eu
• marius.preda@it-sudparis.eu
Thank you !