This document discusses using grid technology for distributed media processing tasks like video transcoding. It presents the MediaGrid concept of sharing heterogeneous storage and computational resources across organizations. Test results show distributing video transcoding across multiple servers can significantly reduce processing time. Simulation results indicate total job time is highly dependent on available WAN bandwidth when outsourcing to remote resource providers. The conclusions are that grid technology is viable for media production tasks by enabling parallelism, but technical limitations exist when using remote resources over insufficient network connections.

1. Grid technology
for next-gen media processing
Jens Buysse - Stijn De Smet - Koen Segers-
Bruno Volckaert

2. Overview
 MediaGrid concept
 Distributed video transcoding
 Enabling technologies
 Setup overview
 Test results
 Simulation results
 Conclusions
2

3. MEDIAGRID CONCEPT
3

4. Originating problems
 Tape-based media to file-based media
 Multitude of file-based media transfers and processing
Storage / retrieval / transfer of media

Conforming

Transcoding

Upscaling

Editing

 Geographically disperse facilities / resources / media storage
4

5. Grid technology as solution?
 Grid technology
a Grid is a distributed processing architecture where heterogeneous
resources are shared between different participating organizations, across
an interconnecting network
 Resources
 Storage (media archive, temporary storage, etc.)
 Computational (rendering farm, work stations, etc.)
 Specialized (broadcasting, ingesting, etc.)
 High speed interconnecting network (1-10 Gbit/s)
5

6. MediaGrid
M M
M hires hires
hires
6

7. MediaGrid
M M
M hires hires
hires
Grid Middleware
EDL
7

8. MediaGrid: enabling virtual organisations
VO 1
VO 2
8

9. DISTRIBUTED VIDEO
TRANSCODING
9

10. Grid technology proof-of-concept
 Investigated the viability of Grid technology for processing tasks
in media production / distribution companies
 Transcoding of media
 Upscaling of media
Video transcoding deals with converting a video signal into
another one with different format, such as different bit rate,
frame rate, frame size, or even compression standard
 Video transcoding is a resource intense process
 I/O
 Processing needs
10

11. Need for transcoded / rescaled video
VRT online media YouTube
http://www.deredactie.be http://www.youtube.com
11

12. Distributed video transcoding
 How can we accelerate this process?
Server 4
Server 3
Server 1 Server 2
00:00:00  00:51:53
00:00:00 00:13:15 00:13:15 00:26:30 00:26:30 00:39:45 00:39:45 00:51:53
12

13. ENABLING TECHNOLOGIES
13

14. Enabling technologies
 OS
 SuSe enterprise
 Transcoding software
 Transcode library
 Grid Middleware
 TORQUE (openPBS)
 Maui scheduler
 Grid distributed transcoder: custom Java application
 Data retrieval / storage technology
 GPFS
14

15. Enabling technologies: TORQUE
 TORQUE : open PBS
TORQUE Server
Maui
Sheduler pbs_mom
Job
Queue 1 Policy
User
Queue 2
pbs_mom
15

16. Enabling technologies
 Job / batch / workflow submitter
 Consider job dependencies
1
2
Stock 1 3 4
5 6
Stock 2
7
8
Stock 3 16

17. Enabling technologies
 Grid distributed transcoding application
17

18. Enabling technologies
 Grid distributed transcoding application
18

19. SETUP OVERVIEW
19

20. Setup overview
 … TORQUE
 … with GPFS cluster as media storage
 … Java distributed transcoding front-end
 … on each computational resource Transcode libraries
 … the will to transcode in a distributed fashion
20

21. First distributed transcoding workflow
TORQUE
1. Split phase 00:00:00 00:13:15
00:13:15 00:26:30
2. Transcoding
3. Merge phase
Node 1
00:26:30 00:39:45
00:39:45
00:51:53
00:00:00 00:51:53
Node 2
User
Node 3
00:13:15 00:26:30
00:26:30 00:39:45
00:39:45 00:13:15
00:00:00
00:51:53
Node 4
00:00:00 00:51:53
GPFS 21

22. Current distributed transcoding workflow
TORQUE Nav.log
1. Preprocess phase 00:00:00 00:13:15
2. Demux phase
3. Transcoding
4. Merge / multiplex
Node 1
Audio.mp3
00:13:15 00:26:30
00:00:00 00:51:53
Node 2
User
00:26:30 00:39:45
Node 3
00:39:45 00:51:53
Node 4
00:00:00 00:51:53
GPFS 22

23. Future distributed transcoding workflow
TORQUE
Node 1
User
Node 2
WAN
1. Prefetch Node 3
2. Preprocess
3. Demux
4. Transcode
5. Merge /
00:00:00 00:51:53
GPFS Remote multiplex 00:00:00 00:51:53
GPFS local
23

24. Discussion
 Old version
 Video files were physically split
 Split / merge step could introduce artifacts
 Current version
 File is inspected and navigation file created allowing for easy frame-
addressing
 Audio ripped and transcoded in separate step
 No artifacts
 Less media-transfers than in previous versions
 Future version
 Pre-fetching / replication of media to remote sites
24

25. TEST RESULTS
25

26. Test topology
Torque Server
Traffic Shaping
GPFS node 26

27. Test results
 Input media
Vob file

MPEG-2 video encoding

AC3 audio encoding

Size: 1,64 GB

 Output media
Avi file

Xvid video encoding

MP3 audio encoding

Size: 700 MB

 Currently no HD video input modules!
 Not the most optimized video transcoders
 Focus on measuring benefits of distributing
27

28. Results – Grid overhead
 Grid Overhead
28

29. Results – Preprocess phase
 Preprocess
29

30. Results – Audio ripping phase
 Rip audio
30

31. Results – Merging phase
 Merging phase
31

32. Results – 1Gbit/s WAN
32

33. Results – Parameterised WAN interconnection
33

34. Video (up)scaling
Video scaling is converting video signals from one size or resolution to another: usually
quot;upscalingquot; or quot;upconvertingquot; a video signal from a low resolution (e.g. standard definition)
to one of higher resolution (e.g. high definition television).
00:00:00 00:51:53
00:00:00 00:51:53
720X576 984x752
34

35. Video (up)scaling results – 52Mbit/s WAN
35

36. Video (up)scaling results – 52Mbit/s WAN
36

37. SIMULATION RESULTS
37

38. Simulation results
 We introduced a WAN connection to a remote
computational resource provider
TORQUE
Node 1
Node 2
User
• 1 Gbit/s
• 100 Mbit/s
Node 3
• 52 Mbit/s
• 35 Mbit/s
Node 4
GPFS 38

39. Simulation results – total job turnaround time
39

40. Comparison with measured results
40

41. Comparison with measured results
41

42. Simulation results
 Simulations provide very accurate total job turnaround times
 Real-life transcoding behaves erroneously when
interconnecting GPFS with computational resource provider by
means of WAN link lower than 35Mbit/s
Control Traffic
Control Traffic
Click Router
Data
Data
Simulation results show what would happen to job turnaround

GPFS
time for lower WAN interconnections
42

43. Simulation results – low-speed WAN
interconnection
43

44. Simulation results – 10 chunks
44

45. CONCLUSIONS
45

46. Conclusions
 Grid technology is a viable technology for dealing with media
production / distribution tasks
 Inherent support for parallelism can seriously decrease the total
processing time
 Need for adaptation of media tasks
 Grid overhead is no issue
 Outsourcing task processing to remote resource providers
 Viable when interconnection is sufficient
 Technical limitations (e.g. GPFS time-outs)
 MediaGrid simulator can provide accurate performance
predictions
46

47. Questions ?
Feel free to e-mail: Bruno.Volckaert@intec.UGent.be