Thumb.co.il is the best web-based video format inspector tool ever created. This is an overview of the reasons for creating thumbcoil and a peek into some of the cool technology contained therein.

1. Thumb.co.il
How we got here...

2. Who am I?
Nom de Plume:
Email:
HLS:
Transmuxer:
Thumbcoil Lib:
Thumb.co.il UI:
Jon-Carlos Rivera
jrivera@brightcove.com
https://github.com/videojs/videojs-contrib-hls
https://github.com/videojs/mux.js
https://github.com/videojs/thumbcoil
https://github.com/videojs/thumb.co.il

3. What is Thumbcoil?
Thumbcoil is a web-based video inspector for:
- MPEG2-TS (also M2TS or just TS)
- fMP4 (fragmented MP4)
- FLV

4. "If you gaze long into a file format, the file format also gazes into you."
-- Nietzsche, probably

5. In the beginning, there was no Thumbcoil
Thumbcoil eventual grew out of our home-grown web-based video format
transmuxer…
Mux.js!
https://github.com/videojs/mux.js
(Did I mention it's open source?)

6. What is Transmuxing a Container?
Container
Synchronization Synchronization
Bitstream
Info
Audio
Bitstream
Audio
Bitstream
Bitstream
Info
Video
Bitstream
Video
Bitstream

7. What is Transmuxing a Container?
M2TS
PES PES
PMT Audio
Bitstream
Audio
Bitstream
PMT Video
Bitstream
Video
Bitstream
MP4
Sample Sample
TRAK Audio
Bitstream
Audio
Bitstream
TRAK Video
Bitstream
Video
Bitstream

8. Transmuxing
Re-muxingDe-muxing
MPEG2-TS
Metadata
Video
Audio
MP4

9. Key Browser
FLV
Transmuxer
video-js-swf
NetStream
Browser
FLV
Transmuxer
video-js-swf
NetStream
MSE
SourceBuffer
MP4
Transmuxer
Browser
Mux.js
M2TS Demux
video-js-swf
NetStream
MSE
SourceBuffer
fMP4 RemuxFLV Remux
2013 2014 2015
Transmuxing on the web…
Native (JS)
Flash
Uggh!

10. Life was great until...

11. Life was great until...
Pretty quickly we ran into issues with fMP4 & MSE

12. Life was great until...
Pretty quickly we ran into issues with fMP4 & MSE
Flash playback of FLV was very resilient

13. Life was great until...
Pretty quickly we ran into issues with fMP4 & MSE
Flash playback of FLV was very resilient
Browser MSE implementations with MP4 were NOT

14. Life was great until...
Pretty quickly we ran into issues with fMP4 & MSE
Flash playback of FLV was very resilient
Browser MSE implementations with MP4 were NOT
MSE was very strict about EVERYTHING

15. Life was great until...
Pretty quickly we ran into issues with fMP4 & MSE
Flash playback of FLV was very resilient
Browser MSE implementations with MP4 were NOT
MSE was very strict about EVERYTHING
(But it has improved somewhat)

16. End result?

17. End result?
Lots of perfectly valid HLS just didn't work!

18. End result?
Lots of perfectly valid HLS just didn't work!
...But worked in Flash

19. End result?
Lots of perfectly valid HLS just didn't work!
...But worked in Flash
...And Safari

20. End result?
Lots of perfectly valid HLS just didn't work!
...But worked in Flash
...And Safari
...And Quicktime

21. End result?
Lots of perfectly valid HLS just didn't work!
...But worked in Flash
...And Safari
...And Quicktime
...And VLC

22. The debugging nightmare
How do you inspect a video to do some sanity checking?
Run ffprobe!
- Seems useful at first
- The output is ofttimes just a pack of lies
- ...because ffprobe hates you

23. How does ffprobe hate?
Problem: Detecting M2TS segments that did not begin with an i-frame
Solution: ffprobe --show_frames ?
Now you have two problems!

24. Our struggle...
Solution: Use mux.js to inspect videos!

25. Our struggle...
Solution: Use mux.js to inspect videos!
It was a GREAT IDEA...

26. Our struggle...
Solution: Use mux.js to inspect videos!
It was a GREAT IDEA...
...so we didn't do that.
Instead, we kept struggling with ffprobe.

27. Solve all of the easy problems...
...and all that remains are the hard problems.
(ffprobe doesn't help you here!)
We needed to go deeper...
...inside the bitstreams.

28. I'm going to level with you
I was mostly just curious

29. Thumbcoil is born
● I (re)wrote a parser for H.264 bitstreams from scratch
● None of the bitstream parsing code is shared with mux.js
● Implemented using PARSER COMBINATORS...
...for science!
http://beta.thumb.co.il
(Did I mention it's open source?)

30. Parser Combinators
One goal was for NAL unit parsers to read like the H.264 spec:

31. Parser Combinators
One goal was for NAL unit parsers to read like the H.264 spec:

32. Parser Combinators
You build a parser by composing high-order functions
All of the combinators in Thumbcoil are designed as "codecs"
- They have a decode function that takes a bitstream and returns an object
representation of the data
- They have an encode function that can take the object representation
and return a bitstream
This is mostly a fun trick that has never been used for anything cool. (Yet!)

33. Let's Parse Audio Bitstreams!

34. Let's Parse Audio Bitstreams!
Worst.

35. Let's Parse Audio Bitstreams!
Worst.
Decision.

36. Let's Parse Audio Bitstreams!
Worst.
Decision.
Ever.

37. What is parsing AAC like?
Imagine trying to find a word in a dictionary where pages appear blank unless
you've completely and accurately read the previous page.
It's like that but worse… because the words are also huffman coded.
Using one of any 15 codebooks... per letter.
And each codeword can code for more than one letter...
...and there are no spaces.
...and most of the words are homonyms.

38. Demo Time
Old Interface: http://thumb.co.il
New Interface: http://beta.thumb.co.il

39. The future...
● Improved GOP view
○ More sorting and display options
● Hex-view
● ID3 Parsing
● Bring back MP4 and FLV parsing
● Support MPEG-DASH manifests
● <Your Idea Here> (Did I mention it's open source?)

40. Fin
Questions?