1. Television Linked To The Web
www.linkedtv.eu
Building the future of television
Lynda Hardman
Centrum Wiskunde & Informatica

2. 2
www.linkedtv.eu
 Web and TV exist in parallel
 86% of TV viewers surf the Web at the same time
(Yahoo!, USA)
 88% of USA consumers use mobile as second
screen (Business Insider, May 2013)
 Manual efforts to link TV program to additional
content (Shazam, IntoNow)
 LinkedTV: interweaving Web and TV into a
single experience
 Mixing lean back and lean forward
 Want to maintain immersion while removing fourth
wall illusion
 Works only for certain genres and audiences
 Does second screen help or hinder?
The LinkedTV vision
linkedtv.eu

3. 3
www.linkedtv.eu
Scenarios and content
Enriched News
 Professional news content
produced by RBB for general
audience
 local news show rbb Aktuell
 People, locations and events
can be linked to previous
events.
Antiques Edutainment
 Program watched
predominantly by 50+ers
 AVRO:
Tussen Kunst en Kitsch
 Presenters, locations and
objects provide hooks to
related materials.
linkedtv.eu

4. 4
www.linkedtv.eu
Enriched link demonstration
linkedtv.eu

5. 5
www.linkedtv.eu
dbpedia.org/resource/Jan_Sluyters
How will it work?
linkedtv.eu
Paintings by Jan Sluijters
“...schilderij van Jan Sluijters....”
Video object and
word detection
Presentation
engine
Selection of
related concepts
Selection of
related content
Connection to
concepts

6. 6
www.linkedtv.eu
 Object detection in video
 Face detection
 Text analysis, keyword extraction
 Multilingual automatic speech recognition
 Combining results from different modality
analyses
 Speaker identification
 Object re-detection
 Output in XML using OpenSource tool:
EXMARaLDA
http://www.exmaralda.org/
Concept Detection
linkedtv.eu
“...schilderij van Jan Sluijters....”
Video object and
word detection

7. 7
www.linkedtv.eu
 Annotations represented in RDF
stored in LinkedTV platform
 Use existing vocabularies
e.g. schema.org, NERD, LSCOM, DBpedia
 Named Entity Recognition using
statistical & knowledge-based approaches
 Enrichment based on textual and visual
analyses
Concept connections
linkedtv.eu
Connection to
concepts
dbpedia.org/resource/Jan_Sluyters

8. 8
www.linkedtv.eu
LinkedTV metadata conversion
WP2 - Linking hypervideos to Web content
http://linkedtv.eurecom.fr/metadata/

9. 9
www.linkedtv.eu
 Additional information, e.g.
 biography of artist
 style of painting
 Related information, e.g.
 artists from same period
 paintings in similar style
 related styles
Concept selection
linkedtv.eu
Paintings by Jan Sluijters
Selection of
related concepts
Jan Sluijters
has art style
luminism
Leo Gestel
Piet Mondriaan
has art style
has art style

10. 10
www.linkedtv.eu
 Selection based on context of scene,
e.g. system knows which topics appear
next
 User preferences, e.g.
closely related information to
uninteresting topic is less useful
 We need to understand user
information needs (per genre), e.g.
don’t miss important news because of
personalisation
 Broadcasters don’t want users to drift
away from program
Content selection
linkedtv.eu
Selection of
related content

11. 11
www.linkedtv.eu
Example user information needs for news
LinkedTV Berlin, 2013
 MORE IN DEPTH SPECIFIC INFORMATION
I was wondering just how expensive it is to make all those
vaccines, you know? I was just wondering if it is a good idea to
even make a lot of it or how hard it is to make.
 LOCAL EFFECT OF GLOBAL NEWS – What is the impact
in my life?
Yes then I would look it up, what is the effect in the Netherlands…
(of the bird flu virus)
 OVERVIEW OF PAST RELATED EVENTS
My search would be something like "South Korea state of war
ramp up” (…)
I would like to know, what happened two years ago and how
they solved it…

12. 12
www.linkedtv.eu
User information needs for news
LinkedTV Berlin, 2013
 Information about how the story ends
 Updates when important changes occur
 More information about the people involved
 Where is that location?
 Consult related publications and documentaries
 Opinions from different experts
 Access related info selected by experts and suggested by broadcasters

13. 13
www.linkedtv.eu
 Many users watch TV news during dinner!
 Use second screen to
 give user more control
 retain sense of immersion via TV screen
 Need to resolve issues of pausing when in
group situation
 There is no single best solution
Content display
linkedtv.eu
Presentation
engine

14. 14
www.linkedtv.eu
How will it work?
linkedtv.eu
Paintings by Jan Sluijters
“...schilderij van Jan Sluijters....”
Video object and
word detection
Presentation
engine
Selection of
related concepts
Selection of
related content
Connection to
concepts
dbpedia.org/resource/Jan_Sluyters

15. 15
www.linkedtv.eu
How can LinkedTV help you?
linkedtv.eu

16. 16
www.linkedtv.eu
LinkedTV platform & player
TV Set
LinkedTV
Server
• Video Analysis
• Enrichment
• Personalization
• Delivery
Browser
CE-HTML
Platform ClientExternal
Broadcast
HTTP
Browser
Variant
HbbTV
Variant
HTTP
Repository
• Annotations
• Links to
external
Resources
Editor
LoD
Cloud
Videos
from
Scenarios
Browser
HTML 5
PC / Tablet
Network
Material
selection,
Annotation
adjustment
A clearly defined and modularized
platform architecture, deployment
decisions made, initial REST API
An interactive video player based on
Web specifications (HTML5) realising
intuitive interaction & enrichment layers
linkedtv.eu

17. 17
www.linkedtv.eu
linkedtv.eu

18. 18
www.linkedtv.eu
 Individual video analysis, annotation, hyperlinking, UI/UX design and
personalisation components integrated into a LinkedTV platform with
end-to-end content workflow
 Provide player for desktop, TV and second screen
 Create interactive scenarios (rbbAktuell and Tussen Kunst en Kitsch) to
engage with users to further understand their interaction and information
needs
LinkedTV in 2013
linkedtv.eu

19. 19
www.linkedtv.eu
linkedtv.eu

20. 20
www.linkedtv.eu
linkedtv.eu

21. 21
www.linkedtv.eu
1. Shot and scene segmentation used for generating Media Fragments
 W3C Recommendation for temporal and spatial fragments
2. Re-use as much as possible common vocabularies in the semantic web
 schema.org, Open Annotations, PROV-O, Ontology for Media
Resources
 FOAF, Dublin Core, NERD, LSCOM, DBpedia Ontology
3. Name Entity Recognition
 Statistical based approaches
 Knowledge Based Approaches (Wikipedia/DBpedia)
 Web-based APIs
4. Enrichment based on textual and visual analysis
 Structured data: LOD cloud accessible through structured queries
(SPARQL)
 Search API: REST based query
 Online content repositories (curated list):
 Crawling, Wrapping, Indexing and Searching (Lucene/Solr)
 Web-based and content-based mining approaches
Linking hypervideos to Web content
WP2 - Linking hypervideos to Web content

22. 22
www.linkedtv.eu
linkedtv.eu
Jan Sluijters
Jan Sluijters

23. 23
www.linkedtv.eu
 Specifying how to refer to video fragments
 Specifying the semantic annotation of media
resources in LinkedTV
 Defining the use of Linked Data vocabularies in
the identification of concepts in the annotations
 Implementing Named Entity Recognition (NER)
over WP1 analysis results
 Defining a Web mining approach to generate
links from video fragments to related Web
content via shared conceptual annotations
Linking hypervideo to Web content
linkedtv.eu

24. 24
www.linkedtv.eu
linkedtv.eu
has art style
Mondriaan
Leo Gestel
luminism
has art style
luminism
Jan Sluijters
has art style
luminism
Leo Gestel
Piet Mondriaan
has art style
has art style