The exponential growth of media sharing and demand on the Web comes with a need for effective methods to explore them. Hence, media hyperlinking, which consists in linking together videos based on their content, uncovering the relation between them, is becoming an important functionality for providing users with a way to navigate between video entities and satisfy their information needs. Thanks to such technology, multimedia search can often be replaced by recommendation. A particular usage of hyperlinking is to provide, through a second screen application, extra information or content about the video watched on a main screen (TV). In this talk, we will focus on media hyperlinking from the news: the task at hand consists in locating and identifying relevant media items, and display them on the second screen. The related material is selected based on underlying events that will be detected in the news: events are seen as structuring elements, defined in terms of date, location, intent and attendance. Two approaches for event-based mining of such additional and related information will be presented. Each of them satisfying a different user information need.

1. Event-based Summarization for
Media Hyperlinking
Mathilde Sahuguet, Xueliang Liu, Benoit Huet
EURECOM, Sophia Antipolis, France

2. Hyperlinking and Second Screen
 Nielsen 12/2012: 85 Percent Of Tablet And
Smartphone Owners Use Devices As “Second
Screen” Monthly, 40 Percent Do So Daily
09/10/2013
Benoit HUET - Event-based Summarization for Media Hyperlinking

3. Hyperlinking and Second Screen
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

4. Audiovisual Explosion
 EU alone hosts 500+ online video platforms
 42.7m hrs of footage in online archives of
broadcasters and producers (61% of archive
footage is online)
 UGC on the advance:
 YouTube receives 60 hrs of video/minute
 Internet video is now 40 percent of consumer
Internet traffic, and will reach 62 percent by the
end of 2015
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

5. Hyperlinking at the fragment level
 MediaMixer:
 “set up and sustain a community of video producers,
hosters and redistributors…”
 “who will be supported in the adoption of semantic
multimedia technology…”
 “to build an European market for media fragment repurposing and re-selling.”
http://community.mediamixer.eu
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

6. Hyperlinking and Second Screen
 LinkedTV: linking television to
Web content
 Second screen scenario for
enriching television content
and achieving interaction
between user and content
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

7. Enrichment framework
Input video
Media Annotation
Segment annotation
Annotated
set of media
items
Media
items
Web Media Search
Multimedia Mining
Set of relevant
media with cues
Personalization,
Display, etc
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

8. Topic Enrichment
 Need to enrich given topics:
 based on terms/topics appearing in video (passive or
active)
 based on user query (active)
 Select and organize media items
 Different approaches for searching for information
 Diverse information rendering
 What are people interested to see ?
 In this work, we propose and investigate,
Event-based socially aware enrichment
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

9. Event-based socially aware enrichment
Publish content
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

10. Event-based socially aware enrichment
Search term
Google
Trends
09/10/2013
Query content
Benoit HUET - Event-based Summarization for
Media Hyperlinking

11. Outline
 Related work
 Framework
 Enrichments for Passive Second Screen
 Query Formulation
 Static Event visualisation
 Enrichments for Active Second Screen
Trends mining
Focused search
Video selection
Evaluation and Results
 Conclusion
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

12. Related work
 Chieu and al. (ACM CIGIR 2004)
 Query based event extraction along a timeline
 Yan et al. (CIKM2012)
 Visualizing timelines: evolutionary summarization via iterative reinforcement
between text and image streams.
 Christiansen et al. (WIDM '12)
 Modeling topic trends on the social web using temporal signatures
 Chen et al (Knowledge and Data Engineering 2007)
 Hot Topic Extraction Based on Timeline Analysis and Multidimensional
Sentence Modeling
 Tan et al. (ACM MM’10)
 Topical summarization of web videos by visual-text time-dependent
alignment.
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

13. Our Contributions
 Event-based methodology
 4W paradigm (Who, What, Where, When)
 Mining events from user contributed data and
search trends
 People-based point of view (not editorial choice)
 Event discovery
 Event illustration
 Mining from a pool of potential media items
 Choice of illustrative media item(s)
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

14. Passive Second Screen Framework
 Aim:
Enrich & Illustrate the Event on the TV Screen
 Step1: parse the semantic from query string
 Step2: Extract events from Social News site
 Step3: Illustrating with tag cloud and image collage
Step3
Step1
query
Step2
time
location
events
topic
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

15. Event Querying
 Events are queried from Digg.com
 a news aggregator with an editorially driven front page
Two advantages:
1. Save storage and
computation
2. Leverage from
collective intelligence
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

16. Textual Summarization
 Source: Full textual search on Twitter;
 query words = event title
 filter by time = event occurrence date
 Representation: Tag Cloud
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

17. Visual Summarization
 Source: Full textual search on Google Images
 Query words = event title
 Sorted by the similarity of image and event title
 Representation: Visual Collage
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

18. Event Summary
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

19. Event Summary
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

20. Active Second Screen Framework
 Aim:
Propose Enrichments which Illustrate the Event
on the TV Screen
 Interaction with the viewer
 Multiple choices
 Dynamic rendering
TV / Main screen might need to be paused
 Leverage on user generated content and collective
intelligence
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

21. Active Second Screen Framework
 Goal = automatically create and illustrate a
timeline of events on a given topic
 Event: occurrence of abnormal activity on a
limited time segment, that captured a lot of
interest and triggered massive web search
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

22. Trends mining
 Bursts = peaks: sudden increase in the search
volume => unexpected event, massive search
behavior
 Long-term events: abnormal high interest
spread over several time units
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

23. Trends mining
 Granularity:
 Trend value: Week (imposed by Google trends)
 Trend keywords: Monthly
 Feature:
 Burst value= left derivative of the trends values
 Threshold to select bursts:
 Adaptive threshold
 mean + standard deviation
 Output:
 Set of week dates
 Objective:
 link trendy time-segment to some event in the real world
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

24. Focused search on time segments
 Goal: search for items relevant to the event on
the extracted dates
 Extracting keywords for a focused search:
 Input: initial search term + dates (time segment)
 Query to Google trends
 Output: list of associated rising search terms
 Focused media search:
 Private / Public Media repositories
 Social Media platform
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

25. Video Selection
 Goal: represent each event with a media item
 Features:
 Using text features (from media title + description)
 Discard non-English content
 TF-IDF with cosine distance
 Media Item Selection
 Item with average similarity to all other media items
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

26. Example query: Snowden
1. Get trends
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 26

27. Example query: Snowden
23rd to 29th June 2013
9th to 15th June 2013
2. Mine time segments
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 27

28. Example query: Snowden
23rd to 29th June 2013
9th to 15th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
3. Extract associated keywords
(month granularity)
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 28

29. Example query: Snowden
23rd to 29th June 2013
9th to 15th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
3. Focused media search:
Tag cloud of associated text
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 29

30. Example query: Snowden
23rd to 29th June 2013
9th to 15th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
4. Illustrative video selection
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 30

31. Example query: Snowden
Snowden identity was
revealed on the 9th
Snowden flied to Russia on
the 23rd
23rd to 29th June 2013
9th to 15th June 2013
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
ed snowden
edward
edward snowden
edward snowden girlfriend
edward snowden news
edward snowden nsa
5. Ground truth
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 31

32. Experiments
 Focus on people: automatic event-based
timeline generation of 5 celebrities
 “Ground truth” extracted from expert
biographies and Wikipedia (from 2004)
Beyoncé
Knowles
28 events
10/21/2013
Mark
Zuckerberg
27 events
Oscar
Pistorius
9 events
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
Batman
9 events
- p 32
Edward
Snowden
43 events

33. Event extraction Evaluation
 Performance of our approach to detect
important events

TP: true positive, TP: false positive, FN: false negative, DE: discovered event
query
# bursts
TP
FP
FN
DE
Beyoncé K.
9
7
1
21
1
M.Zuckerberg
6
2
4
25
0
O.Pistorius
1
1
0
8
0
Batman
6
2
2
7
2
E. Snowden
2
2
0
41
0
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 33

34. Event discovery: Beyoncé
 Burst week: 22nd to 28th July, 2013
 Nothing in the ground truth
 Associated keywords:
 Beyonce fall video
 Beyonce falling
 Beyonce falls
 Beyonce orlando
 Beyonce stairs
 Caida beyonce
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 34

35. Video representative evaluation
 Challenge: choice of an illustrative video
 Event Relatedness ER (mean over all events):
 evaluation of the content of the video (not the quality)
Query
# events
Event
Relatedness
Beyoncé K.
8
0,81
M. Zuckerberg
2
1
O. Pistorius
1
0
Batman
4
1
E. Snowden
2
1
 Based on user generated textual features
10/21/2013
Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13
- p 35

36. Example query: Pistorius
 Week: 10th to 16th February, 2013
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

37. Example query: Pistorius
 Week: 10th to 16th February, 2013
 Associated keywords:
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking

38. Example query: Pistorius
 Week: 10th to 16th February, 2013
 Associated keywords:
 DataSet tag cloud:
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp

39. Example query: Pistorius
 Week: 10th to 16th February, 2013
 Associated keywords:
 Dataset tag cloud:
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp
 Illustrative video:
 Nothing about Pistorius himself or the murder case
 One of his opponents
09/10/2013
Benoit HUET - Event-based Summarization for Media Hyperlinking

40. Example query: Pistorius
 Week: 10th to 16th February, 2013
 Associated keywords:
 Dataset tag cloud:
oscar pistorius girlfriend
oscar pistorius news
pistorius fidanzata
pistorius girlfriend
pistorius news
reeva
reeva pistorius
reeva steenkamp
 Illustrative video:
 Why: associated text
BBC News -Paralympic Star Oscar Pistorius 'shoots girlfriend' 2013 Inscreva-se em
nosso canal. Subscribe yourself to our channel. Suscríbase a nuestro canal, News for
Oscar Pistorius girlfriend shot dead Oscar Pistorius charged with murder after
girlfriend shot dead The Guardian ?- 17 hours ago South African athlete will appear in
court on Friday after Reeva Steenkamp was found dead at his home in Pretoria.
Pistorius' girlfriend killed on Valentine's Day she was looking forward to CNN? - 15
hours ago South African sprinter Oscar Pistorius charged with murder after girlfriend,
Reeva Steenkamp, shot dead in his home New York Daily News? - 6 hours ago
Pistorius 'shot, killed' girlfriend - The Courier-Mail www.couriermail.com.au/.../
09/10/2013
Benoit HUET - Event-based Summarization for Media Hyperlinking

41. Conclusion
 Socially Aware Hyperlinking Approaches
 Automated Event-based Illustration of topics
 Creation of an Event-based media-illustrated
timeline related to a topic
 “new” Events Discovery = non-official but
triggered massive attention/searches
 Future work:
 Work with visual features, ASR.
 Adding facial information
 Work on long-term events
09/10/2013
Benoit HUET - Event-based Summarization for Media Hyperlinking

42. References
 http://www.linkedtv.eu
 http://www.mediamixer.eu
 M. Sahuguet and B. Huet, “Mining the Web for Multimedia-based
Enriching”, Multimedia Modeling 2014, Dublin, Ireland
 X. Liu and B. Huet “Event Representation and Visualization from
Social Media”, Pacific Rim Conference on Multimedia 2013,
Dec 13-16, Nanjing, China
 M. Sahuguet and B. Huet, “Socially motivated multimedia topic
timeline summarization”, 2nd ACM International Workshop on
Socially-Aware Multimedia, In conjunction with ACM Multimedia
2013, 21 October 2013, Barcelona, Spain
 X. Liu and B. Huet “On the automatic online collection of training
data for visual event modeling” Multimedia Tools and
Applications, February 2013, ISSN: 1380-7501
09/10/2013
Benoit HUET - Event-based Summarization for Media Hyperlinking

43. 09/10/2013
Benoit HUET - Event-based Summarization for Media Hyperlinking
43

44. Thank you for your attention
 Questions?
Benoit Huet
EURECOM
tel: +33 (0)493008179
Campus SophiaTech, 450 route des Chappes
fax: +33 (0)493008200
06410 Biot Sophia Antipolis
email: Benoit.Huet@eurecom.fr
France
http://www.eurecom.fr/en/people/huet-benoit
09/10/2013
Benoit HUET - Event-based Summarization for
Media Hyperlinking