The opening keynote of VIGTA 2012 – First International Workshop on Visual Interfaces for Ground Truth Collection in Computer Vision Applications In conjunction with the Advanced Visual Interfaces International Working Conference in Capri Italy, May 21-25, 2012

1. Multimedia Data Collection
using Social Media Analysis
Prof. Benoit HUET
EURECOM, France
Benoit.Huet@eurecom.fr

2. Visual Data in the 90’s
 Huet & Hancock [WACV’96]
Digital Map Corresponding aerial images
Ground Truth taken at different aircraft altitudes
21/05/2012 B. HUET - VIGTA'12 Keynote -2

3. Large Scale in the 90’
 Huet & Hancock [IEEE PAMI’99]
 Cartographic Database
 22 original images
 Aerial scenes
 Main features: roads
 100-1000 lines per image
 Trademarks and logos Database [Flickner et al. ’95]
 Over 1000 original images
 Scanned data
 B&W, Various resolution
 10-5000 lines per image
21/05/2012 B. HUET - VIGTA'12 Keynote -3

4. The TRECVID years (2001- to date)
 2001: 11 hrs from BBC & OpenVideo Project
 2003 first collaborative ground truth annotation
 2005-2006: 170 hrs (Nov.’04 news in Arabic,
Chinese, and English)
 High-level feature extraction (10)
 2007-2009: 100hrs from the Netherlands Institute
for Sound and Vision (news magazine, science news, news
reports, documentaries, educational programming, and archival video)
 2010-2011: 600hrs of MPEG-4 Creative Commons
Videos
 High-level feature extraction (light=50 full=364)
21/05/2012 B. HUET - VIGTA'12 Keynote -4

5. The Trend:
 Datasets are going Large-Scale (Web-Scale)
...slowly...
Multimedia / Computer Vision researchers
are tackling and experimenting
with Large-Scale data
 Issue:
1 research objective <-> 1 data corpus
Annotation -> expensive and demanding process
21/05/2012 B. HUET - VIGTA'12 Keynote -5

6. Talk Outline
 The scene / motivation
 Social Events and Big Data
 Using social platforms for creating a corpus automatically
 Social Event Detection
 Using social media for detecting events
 Social Event Media Mining
 Enriching Event‟s Illustrations through Web Mining
 Conclusions
21/05/2012 B. HUET - VIGTA'12 Keynote -6

7. What’s a Social Event?
21/05/2012 B. HUET - VIGTA'12 Keynote -7

8. What’s a Social Event?
VIGTA
2012
Capri
Italy
21/05/2012 B. HUET - VIGTA'12 Keynote -8

9. Big Data!
21/05/2012 B. HUET - VIGTA'12 Keynote -9

10. Search For media
21/05/2012 B. HUET - VIGTA'12 Keynote - 10

11. Searching for an event
21/05/2012 B. HUET - VIGTA'12 Keynote - 11

12. Data Collection
and
Ground Truth

13. Machine Tags
 A way to integrate Media and Events
LastFM Flickr YouTube
21/05/2012 B. HUET - VIGTA'12 Keynote - 13

14. Media explicitly associated with the event
21/05/2012 B. HUET - VIGTA'12 Keynote - 14

15. REST API for query
21/05/2012 B. HUET - VIGTA'12 Keynote - 15

16. Conclusion
 The medias and events can be linked via
machine tag.
 The relations provided by machine tags can be
taken as ground truth.
 Thanks to the REST API, Events and Media
information can be retrieved effectively.
21/05/2012 B. HUET - VIGTA'12 Keynote - 16

17. Event Detection
by Temporal Analysis
X. Liu, R. Troncy and B. Huet

18. Event Detection - Related Work
 EventBurn.com
 Create summaries about given events (searching
Twitter, Facebook, and Flickr)
 Firan et al. (CIKM’10)
 Event categorization from social media data
 Gao et al. (WWW’11)
 Employing Twitter data to enrich event information
 Liu et al. (ICMR’11)
 Finding media illustrating events
21/05/2012 B. HUET - VIGTA'12 Keynote - 18

19. How to mine events from PhotoSet…
Events ??
21/05/2012 B. HUET - VIGTA'12 Keynote - 19

20. Observation
 Media are captured during events and shared
 Capture Time, Geo-localization
 User Tags (Annotations)
 Machine-Tag (lastfm:event=1337426)
21/05/2012 B. HUET - VIGTA'12 Keynote - 20

21. How fast media are uploaded?
21/05/2012 B. HUET - VIGTA'12 Keynote - 21

22. Experiment Data
 9 Attractive Venues WorldWide
Venue Name NbEvents NbPhotos NbUsers
Melkweg 352 6912 266
Koko 151 3546 155
HMV Forum 106 2650 130
111 Minna Gallery 24 1369 105
HMV Hammersmith Apollo 79 2124 96
Circolo degli Artisti 148 2571 86
Circolo Magnolia 79 2190 76
Ancienne Belgique 212 7831 56
Rotown 204 3623 49
 Event Ground Truth obtained from the official agendas
 available from individual venue websites.
21/05/2012 B. HUET - VIGTA'12 Keynote - 22

23. Detecting and Identifying Events
 Our solution consists of 3 steps:
 Location Monitoring: finding the bounding-box of venues.
 Temporal Analysis: detecting events by analyzing the
uploading behavior along time.
 Event Topic Identification: identifying detected events’ topics
through tag analysis.
14
12
10
8
6
4
2
0
10/05/01 10/05/06 10/05/11 10/05/16 10/05/21 10/05/26 10/05/31
Location Temporal Event Topic
Results
Monitoring Analysis Identification
21/05/2012 B. HUET - VIGTA'12 Keynote - 23

24. Event Detections
 Region Monitoring
21/05/2012 B. HUET - VIGTA'12 Keynote - 24

25. Venue Bounding Box Estimation
1 : INPUT : VenueName
2 : OUTPUT : BoundingBox
3 : PhotoSet []
4 : Center GetInfo(
VenueName)
5 : EventSet GetPastEvents(VenueName)
6 : foreach event in EventSet do
7: photos GetFlickrPhoto(event)
8: PhotoSet.append ( photos)
9 : end
10 : GeoSet GetGeoInfo( PhotoSet)
11 : Filter (GeoSet, Center, threshold 1km)
12 : RETURN MinRect(GeoSet)
21/05/2012 B. HUET - VIGTA'12 Keynote - 25

26. Venue Bounding Boxes (a selection)
Paradiso HMV Hammersmith Apollo
Megwelk KoKo
21/05/2012 B. HUET - VIGTA'12 Keynote - 26

27. Analyzing the number of Photos
L
o
c
a
t
i
o
n
Megwelk
D
a
t
REST
e
Query
21/05/2012 B. HUET - VIGTA'12 Keynote - 27

28. Our Media DataSet
 Flickr Photos
 Taken in May 2010
 In either one of the 9 selected locations:
Number of Photos
Name Overlap Total
Geo-tagged Venue Name tagged
Koko 372 2040 3 2409
Rotown 90 273 1 362
Melkweg 363 700 8 1055
HMV Forum 184 412 0 596
111 Minna Gallery 937 3 0 940
Ancienne Belgique 2206 288 2 2492
Circolo degli Artisti 70 553 1 622
Circolo Magnolia 95 236 0 331
Hammersmith Apollo 287 84 0 371
Total : 4604 4589 15 9178
 Photos rarely have both geo-tag and venue name tag!
21/05/2012 B. HUET - VIGTA'12 Keynote - 28

29. Analyzing the number of Photos
250
200
Events ??
150
100
50
0
10/05/01 10/05/06 10/05/11 10/05/16 10/05/21 10/05/26 10/05/31
Number of Photos taken in Melkweg (NL) in May 2010
21/05/2012 B. HUET - VIGTA'12 Keynote - 29

30. Analyzing the number of Photos Owners
14
12
Events ??
10
8
6
4
2
0
10/05/01 10/05/06 10/05/11 10/05/16 10/05/21 10/05/26 10/05/31
Number of Photo Owners in Melkweg in May 2010
21/05/2012 B. HUET - VIGTA'12 Keynote - 30

31. Event Detection Approach
 Based on media upload activity
 At a given time
 At a given location
 Events can be detected by:
et arg(ti T)
i
 Where
ti N photos * N owners
T : Threshold
 Venue/Event popularity
 Adaptive thresholding
21/05/2012 B. HUET - VIGTA'12 Keynote - 31

32. Event Topics Mining
 Keep the top N most frequent tags
 Result:
melkweg anouk amsterdam jemaine 2010 european flight flightoftheconchords
conchords fotc mckenzie clement tour bret evelyn
21/05/2012 B. HUET - VIGTA'12 Keynote - 32

33. Event Detection Example
Melkweg in May 2010
Number of photos * Number of photo owners
21/05/2012 B. HUET - VIGTA'12 Keynote - 33

34. Event Detection Example
111 Minna Gallery in May 2010
Number of photos * Number of photo owners
21/05/2012 B. HUET - VIGTA'12 Keynote - 34

35. Event Detection Results
 Detection results on different conditions
Source Threshold True Predict False Predict F1
mean 43 21 0.211
Image
median 64 51 0.279
mean 56 56 0.246
Owner
median 58 62 0.251
mean 34 18 0.172
Image*Owner
median 67 53 0.289
21/05/2012 B. HUET - VIGTA'12 Keynote - 35

36. Event Detection Results
 Event Detection Statistics
Our Method
Venues Ground Truth LastFM
Detect Matched Precision Recall
Melkweg 69 15 12 0.800 0.174 44
Koko 20 15 8 0.533 0.400 0
HMV Forum 14 12 9 0.750 0.643 14
111 Minna
Gallery 23 15 2 0.133 0.087 0
Ancienne
Belgique 38 15 9 0.600 0.237 28
Rotown 16 15 8 0.533 0.500 13
Circolo degli
Artisti 22 15 8 0.533 0.364 12
Circolo
Magnolia 25 3 1 0.333 0.040 11
Hammersmith
Apollo 15 15 10 0.667 0.667 14
In total 242 120 67 0.558 0.277 136
21/05/2012 B. HUET - VIGTA'12 Keynote - 36

37. Events Detection at Melkweg
Detection Results Ground Truth LastFM
Venue
Date Tags Date Title LastFM Title
Parkway Drive / Despised Icon /
parkwaydrive drive
melkweg 03/05/2010 03/05/2010 Winds Of Plague / The Warriors / 50 1336473 Parkway Drive
parkway
Lions
flight
Flight Of The Conchords - Flight of the
melkweg 02/05/2010 flightoftheconchords 02/05/2010 1439320
UITVERKOCHT Conchords
conchords
Flight Of The Conchords - Flight of the
melkweg 04/05/2010 flightoftheconchords 04/05/2010 1439407
UITVERKOCHT Conchords
Mayer
mayerhawtorne mayer
melkweg 05/05/2010 05/05/2010 Mayer Hawthorne & The County 1416229 Hawthorne &
hawthorne
The County
melkweg 11/05/2010 bonobo 11/05/2010 Bonobo - UITVERKOCHT 1398102 Bonobo
melkweg 14/05/2010 paulweller paul 14/05/2010 Paul Weller - UITVERKOCHT 1406677 Paul Weller
Broken Social
melkweg 18/05/2010 brokensocialscene 18/05/2010 Broken Social Scene - UITVERKOCHT 1334429
Scene
Mike Stern band with special guest
Richard
melkweg 19/05/2010 mikestern richardbona 19/05/2010
Bona featuring Dave Weckl & Bob
Malach
melkweg 25/05/2010 beattimemelkweg 24/05/2010 Beattime - The Kika Edition
melkweg 26/05/2010 beattime 24/05/2010 Beattime - The Kika Edition
Off Centre - day 3 - night met Kode 9 /
melkweg 28/05/2010 offcentre 28/05/2010
Falty DL / Gold Panda / Kelpe
melkweg 30/05/2010 joannanewsom 30/05/2010 Joanna Newsom 1425481 Joanna Newsom
21/05/2012 B. HUET - VIGTA'12 Keynote - 37

38. Collage For illustration
She & Him in Koko 07/05/2010
21/05/2012 B. HUET - VIGTA'12 Keynote - 38

39. Conclusions on Event Detection
 A novel approach for automatically detecting
social events is presented
 The key idea consists in temporally monitoring
media shared on social web sites at a specific
location (Geo Localized Photo)
 Automatic Efficient Social Event Detection and
Identification can be achieved
21/05/2012 B. HUET - VIGTA'12 Keynote - 39

40. Visual Event Modeling
X. Liu and B. Huet

41. Objective
 Automatically collect training data to build
event visual appearance models
 Model training requires both positive and
negative examples/samples
21/05/2012 B. HUET - VIGTA'12 Keynote - 41

42. Our proposed Automated FrameWork
Positive
Sample
Event
tag1 Pic1
tags tag2 Event Model
Top N Pic2 Top M Negative
tags Photos Sample
tag3 Pic3
tagN ………. PicM ……
Rank tags Rank Photos
by frequency by distance to tags
21/05/2012 B. HUET - VIGTA'12 Keynote - 42

43. Positive Samples Collection
 Machine Tag
 Abbreviation of events name
 For example “ACMMM12” is the tag to query photos from
“ACM Multimedia 2012”
21/05/2012 B. HUET - VIGTA'12 Keynote - 43

44. Negative Samples Collection
 Photos which do not originate from the event.
 Assumption: Photos taken near the location of
the event offer better discriminating power than
random photos.
 Collecting Approach
 Collect the data taken near the event„s location and time
 Extract tag from the collection, and rank them according
to appearance frequency.
 Keep the top tags as common tags and use them to rank
photos by similarity
21/05/2012 B. HUET - VIGTA'12 Keynote - 44

45. The DataSet
 10 LastFM concerts, 3 international conference
and 1 popular carnival
Positive Negative Testing
EventID
Samples Candidate Pos Neg
lastfm:804783 441 1063 466 64
lastfm:1830095 716 748 398 134
lastfm:1858887 408 745 431 266
lastfm:1499065 348 712 16 153
lastfm:1787326 446 913 0 313
lastfm:1351984 307 584 498 19
lastfm:1842684 602 1125 535 78
lastfm:2020655 538 745 750 6
lastfm:1301748 944 541 1157 80
lastfm:1370837 592 1025 592 115
SIGIR2010 100 557 178 23
ACMMM07 30 525 0 201
ACMMM10 118 64 15 44
NICECarnival2011 52 848 60 209
Total 5642 10195 5096 1705
21/05/2012 B. HUET - VIGTA'12 Keynote - 45

46. DataSet Examples
Positive Samples Negative Samples Test Positive Test Negative
21/05/2012 B. HUET - VIGTA'12 Keynote - 46

47. Event Model Training
 Feature:
 400D Bag of Words from SIFT features.
 Model:
 SVM implemented with libSVM
 RBF kernel
 Cross validation is used to
optimize the parameters
21/05/2012 B. HUET - VIGTA'12 Keynote - 47

48. The (Negative Samples) Model Parameters
 R: the location distance between photo taken
and event venue
 D: the time-span between photo taken and
event taken time
-An example on event: lastfm:804783
Conclusion:
Use loose parameters
for both time interval
and location distance
21/05/2012 B. HUET - VIGTA'12 Keynote - 48

50. Visual Event Modeling Results
Random Uniform
EventID Query Our Algorithm k-NN Pruning
Sample Negative
lastfm:804783 87.92 88.68 46.98 50.00 75.85
lastfm:1830095 74.81 78.38 80.26 96.62 84.96
lastfm:1858887 61.84 63.41 63.56 76.47 73.89
lastfm:1499065 9.47 90.53 89.94 92.90 89.35
lastfm:1787326 0.00 98.40 92.65 97.12 42.49
lastfm:1351984 96.32 96.32 55.32 86.65 93.81
lastfm:1842684 87.28 87.93 67.86 79.28 87.11
lastfm:2020655 99.21 91.80 71.69 75.00 94.58
lastfm:1301748 93.53 93.53 73.73 64.83 93.21
lastfm:1370837 83.73 85.15 73.83 60.25 80.62
SIGIR2010 0.00 60.19 42.28 16.41 22.38
ACMMM07 25.01 57.62 46.61 28.81 27.18
ACMMM10 85.83 91.04 87.56 86.57 89.05
NICECarnival2011 22.30 76.58 59.10 55.39 56.51
Average 69.41 83.31 68.64 70.07 73.42
21/05/2012 B. HUET - VIGTA'12 Keynote - 50

51. Conclusions
 Event-based approach for users to explore,
annotate and share media
 Improving user experience
 Outstanding challenges in interlinking and curating the
data
 Device and User Metadata provide interesting
and valuable clues
 Detecting Events from social media activity
 Visual Event Media Enrichment
21/05/2012 B. HUET - VIGTA'12 Keynote - 51

52. Conclusions and Future Work
 Combine multiple information sources
(Tweets, Social Graph, etc…) to detect and
media enrich events.
 Meta-Objective: Social Event analysis based on
connections between events, media and participants
 Can the approach be extended to private
events?...
 MediaEval: Social Event Detection Task
 www.mediaeval.org
21/05/2012 B. HUET - VIGTA'12 Keynote - 52

53. Questions?
 IEEE Multimedia Special Issue on
Large-Scale Multimedia Data Collection
(to appear in summer 2012)
 Thank you for your attention.
21/05/2012 B. HUET - VIGTA'12 Keynote - 53