1.
Proprietary and Confidential
Exploration is the New Search
/ Lora Aroyo
/ Vrije Universiteit Amsterdam
/ IBM Netherlands (CAS)
/ Tagasauris Inc
/ http://lora-aroyo.org
/ @laroyo

2.
2
massive amount of online digital video content

3.
3
online video content is 64% of Internet traffic

4.
4
300h of video uploaded each min on YouTube

5.
5
in 2020 it would take a person more than 5 million
years to watch the videos shared online in a month

6.
6
there are videos out there
relevant to you …

7.
7
but …

8.
8
most online videos
NOT DESCRIBED
CAN’T BE FOUND
WON’T BE SEEN

9.
9
what can we do
to describe videos better?

10.
1
machine computation can do a lot

11.
11
recognise objects & concepts
Proprietary and Confidential

12.
unlock rich semantics
Proprietary and Confidential
12

13.
Proprietary and Confidential
[setting] WESTERN
[character] DOLORES
[story arc] FLASHBACK 7
[object] TRAIN
[action] SHOOTOUT
[mood] SURREAL
[audience] SHOCKED
[extras] 22
[role] BYSTANDER
13
discover creative contexts

14.
14
link across online videos
Proprietary and Confidential

15.
15
put all that descriptive information together
to know what’s in a video
on a second-by-second basis
6
so, we can …
Proprietary and Confidential

16.
16
add also behavioral information
to know how people watch & talk about videos
6
and, we can …
Proprietary and Confidential
Viewership
Channels profiles
Social Media Buzz
Audience Demographics
Audience Contexts
Platform profiles

17.
17
all this will make
video search better

18.
18
but we need more …

19.
19
SEARCH vs. EXPLORATION
/ helps only when you know what to search for
/ assumes you understand a topic
/ accuracy driven
/ click-through driven
/ helps when you don’t know what to search for /
/ helps you understand & deepen in a topic /
/ serendipity driven /
/ focused on engagement /

20.
20
what can we do
to support video exploration?

21.
2
human computation can do a lot

22.
2
CrowdTruth.org
bring more semantics of the real world

23.
2
diversity of opinions, sen)ments, contexts
collected in a decentralized way
aggregated views over collec1ons
introduces perspectives
http://crowdtruth.org

24.
Proprietary and Confidential
24
consolidate experts & end users perspec)ves
for con1nuous improving video descrip1ons

25.
25
humans & machines can go a long way

26.
Proprietary and Confidential
HUMAN-IN-THE-LOOP AI FOR VIDEO EXPLORATION
/ Machines help to break-down video into granular moments, i.e. shots & scenes
/ Machines generate multitude of paths within and across videos
/ Humans perform simple actions, e.g. watching, following and rating a path
/ Machines generalise from these actions using explicit semantics
/ Machines learn to evolve & improve exploration path
/ Orchestrate a continuous human and machine symbiosis
/ The ultimate aim is to reach a tipping point for video exploration,
e.g. web search, speech recognition

27.
2
to transform the online video experience

28.
2
Multimedia Explorations for
Smart Culture

29.
2
Interac1ve Explora)on & Discovery in Context
linking objects to events and en&&es
building automa1c storylines (narra1ves) on topics or objects
Access to integrated online mul)media collec1ons
using Linked Open Data to integrate metadata of various
heritage collec1ons
DIVE+
Aggregated views over the collec1on
collec1ng perspec&ves from crowds & niches
http://diveproject.beeldengeluid.nl/

30.
3
Erp, M. van; Oomen, J.; Segers, R.; Akker, C. van de; Aroyo, L.; Jacobs, G.; Legêne, S; Meij, L. van der;O ssenbruggen, J.R. van; Schreiber, G.
AutomaDc Heritage Metadata Enrichment with Historic Events Museums and the Web 2011 hKp://www.museumsandtheweb.com/mw2011/
papers/automaDc_heritage_metadata_enrichment_with_hi

31.
31
types

32.
32
filters

33.
3
explore entities
related
entities

34.
3
entity
exploration

35.
3
object
exploration

36.
3
people filter

37.
3
event filter

38.
3
narrative

39.
3

40.
4
Video Explorations with
Micro-moments

41.
MOMENTS REDEFINE CURRENT VIDEO SEARCH
MOBILE/SOCIAL
OPTIMIZED
RELEVANT SEARCH RESULTS
PREVIEW SPECIFIC MOMENTS
OR WATCH FULL VIDEOS
DISCOVER ACTIONABLE
MOMENTS WITHIN VIDEO

42.
MOMENTS: PERSONAL VIDEO CHANNEL
SEARCH FOR
ANYTHING, WITH
ANYTHING
HYPERMEDIA PLAYER
W/ LINKED MOMENTS &
FULL VIDEOS
AI ACTIVELY
LEARNS USER
PREFERENCES
DISCOVERS
MORE & MORE
CONTENT

43.
AI, DEEP LEARNING & NETWORK EFFECTS
Ensemble algorithmic processing
Video, audio & text
Extract & understand entities
Does what computers are good at
AI & DEEP LEARNING:
//
//
// Human assisted computing
Collective intelligence (incl. fans)
Waze effect
Does what humans are good at
NETWORK EFFECTS:
//
//
//
OBSERVE & LEARN
VERIFY & EXTEND
*U.S. Patent Application #13/863,751
Web-scale layer of structured, linked data.
////
DOMAIN-SPECIFIC DATA
MOMENTS

44.
4

45.
4

46.
4

47.
Proprietary and Confidential
HUMAN-IN-THE-LOOP AI FOR VIDEO EXPLORATION
/ Machines help to break-down video into granular moments, i.e. shots & scenes
/ Machines generate multitude of paths within and across videos
/ Humans perform simple actions, e.g. watching, following and rating a path
/ Machines generalise from these actions using explicit semantics
/ Machines learn to evolve & improve exploration path
/ Orchestrate a continuous human and machine symbiosis
/ The ultimate aim is to reach a tipping point for video exploration,
e.g. web search, speech recognition

48.
49
to explore strange new worlds ….
to boldly WATCH what no-one has WATCHED before

49.
Proprietary and Confidential
Exploration is the New Search
/ Lora Aroyo
/ Vrije Universiteit Amsterdam
/ IBM Netherlands
/ Tagasauris Inc
/ http://lora-aroyo.org
/ @laroyo