1. © Ramesh Jain
Ramesh Jain
and
Pinaki Sinha
Department of Computer Science
University of California, Irvine
jain@ics.uci.edu
Content Without Context
is Meaningless

2. © Ramesh Jain
Brave New Ideas
 On a Nagging Old Problem – that won’t go
away.
 About the dead elephant in the room.
 Or about the Emperor’s new clothes.
“If you keep doing what you’ve always done,
you’ll keep getting what you’ve always got.”
This Presentation

3. © Ramesh Jain
Semantic Gap
Discussions
Semantic Gap
remains
strong.
Lets stop ignoring the Elephant in the room.
Our approaches have
completely failed for
the last two decades.

4. © Ramesh Jain
What is our Goal?
1. Using Intensity values to bridge the Semantic
Gap.
Or
2. Bridging the Semantic Gap in (multimedia)
data.
Or
3. Publishing another paper in ACM
Multimedia (or CVPR).

5. © Ramesh Jain
You will be disappointed in this
presentation if you are interested in
1. Using Intensity values to bridge the Semantic
Gap.
Or
3. Publishing another paper in ACM
Multimedia (or CVPR).

6. © Ramesh Jain
Reduction: Popular Research Approach 1
 X is an important problem, let me work on it.
 X is too complex. Simplify it to X’.
 X’ is too complex. Simplify it to X2
’.
 …
 X99
’ can be formally defined, rigorously explored,
and clearly solved. Solve it.
 Publish it in a conference with less than 16%
acceptance rate.
 Announce that X is a solved problem.

7. © Ramesh Jain
Data Discovery: Popular Research Approach 2
 Your algorithm fails on a real data set D.
 Select a more relevant data set D1. Fails again.
 Carefully select a subset of Flickr data. Still Fails.
 …
 Select a better dataset - Coral Photos, or create a
set and somehow make it Work.
 Run several experiments – have several graphs so
reviewers don’t complain.
 Publish it in a conf. with < 16% acceptance rate.

8. © Ramesh Jain
Back to Content and Context
Content
Meaning or message: The meaning or message
contained in data, as distinct from its appearance,
form, or style.
Context
Surrounding conditions: The circumstances or
events that form the environment within which
something exists or takes place.

9. © Ramesh Jain
Falling Tree and George Berkeley
 "If a tree falls in a forest and no one is
around to hear it, does it make a sound”
 "No. Sound is the sensation excited in the
ear when the air or other medium is set in
motion.“
 Observation, Reality, and Perception.

10. © Ramesh Jain
Data
Streams are
all around
us.

11. © Ramesh Jain
Data Streams are Omnipresent
Visual DATA
Audio DATA
Text DATA
Location DATA
EXIF DATA
Time Line
DataType

12. © Ramesh Jain

13. © Ramesh Jain

14. © Ramesh Jain
The Challenge
Connecting

15. © Ramesh Jain
Bits and Bytes
Alphanumeric Characters
Lists, Arrays, Documents, Images …
Transformations

16. © Ramesh Jain
Semantic Gap
The semantic gap is the lack of coincidence
between the information that one can extract
from the (visual) data and the interpretation
that the same data have for a user in a given
situation. A linguistic description is almost
always contextual, whereas an (image) may
live by itself.
Content-Based Image Retrieval at the End of the Early Years
Found in: IEEE Transactions on Pattern Analysis and Machine Intelligence
Arnold Smeulders , et. al., December 2000

17. © Ramesh Jain
M ultim edia
R esearcher
Hammer
 "To a man with a
hammer, everything
looks like a nail." —
Mark Twain
Where is the
Semantic Gap?
I got my Machine
Learning Hammer.

18. © Ramesh Jain
Learning from Birds
 The birds flying at the tips and at the front
are rotated in a timely cyclical fashion to
spread flight fatigue equally among the flock
members.
 The formation makes communication easier
and allows the birds to maintain visual
contact with each other.

19. © Ramesh Jain
Data is just that --- DATA, but
Some of us have introduced a caste system in
multimedia!!!
Video
Meta Data
Audio
Text
Multimedia Caste SystemTraditional Caste System
Priest
Warrior
Traders
Who Cares

20. © Ramesh Jain
Modeling the World
 Data
 Objects
 Events

21. © Ramesh Jain
Events and Objects
 Exist in the real world.
 Captured using different sensory
mechanism.
 Each sensor captures only a limited aspect.
 Are used to understand a Situation.

22. © Ramesh Jain
Events take place in the real world.
Events result in Data and Documents

23. © Ramesh Jain
Events are ‘Connectors’
Events create ‘Context’
SpatialCausal
Experiential Informational
TemporalStructural
People
Things
Places
Time
Experiences
Events
Events Connect:
Events are represented
using 6 facets.

24. © Ramesh Jain
We are in the midst of the greatest
Media Explosion.

25. © Ramesh Jain
Dealing with Photo Explosion
 3 Billion Photos uploaded on Facebook –
every month.
 We know that most people in the world are
not yet addicted to the Web, but do take
photos.

26. © Ramesh Jain
Context Starts Before the Photo is
Taken
 Where
 When
 Why
 Who (Photographer)
 Which device
 Parameters of the device

27. © Ramesh Jain
Types of Context
 Context in Content
 Device Parameters
 Data Acquisition Context
 Perceiver
 Interpretation Context

28. © Ramesh Jain
Context in Content
 Relationship among different objects and
even in their subparts in real world can be
utilized in analysis of data.
 This has been studied since early days in
computer vision and is being rediscovered
once again.

29. © Ramesh Jain
Device Parameters
 Environmental parameters of the digital
devices at the time of photo taking may play
key role.
 What is a camera?

30. © Ramesh Jain
Data Acquisition Context
 Knowledge about the person taking photos,
location, and environmental conditions at the
time of photo acquisition (e.g., sun angle,
cloudy, rainy, night, indoor, etc.) affect the
content of the image.

31. © Ramesh Jain
Where am I?

32. © Ramesh Jain
Perceiver
 The knowledge and personality of the
perceiver play a key role in interpretation of
data.
Knowledge
Personality:
Rohrsach
Tests

33. © Ramesh Jain
Interpretation Context
 Real world situation in which the data is
interpreted results in focus on different
aspects of the data.

34. © Ramesh Jain
What is a Camera?
 Kodak ‘Moment’.
 EXIF data is all metadata related to the Event.
Exposure Time
Aperture Diameter
Flash
Metering Mode
ISO Ratings
Focal Length
Time
Location
Face

35. © Ramesh Jain
Sony CyberShot DSC-T2 Touchscreen 8MP Digital
Camera with Smile Detection
Camera = Event Capture Device

36. © Ramesh Jain
Meta Data for photos = Event Data

37. © Ramesh Jain
A new problem statement for media analysis
 Given apriori knowledge about the world (concepts and
relations between concepts)
 Given a priori knowledge about events (lattice of events)
 Given as contextual information
 A set of data/ media
 A possibly empty sets of tags (natural language
descriptions)
 The current event type to be instantiated (possibly to be first
discovered)
 Identify the entities occurring in the event, as described by
media, data and text
 with >98% precision and recall
 with minimal/ acceptable user involvement
Fausto Giunchiglia, DISI, University of Trento

38. © Ramesh Jain
Annotation Sources
Calendars
Annotations
Environment
Conditions
Application
Knowledge
Geographic
Landmarks
Camera
Parameters

39. © Ramesh Jain
Annotation Process
Calendars
Annotations
Environment
Conditions
Application
Knowledge
Geographic
Landmarks
Camera
Parameters
Keynote Talk at ACM MM 2010,
Speaker: Duncan WattsLocation: Palazzo dei Congressi
EXIF: Indoor Scene; Face
Keynote Talk
at ACMMM10
by Duncan Watts
on
Using the Web to
do Social Science

40. © Ramesh Jain
Semantic Gap is a Tough Problem –
Lets Fight with it using all tools.
Fighting
with
a
Monster.

41. © Ramesh Jain
Thanks.
For more
information,
jain@ics.uci.edu
?
ACM MM
Paper
Rejection