A talk I was given in Delft University of Technology at Amsterdam, the Netherlands on June 23, 2010.

1. Towards Web-Scale
Content-Aware Image Search
Xian-Sheng Hua
Lead Researcher, Microsoft Research Asia
June 23 2010 - TUDelft

2. Xian-Sheng Hua, MSR Asia
About the Title
• Towards Web-Scale Content-Aware Image Search
̵ Towards …
̵ Web-Scale …
̵ Content-Aware …
̵ Image Search …

3. Xian-Sheng Hua, MSR Asia
Evolution of CBIR
Annotation Based
Commercial Search ~2006 
Engines
~2003 (1) Tagging Based
(2) Large-Scale CBIR/CBVR
Conventional CBIR ~2001
Direct-Text Based
1990’ Academia
Prototypes
1970~
1980’
Manual Labeling Based
3

4. Xian-Sheng Hua, MSR Asia
Three Schemes
Example-Based
Annotation-Based
Text-Based

5. Xian-Sheng Hua, MSR Asia
Limitation of Text-Based Search
Surroundings
Text
Query
Social Tags Association

6. Xian-Sheng Hua, MSR Asia
Limitation of Text-Based Search
• High noises
̵ Surrounding text: frequently not related to the visual content
̵ Social tags: also noisy, and a large portion don’t have tags
• Mostly only covers simple semantics
̵ Surrounding text: limited
̵ Social tags: People tend to only input simple and personal tags
̵ Query association: powerful but coverage is still limited (non-clicked
images will never be well indexed)
A complex example:
Finding Lady Gaga walking on red carpet and wearing in black

8. Xian-Sheng Hua, MSR Asia
Limitation of Annotation-Based Search
• Low accuracy
̵ The accuracy of automatic annotation is still far from satisfactory
̵ Difficult to be solved
• Limited coverage
̵ The coverage of the semantic concepts is still far from the rich content
that is contain in an image
̵ Difficult to extend
• High computation
̵ Learning costs high computation
̵ Recognition costs high if the number of concepts is large
Still has a long way to go

9. Xian-Sheng Hua, MSR Asia
Limitation of Large-Scale Example-Based Search
• You need an example first
̵ How to do it if I don’t have an initial example?
• Large-scale (semantic) similarity search is still difficult
̵ Though large-scale (near) duplicate detection is good
An example: TinEye

10. Xian-Sheng Hua, MSR Asia
Possible Way-Outs?
• Large-scale high-performance annotation?
̵ Model-based? Data-driven? Hybrid? …
• Large-scale manual labeling?
̵ ESP game? Label Me? Machinery Turk? …
• New query interface?
̵ Interactive search?
Or … To combine text, content and interface?

11. Xian-Sheng Hua, MSR Asia
Simple Attempts
• Exemplary attempts based on this idea
̵ Search by dominant color (Google/Bing)
̵ Show similar images (Bing)/Find similar images (Google)
̵ Show more sizes (Bing)

18. Xian-Sheng Hua, MSR Asia
Our Solutions: Two Recent Projects
• Image Search by Color Sketch
• Image Search by Concept Sketch

19. Search by Color Sketch
WWW 2010 Demo

20. Xian-Sheng Hua, MSR Asia
Seems It Is Not New?
• “Query by sketch” has been proposed long time ago
̵ But on small datasets and difficult to scale up
̵ And seldom work well actually
̵ Not use to use – “object sketch”

22. It is based on a SIGGRAPH 95’s paper:
C. Jacobs et al. Fast Multiresolution Image Querying. SIGGRAPH 1995
on small scale image set and difficult to scale-up (0.44 second for 10,000 images)

23. Xian-Sheng Hua, MSR Asia

25. Xian-Sheng Hua, MSR Asia
Our Approach
• Color sketch …
̵ Is not “object sketch”
̵ But only rough color distribution
̵ Supports large-scale data
̵ And uses text at the same time

28. Xian-Sheng Hua, MSR Asia
Our Approach
• Color sketch …
̵ Is not “object sketch”
̵ But only rough color distribution
̵ Supports large-scale data
̵ And uses text at the same time
• Advantages of “Color Sketch”
̵ More presentative than dominant color (and text only)
̵ More robust than “object sketch”
̵ Easy to use compared with “object sketch”
̵ Easy to scale up

29. Xian-Sheng Hua, MSR Asia
Example: Blue flower with green leaf

30. Xian-Sheng Hua, MSR Asia
Example: Brooke Hogan walking on red carpet

31. Xian-Sheng Hua, MSR Asia
Color Sketch Is More Powerful
Blue Flower with Green Leaf Brooke Hogan walking on red carpet

32. Xian-Sheng Hua, MSR Asia
Color Sketch Is More Powerful
32

33. Xian-Sheng Hua, MSR Asia
The Techniques
• Color Sketch: Image Representation
̵ Capturing both color and spatial information
̵ Low computation and storage costs
• Intention Map: Intention Representation
̵ Rough: Tolerance to the errors in the colors and positions of the strokes
̵ Sparse: Intention estimation from sparse strokes
• Similarity Evaluation
̵ Effective and fast

34. Xian-Sheng Hua, MSR Asia
Color Sketch: Image Representation
Image with grids Color Sketch
Computation Cost: less than 50ms/image
Storage Cost: 80 byte in average/image

35. Xian-Sheng Hua, MSR Asia
Intention Map:
From User’s Input to Intention Estimation
Target: user’s input
(1) Do not require users to input the color map exactly
(2) Robust to rough and sparse input

36. Xian-Sheng Hua, MSR Asia
>
Target Image 1 Image 2
2010/6/24

37. Xian-Sheng Hua, MSR Asia
Intention Analysis 2: Relation
>
Target Image 1 Image 2
2010/6/24

38. Xian-Sheng Hua, MSR Asia
Intention Analysis 3: Intention Propagation
<
Propagated Image 1 Image 2
Since the target color sketch is rough, the cells nearby the scribbled cells may have larger
chance to have the similar color. Hence, image 2 should be more similar as the blue color
appearing in the cells near the scribbled cells.
2010/6/24

39. Xian-Sheng Hua, MSR Asia
Intention Map: An Example
User’s input
Intention map for blue
In scribbled cells, the intention weight is the largest, nearby cells has relatively smaller
intention weight, but the cells scribbled in green and its nearby cells have negative
intention weight for blue color.

40. Xian-Sheng Hua, MSR Asia
Similarity Evaluation
• Image color map
• Intention map
• Similarity

41. Xian-Sheng Hua, MSR Asia
Experiment: Intention Justification
C
C+R
C+P
C+R+P
C: Consistency R: Relation P: Propagation
51 text query, each 3 color sketches

42. Xian-Sheng Hua, MSR Asia
Sensitivity to Grid Dimension

43. Xian-Sheng Hua, MSR Asia
Comparison

44. Xian-Sheng Hua, MSR Asia
Demo
• It has been productized in Bing

62. Gambia flag

79. Xian-Sheng Hua, MSR Asia
Summary – Why it Works
• What are difficult
̵ Semantics is difficult
̵ Intent prediction is difficult
• What are easier
̵ Use color to describe images’ content
̵ Use color to describe users’ intent
content color sketch intent
Color sketch is an intermediate representation to connect images’ content and users’ intent.

80. Xian-Sheng Hua, MSR Asia
Limitation of Color Sketch
• Only works well for those semantics that can
be described by color distribution
• Only works well when people are able to
transfer their desire into color distribution
A more complex example:
A flying butterfly on the top-left of a flower.

81. Search by Concept Sketch
SIGIR 2010

82. Xian-Sheng Hua, MSR Asia
Search By Concept Sketch
• A system for the image search intentions concerning:
̵ The presence of the semantic concepts (semantic constraints)
̵ the spatial layout of the concepts (spatial constraints)
butterfly
flower
A concept map query Image search results of our system

83. Xian-Sheng Hua, MSR Asia
Framework
A concept map query A candidate image
1
A: butterfly Create Candidate
B: flower Image Set
butterfly
A: X=0.5, Y=0.5
B: X=0.8, flower
Y=0.8
2 3
Instant Concept Concept Distribution
Modeling Estimation
Concept Models Estimated distributions
butterfly butterfly
4 flower
Intent flower
Interpretation
Desired distributions
5
Spatial distribution
butterfly comparison
flower
Relevance score

84. Xian-Sheng Hua, MSR Asia
Creating Candidate Image Set
• Goal: get a group of candidate images from Web
̵ Ensure the presences of the concepts
̵ Reduce computational cost
butterfly flower
flower butterfly
A: butterfly
Candidate Images:
B: flower
Images related to
butterfly butterfly & flower
flower Training data for
concept “butterfly”
Training data for
concept “flower”

85. Xian-Sheng Hua, MSR Asia
Instant Concept Modeling
• A possible solution – manual tag-to-region
Manual tagging is expensive

86. Xian-Sheng Hua, MSR Asia
Instant Concept Modeling
• Goal: To learn a simple (but works well on the
candidate image set) visual detector for each concept
butterfly
[Visual instances]
…
Training data for Instance-based
“butterfly” visual detector of
“butterfly”
Graph-based representative image selection

87. Xian-Sheng Hua, MSR Asia
Concept Distribution Estimation
• Goal: Automatic concept localization leveraging Web-
image resources
Web images
A: A candidate image
Instant Concept B: Distribution
A: butterfly
B: flower Modeling estimation
Estimated distributions
Advantages: Scalable to Web-scale concepts butterfly
by exploiting rich Web-image resources
flower

88. Xian-Sheng Hua, MSR Asia
Concept Distribution Estimation
• Goal: estimate the distribution of a concept in an image
[Visual instances]
…
A visual instance
A candidate image
…
…
Estimated distributions of representative butterflies
in the candidate image Estimated distribution

89. Xian-Sheng Hua, MSR Asia
Intention Interpretation
• Goal: represent user’s spatial intention as 2D distributions
• Principles
̵ A concept should appear near the specified position
̵ A concept should not appear at the positions of the other concepts
Desired distributions Desired distributions
butterfly butterfly
butterfly butterfly
flower
flower

90. Xian-Sheng Hua, MSR Asia
Spatial Distribution Comparison
• Goal: compute the similarity of the desired distributions and
the estimated distributions
Desired distributions Estimated distributions
butterfly …
butterfly
flower
…
flower
Distribution comparison
Fusion
Relevance score

91. Xian-Sheng Hua, MSR Asia
Demo & Experiments

92. Xian-Sheng Hua, MSR Asia
Search Results Comparison
Task: searching for the images with “snoopy appear at right”
snoopy
Text-based image search
“show similar image”
snoopy
Image search by concept map

93. Xian-Sheng Hua, MSR Asia
Search Results
Task: searching for the images with “snoopy appear at top” / “snoopy appear at left”

94. Xian-Sheng Hua, MSR Asia
Search Results Comparison
Task: searching for the images with “a jeep shown above a piece of grass”
jeep grass

95. Xian-Sheng Hua, MSR Asia
Search Results Comparison
Task: searching for the images with “a car parked in front of a house”
house car

96. Xian-Sheng Hua, MSR Asia
Search Results Comparison
Task: searching for the images with “a keyboard and a mouse being side by side”
keyboard mouse

97. Xian-Sheng Hua, MSR Asia
Search Results Comparison
Task: searching for the images with “sky/Colosseum/grass appear from top to bottom”
Colosseum grass

98. Xian-Sheng Hua, MSR Asia
Quantitative Evaluation

99. Xian-Sheng Hua, MSR Asia
Advanced Function:
Influence Scope Adjustment
• Allow users to explicitly specify the occupied region of a
concept
house
Default
house
lawn lawn
house
lawn
Search for the long narrow lawn at the bottom of the image

100. Xian-Sheng Hua, MSR Asia
Advanced Function:
Influence Scope Adjustment
Searching for small windmill
Searching for large windmill

101. Xian-Sheng Hua, MSR Asia
Advanced Function:
Influence Scope Adjustment

102. Xian-Sheng Hua, MSR Asia
Advanced Function:
Visual Modeling Adjustment
sky
house
lawn
Clear Search
Advanced Function
Visual Assistor
Previous Next
Allow users to indicate or narrow down
what a desired concept looks like (visual constraints)

103. Xian-Sheng Hua, MSR Asia
Advanced Function:
Visual Modeling Adjustment
Searching for
front-view jeeps
jeep
Searching for
side-view jeeps
Visual instances

104. Xian-Sheng Hua, MSR Asia
Advanced Function:
Visual Modeling Adjustment
Searching for
butterfly with
yellow flower
butterfly
flower
Searching for
butterfly with
red flower
Visual instances

105. Xian-Sheng Hua, MSR Asia
Conclusions
• The approaches of realizing web-scale content-
aware image search
̵ Combing text and content
̵ Providing interface to express search intent
̵ Indexing and searching efficiently
• Two exemplary approaches introduced
̵ Search by color sketch
̵ Search by concept sketch

106. Xian-Sheng Hua, MSR Asia
Thank You