semantic improved color imaging application

1. SemanticImproved Color Imaging
Applications:
Its All About Context
Under The Guidance Of :
Mr. Ritesh Kumar
Presented By:
AMAL CHACKO

2. AVast Challenge/Opportunity
[Jonathan Good]
2

3. AVast Challenge/Opportunity
“This suggests that there are,at the very
least, aquarter of amillion distinct
English words,excluding inflections,and
words from technical and regional
vocabulary[JonnatohantGocod]overed by the OED ...”
2
[Oxford English Dictionary]

4. AVast Challenge/Opportunity
[Jonathan Good]
“This suggests that there are,at the very
least, a quarter of a million distinct
English words,excluding inflections,and
words from technical and regional
vocabulary not covered by the OED ...”
2
[Oxford English Dictionary]
Novel methods and applications to link
digital image content with human language.

5. Presentation Overview
method
applications
statistical framework
−15
−5
−10
0
5
sunset layout
Method:
3

6. Presentation Overview
method
applications
statistical framework
−15
−5
−10
0
5
sunset layout
1.semantic image enhancement
input grass
Applications:
Method:
3

7. Presentation Overview
method
applications
statistical framework
−15
−5
−10
0
5
sunset layout
1.semantic image enhancement
input grass
Applications:
Method:
2.color naming
periwinkle blue
3

8. Statistical Framework
Link image characteristics with keywords.

9. Image Database
[Huiskes et al.,ACMMM,2010]5
• MIR Flickr database,1 Million annotated images.
• Selection based on Flickr’s“interestingness” score.
• 1 MegaPixel,assume sRGB.
gold,oregoncoast,fojrtstevens,astoria,outside,
lightroom,sigma,1020mm,nikon,d40,
diamondclassphotographer,grass,yellow,blue,sky,
clouds,singlecloud,color,saturated,happy,fieldMeredith_Farmer (cc)

10. Statistical Framework
1M images + keywords
6

11. Statistical Framework
gold gold
6
996’6883312

12. Statistical Framework
gold gold
stevewhis (cc) laura.bell (cc)
paige_eliz (cc) Arty Smokes (cc)
TW Collins (cc)
raketentim (cc)
golfnride (cc)Dunechaser (cc)
Tal Bright (cc)
7
10863752@N00 (cc)
4 6

13. Statistical Framework
TW Collins (cc)
raketentim (cc)
stevewhis (cc) laura.bell (cc)
Dunechaser (cc) golfnride (cc)
paige_eliz (cc) Arty Smokes (cc)
Tal Bright (cc) 10863752@N00 (cc)
gold gold
0% 3%
5% 70%
8% 30%
90% 10%
2% 9%
percentage of yellow pixels
8

14. Statistical Framework
9
sorted list: 0% 2% 3% 5% 8% 9% 10% 30% 70% 90%

15. Statistical Framework
9
0% 2% 3% 5% 8% 9% 10% 30% 70% 90%
1 5
sorted list:
rank index:
ranksum:
2 3 4 6 7 8 9 10
T = 4 + 7 + 9 + 10 = 30

16. Mann-Whitney-Wilcoxon ranksum test
[F. Wilcoxon,Individual comparisons by ranking
o2
T =
12
µT =
nw(nw + nw + 1) nwnw(nw + nw + 1)
2
cardinalities
of both sets
nw, nw
oT
9 methods,Biometrics Bulletin,1(6):80–83,1945]
T —µT 30 —22
z = = 4.69
⇡ 1.71
Statistical Framework
0% 2% 3% 5% 8% 9% 10% 30% 70% 90%
1 5
sorted list:
rank index:
ranksum:
2 3 4 6 7 8 9 10
T = 4 + 7 + 9 + 10 = 30

17. Mann-Whitney-Wilcoxon ranksum test
[F. Wilcoxon,Individual comparisons by ranking
methods,Biometrics Bulletin,1(6):80–83,1945]
o2
T =
12
µT =
nw(nw + nw + 1) nwnw(nw + nw + 1)
2
cardinalities
of both sets
nw, nw
oT 4.69
T —µT 30 —22
z = = ⇡ 1.71
Statistical Framework
0% 2% 3% 5% 8% 9% 10% 30% 70% 90%
1 2 3 4 5
sorted list:
rank index:
ranksum:
6 7 8 9 10
T = 4 + 7 + 9 + 10 = 30
significantly more yellow pixels in gold images.z > 0
9

18. •
• CIELAB histogram
15x15x15 bins.
values indicate
significance of a
keyword w.r.t. to a
characteristic.
Distribution
10
gold

19. Other Characteristics
Spatial lightness layout.
light
1
0
−1
−2
−3
−4
−5
−6
−7
12

20. Other Characteristics
Spatial chroma layout.
barn
8
6
4
2
0
−2
−4
−6
13

21. Other Characteristics
Spatial Gabor filter layout.
fireworks
5
0
−5
−10
14

22. Summary
15
• Link any characteristic to any keyword.
• Fast and highly scalable:
millions of images and thousands of keywords.
• Base for subsequent imaging applications with
semantic awareness.

23. Semantic Image
Enhancement
[Lindner et al.,ACM Multimedia2012,long paper]

24. Which image is better?
17

25. dark snow
Which image is better?
17

26. Which image is better?
sand
18
sunset

27. Which image is better?
sand sunset
No decision possible based on pixel values only.
18

28. Which image is better?
sand sunset
No decision possible based on pixel values only.
Auto-adjust contrast/colors.
18

29. Which image is better?
sand sunset
No decision possible based on pixel values only.
Manual editing.
18

30. Which image is better?
sand sunset
No decision possible based on pixel values only.
Automatic Enhancement with Semantics.
18

31. Today’s Solutions
19
• Modes:
Camera:“portrait”,“nature”,“firework”.
Printer:“draft”,“presentation”,“text”.

32. Today’s Solutions
19
• Modes:
Camera:“portrait”,“nature”,“firework”.
Printer:“draft”,“presentation”,“text”.
• Classification + enhancement:
skin,sky or other classes.
Park et al.06,Ciocca et al.07,Kaufman et al.12.

33. Today’s Solutions
19
• Modes:
Camera:“portrait”,“nature”,“firework”.
Printer:“draft”,“presentation”,“text”.
• Classification + enhancement:
skin,sky or other classes.
Park et al.06,Ciocca et al.07,Kaufman et al.12.
•Difficult to scale to large vocabularies.

34. Semantic Image Enhancement
Gray scale tone mapping snow
20

35. Semantic Image Enhancement
Gray scale tone mapping snow
blue
20
Color enhancement

36. Semantic Image Enhancement
Gray scale tone mapping snow
blue
Color enhancement
macro
Change depth-of-field
[Zhuo and Sim,2011]
20

37. Semantic Image Enhancement
Gray scale tone mapping snow
blue
Color enhancement
macro
Change depth-of-field
[Zhuo and Sim,2011]
20

38. Semantic Enhancement
semantic
processing
input
outputimage
component
semantic
component
blue
characteristics
21

39. Semantic Enhancement
semantic
processing
input
outputimage
component
semantic
component
Blue
characteristics
semantic
component
21

40. Semantic Component
0 50 200 250
6
5
4
3
2
1
0
100 150
pixel value
v
al
u
e
z
red
green
blue
significance values for rose
22

41. Semantic Component
0 50 200 250
6
5
4
3
2
1
0
100 150
pixel value
v
al
u
e
z
red
green
blue
0 200
0
50
100
150
200
250
100
input value
outputvalue
red
green
blue
identit
f 0 =
⇢
1/ (1 + Sz)
1 + S|z|
if z Ç 0
if z < 0
S global scale parameter
significance values for rose Tone mapping function f
22

42. Semantic Enhancement
semantic
processing
input
outputimage
component
blue
characteristics
image
component
0
0
50
250
200
150
100
outputvalue
red
greenbl
identity
100 200
input value
23

43. Image Component
rose
24

44. Image Component
rose weight map
⇥
24
! = go ⇤zw
.
col(p)
.⇤1
0
go Gaussian blurring kernel
(1%of image diagonal)
·
⇥ ⇤1
0
normalization operator

45. Semantic Enhancement
semantic
processing
input
outputimage
component
blue
characteristics
0
0
50
250
200
150
100
outputvalue
red
greenblue
identity
100 200
input value
25

46. Semantic Enhancement
blue
0
0
50
250
200
150
100
outputvalue
re
greenblue
identity
100 200
input value
Enhance relevant characteristics in relevant regions.
input characteristics
output
26
Iout = (1 —! ) · Iin
+ ! ·
Itmp
Itmp

47. Semantic Enhancement
Enhanced Image
Blue

48. sand

49. sand

50. snow

51. snow

52. strawberry

53. strawberry

54. macro

55. macro

56. Automatic Color Naming
[Lindner et al.,IS&T CIC 2012]
&
[Lindner et al.,IS&T CGIV 2012]

57. Introduction
Standard psychophysical color naming experiment:
green
observer
45

58. Introduction
Standard psychophysical color naming experiment:
green
observer
Our approach:
statistical
framework
green
45

59. 9000+ Color Names
46
• XKCD color survey,psychophysical experiment.

60. 9000+ Color Names
47
• XKCD color survey,psychophysical experiment.
• 950 English color names + color values.

61. 9000+ Color Names
48
• XKCD color survey,psychophysical experiment.
• 950 English color names + color values.
• Translate to 9 other languages:
Chinese,French,German,Italian,Japanese,Korean,
Portuguese,Russian,and Spanish.

62. 9000+ Color Names
49
• XKCD color survey,psychophysical experiment.
• 950 English color names + color values.
• Translate to 9 other languages:
Chinese,French,German,Italian,Japanese,Korean,
Portuguese,Russian,and Spanish.
• Example:柔和的粉红色,soft pink, rose tendre,sanftes
pink, rosa tenue,ソフトピンク,부드러운녹색,rosa
suave,нежно розовый,rosa suave.

63. DataAcquisition
Google Image:soft pink
50

64. DataAcquisition
Google Image:soft pink
51

65. DataAcquisition
Google Image:soft pink
• 100 images per color name.
• Language and country restrict.
• Assume sRGB encoding.
• Almost 1M images.
51

66. • CIELAB histogram
15x15x15 bins.
52
Distribution
soft pink, English

67. • CIELAB histogram
15x15x15 bins.
Distribution
soft pink, English
sRGB:238,197,203
52

68. Soft Pink
柔和的粉红色,cn
soft pink,en
rose tendre,fr
sanftes pink,de
rosatenue,it
ソフトピンク,jp
부드러운녹색,ko
rosasuave,pt
нежно розовый,ru
rosasuave,es
53

69. Soft Pink
柔和的粉红色,cn
soft pink,en
rose tendre,fr
sanftes pink,de
rosatenue,it
ソフトピンク,jp
부드러운녹색,ko
rosasuave,pt
rosasuave,es
54

70. Soft Pink
柔和的粉红色,cn
soft pink,en
rose tendre,fr
sanftes pink,de
rosatenue,it
ソフトピンク,jp
부드러운녹색,ko
rosasuave,pt
rosasuave,es
Language and country restrict.
54

71. Color Estimations
Chinese
English
French
German
Italian
Japanese
Korean
Portuguese
Russian
Spanish
55

72. Conclusions & FutureWork
sunset layout 5
0
−5
−10
−15
61
Easily scalable statistical framework.

73. Conclusions & FutureWork
sunset layout 5
0
−5
−10
−15
Easily scalable statistical framework.
input grass
61
Semantic image enhancement for
tone-mapping,color and depth-of-field.

74. Conclusions & FutureWork
sunset layout 5
0
−5
−10
−15
Easily scalable statistical framework.
input grass
Semantic image enhancement for
tone-mapping,color and depth-of-field.
periwinkle blue
Automatic color naming and an
interactive online color thesaurus.
61

75. •Du-Sik Park,Youngshin Kwak,Hyunwook Ok and Chang-Yeong Kim,Preferred skin color
reproduction on the display,JEI,2006.
•Gianluigi Ciocca,Claudio Cusano,Francesca Gasparini and Raimondo Schettini,ContentAware
Image Enhancement,Artificial Intelligence and Human-Oriented Computing,2007.
•Liad Kaufman.Dani Lischinski and MichaelWerman,Content-AwareAutomatic Photo
Enhancement,Computer Graphics Forum,2012.
•BaoyuanWang,YizhouYu,Tien-TsinWong,Chun Chen andYing-Qing Xu, Data-Driven Image
ColorTheme Enhancement,ACM SIGGRAPH,2010.
•NailaMurray,Sandra Skaff and Luca Marchesotti,Towards Automatic ConceptTransfer,
SIGGRAPH/Eurographics Symposium on Non-PhotorealisticAnimation and Rendering,2011.
•FrankWilcoxon,Individual Comparisons by Ranking Methods,Biometrics Bulletin,1945.
•Shaojie Zhuo andTerence Sim,Defocus map estimation from a single image,Pattern
Recognition,2011.
•Sung Ju Hwang,Ashish Kapoor and Sing Bing Kang,Context-BasedAutomatic Local Image
Enhancement,ECCV,2012.
Reference