http://2016.semantics.cc/shuangyong-song

1. 関係者外秘関係者外秘
Shuangyong Song
Fujitsu R&D Center Co., Ltd.
Linking Images to Semantic Knowledge
Base with User-generated Tags
Copyright 2013 Fujitsu R&D Center Co.,LTD

2. 関係者外秘関係者外秘
Outlines
 Motivation
 Method
 Experimental results
 Conclusion
1Copyright 2010 FUJITSU LIMITED

3. 関係者外秘関係者外秘
Outlines
 Motivation
 Method
 Experimental results
 Conclusion
2Copyright 2010 FUJITSU LIMITED

4. 関係者外秘関係者外秘
Motivation
 Images account for an important part of Multimedia Linked
Open Data. (LOD4ALL)
 1. Wide semantic annotation of images with image
understanding is difficult and fallible; 2. semantic annotation
with manual labelling is time-consuming.
 Therefore, automatic semantic annotation of images with using
other kind of crowdsourced ‘manual labelling’ is considered.
3Copyright 2010 FUJITSU LIMITED
Tags Semantic
KnowledgeImages
medium

5. 関係者外秘関係者外秘
Outlines
 Motivation
 Method
 Experimental results
 Conclusion
4Copyright 2010 FUJITSU LIMITED

6. 関係者外秘関係者外秘
Method
 Module 1. Linking tags to knowledge
 Module 2. Detecting semantic relations between tags
 Module 3. Extending knowledge linked tags
 Module 4. Linking images to knowledge
Copyright 2013 Fujitsu R&D Center Co.,LTD5
[M1] [M2] [M3] [M4]
Semantic
Knowledge
Semantic
Knowledge
Semantic
Knowledge
Semantic
Knowledge
Semantic
Knowledge
Images Tags Topics

7. 関係者外秘関係者外秘
Method – Offline Part
 Module 1. Linking tags to knowledge
 Tag ‘Somophyllin-DF’: search it in dbo or dbr, and detect the
unambiguous matched ontology as ‘dbr:Somophyllin-DF’,
 ‘Apple’, we may not be able to link it with ‘dbr:Apple_Inc.’ or
‘dbr:Apple_III’ or some other ontologies which related to
‘Apple’.
6Copyright 2010 FUJITSU LIMITED
1 1
( , )* ( , )
( , ) ( , , )
( )
a u u uU U
i k k ja u a u u
i j i k j u
k k k
C t t C t t
R t t R t t t
F t 
  

8. 関係者外秘関係者外秘
Method – Offline Part
 Module 2. Detecting semantic relations between tags
7Copyright 2010 FUJITSU LIMITED
T DNd
θ(d)
α
z
wϕ(z)
β
Image data
Tags
Topic (vector)
Tags  Topical vectors
For example, tags ‘programmer’ and ‘ilife’
are detected as related tags.
Image-tag matrix is sparse and difficult to be well analyzed,
topical information can help to discover the implicit semantic relations between tags.

9. 関係者外秘関係者外秘
Method – Offline Part
 Module 3. Extending knowledge linked tags
 The rules are as below: (very strict for keep precision)
 1) If t1 and t2 have ‘inclusion relation’, such as ‘motor’ and ‘motorcycle’, and cosine-
similarity value between their topical vectors is bigger than σ, we judge them as
similar tags;
 2) If ‘Levenshtein distance’ value between t1 and t2 is equal to or smaller than a
threshold β, which is a positive integer, and cosine-similarity value between their
topical vectors is bigger than σ, we judge them as similar tags;
 3) If t1 and t2 don’t have above relations, we judge them as dissimilar tags.
 If t1 and t2 are judged as similar tags, and just one of them has semantic
link to knowledge base, we link the other one to the same knowledge with a
probability, of which the value equals to C(t1,t2), which is the cosine-
similarity value between t1 and t2.
8Copyright 2010 FUJITSU LIMITED

10. 関係者外秘関係者外秘
Method – Online Part
 Module 4. Linking images to knowledge
9Copyright 2010 FUJITSU LIMITED
 Define the linking types <subject, predicate, object>
 We use 77 million triples collected from our semantic database ‘LOD4ALL’ as
criteria for predicate selection. We create links between images and objects with
unambiguous predicate. For example, we use predicate ‘dbo:locationCountry’ to
create links to objects such as ‘China’ or ‘Denmark’.
 We use ‘rdfs:seeAlso’ to be predicate for ambiguous triples.
 In particular, we also detect some tag-combined knowledge for expanding the
links’ range. For example, if an image has both ‘DigitalCamera’ and ‘Conon’ as its
tags, we will check if there is ‘DigitalCamera_Conon’ or ‘Conon_ DigitalCamera’
in dbo or dbr, and link this image to the detected knowledge.

11. 関係者外秘関係者外秘
Outlines
 Motivation
 Method
 Experimental results
 Conclusion
10Copyright 2010 FUJITSU LIMITED

12. 関係者外秘関係者外秘
Experimental results
 1. Experimental Datasets
 2. Methods for Comparison
 3. Evaluation Metrics & Test Data Preparation
 4. Evaluation Results
11Copyright 2010 FUJITSU LIMITED

13. 関係者外秘関係者外秘
Experimental results
 1. Experimental Datasets
 mirFlickr 1 , 690,649 images and 659,227 tags.
 all the 7,401 ontologies in dbo and 7,925,232 resources in dbr were used
as semantic KB.
 After removing 41,127 images without any tag and 392 attached with
‘stopword’ tags, we had 649,130 images left, and those images and their
tags were used to train the topic model with an empirical K value of 200.
12Copyright 2010 FUJITSU LIMITED
1
Huiskes, M. J., Lew, M. S. The MIR Flickr Retrieval Evaluation. In: Proceedings of the 1st ACM SIGMM
International Conference on Multimedia Information Retrieval, 2008, pp.39-43.

14. 関係者外秘関係者外秘
Experimental results
 1. Experimental Datasets
 2. Methods for Comparison (our model: LDA based)
13Copyright 2010 FUJITSU LIMITED
Word2vec based
model (W2V)
Word2vec is a recently popular method for getting distributed
representation of words, of which the output form is similar to
LDA. However, we suppose that W2V is unfit for the image-
knowledge linking task, since W2V does well in detecting context
similarity, such as the similarity between ‘Paris’ and ‘Beijing’.
We will evaluate it in the following part.
Co-occurrence based
model (Co-occur)
Co-occur is a very simple yet effective method for detecting
relationship between words, we also take it as a baseline on
the tag similarity calculation subtask instead of LDA method
in subsection 2.3, and evaluate it in in the following part.
Compared to our model, Co-occur is a weaker tool on detecting
latent semantic information of tags.

15. 関係者外秘関係者外秘
Experimental results
 1. Experimental Datasets
 2. Methods for Comparison
 3. Evaluation Metrics & Test Data Preparation
14Copyright 2010 FUJITSU LIMITED
To evaluate the effects of different d when β = 2, we manually label 200 images.
We roughly set the recall as 1.0 when d = 1 considering we are unable to collect
all possible related knowledge to an image. After getting both precision and
recall with different d, we also use F1-value as evaluation criterion to evaluate d.
precision ; recall ; F1-value.

16. 関係者外秘関係者外秘
Experimental results
 1. Experimental Datasets
 2. Methods for Comparison
 3. Evaluation Metrics & Test Data Preparation
 4. Evaluation Results
15Copyright 2010 FUJITSU LIMITED
car girl flower bird dog sport
W2V 0.217 0.531 0.233 0.389 0.325 0.442
Co-occur 0.683 0.724 0.631 0.557 0.626 0.702
Our model 0.881 0.767 0.854 0.871 0.873 0.801
Result comparison with F1-value

17. 関係者外秘関係者外秘
Experimental results - examples
Copyright 2013 Fujitsu R&D Center Co.,LTD16
schema:Brand
dbr:Citroen_XM
dbr:French
rdfs:seeAlso
schema:inLanguagedbr:Citroen
dbr:Car
schema:category
schema:location
dbr:Mallorca
dbr:Palma
schema:location
schema:location
dbr:Baleares
dbr:Sineu
schema:location
dbr:Geotagged
schema:indication

18. 関係者外秘関係者外秘
Outlines
 Motivation
 Method
 Experimental results
 Conclusion
17Copyright 2010 FUJITSU LIMITED

19. 関係者外秘関係者外秘
Conclusion
Summary:
In this paper, we address the problem of automatically linking
images to semantic KB by using images’ tagging information. Our
framework lifted the F1-value of images’ semantic link creation by
62.43%.
Future works:
However, this paper is only a preliminary work, and named entity
disambiguation and named entity normalization will be considered
in our next step work. Besides, predicate discriminate should be
performed by considering adjacent tags as context. In addition, we
will try to create ground-truth datasets and try some classification
models on this task.
18Copyright 2010 FUJITSU LIMITED

20. 関係者外秘関係者外秘
Thank you！
Copyright 2013 Fujitsu R&D Center Co.,LTD19

21. 関係者外秘関係者外秘 20 Copyright 2013 Fujitsu R&D Center Co.,LTD