2. 自己紹介
2014.3 博士(情報理工学)、東京大学
2014.4~2016.3 NTT CS研 研究員
2016.4~2018.9 東京大学 講師 (原田・牛久研究室)
2016.9~ 産業技術総合研究所 協力研究員
2016.12~2018.9 国立国語研究所 共同研究員
2018.10~ オムロンサイニックエックス株式会社
Principal Investigator
[Ushiku+, ACMMM 2012]
[Ushiku+, ICCV 2015]
画像キャプション生成 主観的な感性表現を持つ
画像キャプション生成
動画の特定区間と
キャプションの相互検索
[Yamaguchi+, ICCV 2017]
A guy is skiing with no shirt on
and yellow snow pants.
A zebra standing in a field with
a tree in the dirty background.
[Shin+, BMVC 2016]
A yellow train on the tracks near
a train station.
9. ユーザー生成コンテンツの爆発的増加
特にコンテンツ投稿・共有サービスでは…
• Facebookに画像が2500億枚 (2013年9月時点)
• YouTubeにアップロードされる動画
1分間で計400時間分 (2015年7月時点)
Pōhutukawa blooms this
time of the year in New
Zealand. As the flowers
fall, the ground
underneath the trees look
spectacular.
画像/動画と
関連する文章の対
→大量に収集可能
これらの背景から…
つぎのような様々な取り組みが!
10. 画像キャプション生成
Group of people sitting
at a table with a dinner.
Tourists are standing on
the middle of a flat desert.
[Ushiku+, ICCV 2015]
11. 動画キャプション生成
A man is holding a box of doughnuts.
Then he and a woman are standing next each other.
Then she is holding a plate of food.
[Shin+, ICIP 2016]
22. どれがどれくらい良いキャプションなのか?
CoSMoS [Ushiku et al., ICCV 2015]
Group of people sitting at a table with a dinner.
Corpus-Guided [Yang et al., EMNLP 2011]
Three people are showing the bottle on the street
Midge [Mitchel et al., EACL 2012]
people with a bottle at the table
アンケートによる比較:相対的な良さの評価
• 毎回ほかの手法と比較してもらわなければならない
• 絶対的なキャプションの良さの評価がほしい
23. 定量評価指標
機械翻訳では…
• テスト文に複数の参照訳が付随(通常5文)
• これらの参照訳と近い訳文が「良い」
• 既存の評価指標(BLEUやMETEOR、ROUGEなど)
• キャプション生成の評価指標(CIDErやSPICEなど)
One jet lands at an airport while another takes off next to it.
Two airplanes parked in an airport.
Two jets taxi past each other.
Two parked jet airplanes facing opposite directions.
two passenger planes on a grassy plain
キャプション生成の評価でも同様の流れ
PASCAL Sentenceの画像と参照キャプションの例
35. 問題の発展:キャプション列生成
アルバムのような系列画像に対して
The family
got
together for
a cookout.
They had a
lot of
delicious
food.
The dog
was happy
to be there.
They had a
great time
on the
beach.
They even
had a swim
in the water.
[Park+Kim, NIPS 2015][Huang+, NAACL 2016]
36. 問題の発展:キャプション列生成
A man is holding a box of doughnuts.
Then he and a woman are standing next each other.
Then she is holding a plate of food.
[Shin+, ICIP 2016]
シーンの切り替わる動画に対して
37. 問題の発展:キャプション列生成
A boat is floating on the water near a mountain.
And a man riding a wave on top of a surfboard.
Then he on the surfboard in the water.
[Shin+, ICIP 2016]
42. VQA: Visual Question Answering
• ビジュアル質問応答を分野として確立
– ベンチマークデータセットの提供
– ベースとなるパイプラインでの実験
• ポータルサイトも運営
– http://www.visualqa.org/
– 国際コンペティションも開催
[Antol+, ICCV 2015]
What color are her eyes?
What is the mustache made of?
48. VQA Challenge
コンペティション参加チームの解答例から
Q: Why is there snow on one
side of the stream and clear
grass on the other?
GT A: shade
Machine A: yes
Q: Is the hydrant painted a new
color?
GT A: yes
Machine A: no
54. Visual Question Generation
• 視覚的質問生成
(VQG)の提案
質問生成は画像
キャプションの
生成か検索を検討
• Qに対する要求
– その質問から会話が始まるような質問
– 画像を見てわかるような質問ではだめ
✗ How many horses
are in the field?
✓ Who won the race?
[Mostafazadeh+, ACL 2016]
55. 未知物体についてのVQG
画像認識器が知らない物体: 人から教わりたい
• 質問なら何でもいいわけじゃない
• 「なにこれ?」のような曖昧な質問だと…
回答も「物体」のように曖昧になりそう
• 学習して自動生成できた質問の例
What is the
woman
holding in
her right
hand?
What type
of shirt is
the man
wearing?
What in
on the
man’s
lap?
?
[Uehara+, ECCV 2018]
62. GuessWhat?!
連続するYes/No型のVQAデータ
Is it a person? No
Is it an item being worn or held? Yes
Is it a snowboard? Yes
Is it the red one? No
Is it the one being held by the Yes
person in blue?
Is it a cow? Yes
Is it the big cow in the middle? No
Is the cow on the left? No
On the right? Yes
First cow near us? Yes
[de Vries+, CVPR 2017]
63. GuessWhat?! の概要
• Questioner
– Guesserが何を見ているのかを知るために質問
• Guesser
– 自分が見ているものに応じてYes/Noで応答
• MS COCOを利用
– 画像数 64,000
– 対話 135,000
– 質問 673,000 Is it a vase? Yes
Is it partially visible? No
Is it in the left corner? No
Is it the turquoise and Yes
purple one?
[de Vries+, CVPR 2017]
65. Visual Dialog (VisDial)
Questioner Answerer
A couple of people
in the snow on skis.
What are their genders?
Are they both adults?
Do they wear goggles?
Do they have hats on?
Are there any other people?
What color is man’s hat?
Is it snowing now?
What is woman wearing?
Are they smiling?
Do you see trees?
1 man 1 woman
Yes
Looks like sunglasses
Man does
No
Black
No
Blue jacket and black pants
Yes
Yes
[Das+, CVPR 2017]
70. MNIST Dialog
VisDialのMNIST版
• 4x4のMNIST画像(白黒)に
– 文字色5種、背景色5種、スタイル2種を適用
– 画像数 50,000
– 対話 3 dialogs/image
– 質問 10 Q&As/dialog
How many 9’s are there in the image? four
How many brown digits are there among them? one
What is the background color of the digit at the left of it? white
What is the style of the digit? flat
What is the color of the digit at the left of it? blue
What is the number of the blue digit? 4
Are there other blue digits? two
[Seo+, NIPS 2017]
75. その他にも…
• Image-Grounded Conversation [Mostafazadeh+,
IJCNLP 2017]
– VisDialはVQAベース
No
Any huge pumpkins?
No
Do you see trees?
No
Do you see anyone?
That is possible
Do you think it's for Halloween?
Possibly
Is this at a farm?
No
Is the photo close up?
Yes
Is the photo in color?
76. その他にも…
• Image-Grounded Conversation [Mostafazadeh+,
IJCNLP 2017]
– VisDialはVQAベース
– 雑談の様な対話を収集
Place near my house is getting
ready for Halloween a little early.
Don't you think Halloween
should be year-round, though?
That'd be fun since it's
my favorite holiday!
It‘s my favorite holiday as well!
I never got around to carving a
pumpkin last year even though I
bought one.
Well, it's a good thing that they are
starting to sell them early this year!
77. その他にも…
• DialEdit [Ramesh+, 2018]
• Video Scene-Aware Dialog Data [Hori+, 2018]
– Dialog System Technology Challenge (DSTC) 7
– VisDialの動画バージョン
93. SL-Pretrained(教師あり事前学習のみ)
Questioner Answerer
A couple of people
in the snow on skis.
2
I can’t tell
I can’t tell, they are far away
I can’t tell
Yes
Yes
Yes
Yes
Yes
Yes
How many people?
Male or female?
What are they wearing?
What color are skis?
Are they wearing goggles?
Are they wearing goggles?
Are they wearing goggles?
Are they wearing goggles?
Are they wearing hats?
Are they wearing goggles?
94. RL-full-QAf(提案手法)
Questioner Answerer
A couple of people
in the snow on skis.
Are people male or female or both?
What are they wearing?
Is it snowing?
Can you see any trees?
Can you see any buildings in background?
Can you see any trees?
Can you see any buildings in background?
Can you see any trees?
Can you see any buildings in background?
Does it look like they are in park?
I think 1 is male, can’t see other 1 ...
They are all wearing snow pants and jackets
It does not appear to be snowing at moment
Yes, there are lot of trees in background
No, I can not
Yes, there are lot of trees in background
No, I do not see any buildings
Yes, there are lot of trees in background
No , I do not see any buildings
It does not appear to be
You can easily know the answer because the official site still has the information about ILSVRC 2012.
Yes, the 1st team with deep learning achieved 15% error, the 2nd team without deep learning achieved 26% error … and if you scroll down this web page, the members of the second team are shown in a table. There seems to be several guys in the second team, and now please remember this name. It is hard to pronounce. Yoshitaka Ushiku.
You can easily know the answer because the official site still has the information about ILSVRC 2012.
Yes, the 1st team with deep learning achieved 15% error, the 2nd team without deep learning achieved 26% error … and if you scroll down this web page, the members of the second team are shown in a table. There seems to be several guys in the second team, and now please remember this name. It is hard to pronounce. Yoshitaka Ushiku.