SlideShare une entreprise Scribd logo
Multimodal Transformer for Unaligned
Multimodal Language Sequences
유용상
ACL 2019
2022.11.8 NLP 스터디
Introduction
• 멀티모달이 필요한 이유?
: 언어는 글로만 이루어져 있지 않고 말하는 사람의 목소리, 표정 등이 전부 발화의 의미를 담고
있음
-> 텍스트 데이터에 더해 음성 데이터, 얼굴 표정 데이터 등을 결합해 모델로 하여금 더 잘
이해할 수 있도록 함
Multimodal model Sentimental analysis
Text
Image
Speech
기존 연구의 한계점
서로 다른 modality => 서로 다른 시퀀스 길이, 따라서 길이를 동일하게 맞추는 작업이 필요함 : Alignment
이것은 LSTM에 적용할 경우 long term multimodality를 반영하기 힘듦
Overall Architecture
Vectorize
Glove Vector 사용
Facet 라이브러리 사용
COVAREP 라이브러리 사용
1D convolution + positional embedding
• 어텐션 스코어끼리의 연산을 위해 차원수를 맞춰줌
• Local dependency 반영을 기대함
• Learned Vector 대신 original transformer와
같이 Sinusoidal Vector 사용
Crossmodal Transformer
Target Source
Crossmodal Transformer
Prediction
FC layer
experiments
experiments
experiments

Contenu connexe

Plus de YongSang Yoo

20230727_tinystories
20230727_tinystories20230727_tinystories
20230727_tinystories
YongSang Yoo
 
20230608_megabyte
20230608_megabyte20230608_megabyte
20230608_megabyte
YongSang Yoo
 
221220_페르소나챗봇
221220_페르소나챗봇221220_페르소나챗봇
221220_페르소나챗봇
YongSang Yoo
 
220920_AI ETHICS
220920_AI ETHICS220920_AI ETHICS
220920_AI ETHICS
YongSang Yoo
 
230309_LoRa
230309_LoRa230309_LoRa
230309_LoRa
YongSang Yoo
 
230305_Characterizing English Variation across Social Media Communities with ...
230305_Characterizing English Variation across Social Media Communities with ...230305_Characterizing English Variation across Social Media Communities with ...
230305_Characterizing English Variation across Social Media Communities with ...
YongSang Yoo
 
230223_Knowledge_Distillation
230223_Knowledge_Distillation230223_Knowledge_Distillation
230223_Knowledge_Distillation
YongSang Yoo
 
221011_BERT
221011_BERT221011_BERT
221011_BERT
YongSang Yoo
 
220910_GatedRNN
220910_GatedRNN220910_GatedRNN
220910_GatedRNN
YongSang Yoo
 
220906_Glove
220906_Glove220906_Glove
220906_Glove
YongSang Yoo
 

Plus de YongSang Yoo (10)

20230727_tinystories
20230727_tinystories20230727_tinystories
20230727_tinystories
 
20230608_megabyte
20230608_megabyte20230608_megabyte
20230608_megabyte
 
221220_페르소나챗봇
221220_페르소나챗봇221220_페르소나챗봇
221220_페르소나챗봇
 
220920_AI ETHICS
220920_AI ETHICS220920_AI ETHICS
220920_AI ETHICS
 
230309_LoRa
230309_LoRa230309_LoRa
230309_LoRa
 
230305_Characterizing English Variation across Social Media Communities with ...
230305_Characterizing English Variation across Social Media Communities with ...230305_Characterizing English Variation across Social Media Communities with ...
230305_Characterizing English Variation across Social Media Communities with ...
 
230223_Knowledge_Distillation
230223_Knowledge_Distillation230223_Knowledge_Distillation
230223_Knowledge_Distillation
 
221011_BERT
221011_BERT221011_BERT
221011_BERT
 
220910_GatedRNN
220910_GatedRNN220910_GatedRNN
220910_GatedRNN
 
220906_Glove
220906_Glove220906_Glove
220906_Glove
 

221108_Multimodal Transformer