Speech To Sign Language Interpreter System

By: Khalid El-Darymli G0327887 Speech to Sign Language Interpreter System Supervisor: Dr. Othman O. Khalifa International Islamic University Malaysia Kulliyyah of Engineering, ECE Dept.

OUTLINE ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Problem Statement ,[object Object],[object Object],[object Object],! IS IT FAIR ?

RESEARCH GOAL AND OBJECTIVES ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Main Parts of Speech to Sign Language Interpreter System Speech-Recognition Engine ASL pre-recorded Video-clips Database Recognized Text ASL Translation Continuous Input Speech Recognized Text

Automatic Speech Recognition ( ASR ): ,[object Object],[object Object],SR Engine Recognized Text Input Voice

The Structure of SR Engine (LVCSR) Signal Processing AM P ( A 1 , …, A T | P 1 ,… , P k ) Dictionary P ( P 1 , P 2 , …, P k | W ) LM P ( W n | W 1 , …, W n-1 ) X={x 1 ,x 2 , …, x T } Hypothesis Evaluation Decoder P(X | W)*P(W) TRAINING DECODING Best Hypotheses H = {W 1 , W 2 , …, W k } W BEST Input Audio

SIGNAL PROCESSING (FRONT-END) : Pre-emphasis Framing Windowing Speech waveform y[n] y t ` [n] Power Spectrum Calculation y t [n] Mel Filterbank S t [k] ln| | 2 IDFT 13 c t [n] 13  c t [n] 13  c t [n] x[n] , 16-bits integer data S t [m] Pre-emphasis  is the pre-emphasis parameter. ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Speech waveform of a phoneme “e” ,[object Object],After pre-emphasis and Hamming windowing Power spectrum MFCC

TRAINING ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],AM P ( A 1 , …, A T | P 1 ,… , P k ) Dictionary P ( P 1 , P 2 , …, P k | W ) LM P ( W n | W 1 , …, W n-1 )

HMM s ,[object Object],[object Object],[object Object],S 0 S 1 S 2 S 3 a 00 a 11 a 22 b 0 (k) b 1 (k) b 2 (k)

Dictionary : ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],AM P ( A 1 , …, A T | P 1 ,… , P k ) Dictionary P ( P 1 , P 2 , …, P k | W ) LM P ( W n | W 1 , …, W n-1 )

Language Model (LM): ,[object Object],[object Object],[object Object],AM P ( A 1 , …, A T | P 1 ,… , P k ) Dictionary P ( P 1 , P 2 , …, P k | W ) LM P ( W n | W 1 , …, W n-1 )

RECOGNITION ,[object Object],[object Object],[object Object],Dynamic Structure Search Algorithm S * Static Structure  S t , P(x t ,{s t }| {s t-1 } ,  ) {S t-1 } x t

The Veterbi Beam search ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

SIGN LANGUAGE ,[object Object],[object Object]

AMERICAN SIGN LANGUAGE ( ASL ) ,[object Object],[object Object],[object Object],[object Object]

ASL ALPHABETS ,[object Object],[object Object],[object Object],[object Object],[object Object],Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Ww Xx Yy Zz

SIGNED ENGLISH ( SE ): ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

ASL vs. SE (an Example) It is alright if you have a lot ASL Translation SE Translation IT I S ALL RIGHT IF YOU HAVE A LOT

DEMONSTRATION OF THE ASL IN OUR SW: A number of 2,600 ASL prerecorded video clips In case of nonbasic word, extract the basic word out of it Recognized Word (SR engine’s output) Is the basic word within the ASL database vocabulary? The American Manual Alphabet Only in case of a nonbasic input word, append some suitable marker Final Output None of the database contents matched the input basic word No Yes Fingerspelling of the original input word The equivalent ASL video clip of the input word, some marker could be appended

Speech to Sign Language Interpreter System - MILESTONE Thesis Writing Outline & Progress SW Development & Progress % Drafted Chapter 2: State-of-the-Art of SR Chapter 3: Sphinx SR Chapter 4: Sphinx Decoder Chapter 5: Sign Language Chapter 6: SW Demo ., Conclusions & Further Work Appendices SR Engine ASL Database Overall Integrated SW Chapter 1: Introduction % Completed

Thank You ,[object Object],[object Object]

Speech To Sign Language Interpreter System

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Speech To Sign Language Interpreter System

Similaire à Speech To Sign Language Interpreter System (20)

Plus de kkkseld

Plus de kkkseld (11)

Dernier

Dernier (20)

Speech To Sign Language Interpreter System