At least 360 million people worldwide have disabling hearing loss that frequently causes difficulties in day-to-day conversations. Traditional technology (e.g., hearing aids) often fails to offer enough value, has low adoption rates, and can result in social stigma. Speechreading can dramatically improve conversational understanding, but speechreading is a skill that can be challenging to learn. To address this, we developed a novel speechreading acquisition framework that can be used to design Speechreading Acquisition Tools (SATs) – a new type of technology to improve speechreading acquisition. We interviewed seven speechreading tutors and used thematic analysis to identify and organise the key elements of our framework. We then evaluated our framework by using it to: 1) categorise every tutor-identified speechreading teaching technique, 2) critically evaluate existing conversational aids, and 3) design three new SATs. Through the use of SATs designed using our framework, the speechreading abilities of people with hearing loss around the world should be enhanced, thereby improving the conversational foundation of their day-to-day lives. Full Presentation: https://www.dropbox.com/s/qts1g2nguj6xixv/CHI_2017_Presentation.key?dl=0 Teaser Video: https://vimeo.com/196870070 Paper: http://dl.acm.org/citation.cfm?doid=3025453.3025560

1. Benjamin M. Gorman
AFramework for
Speechreading
AcquisitionTools
David R. Flatla

2. Benjamin M. Gorman
AFramework for
Speechreading
AcquisitionTools
David R. Flatla

4. CAN YOU READ MY LIPS?
from Little Moving Pictures

5. CAN YOU READ MY LIPS?
from Little Moving Pictures

6. 1 in 6 People
Action on Hearing Loss Information 2011 (UK)

7. Million People
360
~5% World’s population have
disabling hearing loss
World Health Organization 2015

8. ?
COMMUNICATION
Barriers

9. Only of people with hearing loss
regulary use hearing aids
14%
Action on Hearing Loss Information,2011 (UK)

10. SPEECHREADING
can break barriers and
improve understanding

11. SPEECHREADING
can break barriers and
improve understanding

12. Teeth
Nasal Cavity
Lips
Vocal Chords
Tongue

13. Teeth
Nasal Cavity
Lips
Vocal Chords
Tongue

14. Cece Bell, El Deafo (2014)

15. Speechreading can improve:
Educational outcomes
Employability
Social isolation
And restore
conversational con!dence

16. To ManyOne
VISEME PHONEMES

17. Cece Bell, El Deafo (2014)

18. Cece Bell, El Deafo (2014)

19. SPEECHREADING
CLA SSES

20. 50Classes Available
Across Scotland

21. 275More Classes To Meet Demand
There needs to be

22. Speechreading can dramatically
improve conversational
understanding.
However, speechreading is a skill
that can be challenging to learn.
We want to design technology
to help address this problem.

23. Speechreading can dramatically
improve conversational
understanding.
However, speechreading is a skill
that can be challenging to learn.
We want to design technology
to help address this problem.

24. Speechreading can dramatically
improve conversational
understanding.
However, speechreading is a skill
that can be challenging to learn.
We want to design technology
to help address this problem.

25. §We developed a framework that
can be used to design
– a new type of technology to improve
speechreading acquisition
Speechreading
Acquisition Tools

26. §We developed a framework that
can be used to design
– a new type of technology to improve
speechreading acquisition
Speechreading
Acquisition Tools

27. RESEARCH QUESTIONS
1. Do tutors teach analytic or synthetic training methods?
2. What do tutors consider to be the teachable unit of
lipreading?
3. What technology if any is currently used in teaching?
4. Could lipreading as a technique be improved with new
technology or training approaches?
5. How do students practice when they are not in class?

28. RESEARCH QUESTIONS
1. Do tutors teach analytic or synthetic training methods?
2. What do tutors consider to be the teachable unit of
lipreading?
3. What technology if any is currently used in teaching?
4. Could lipreading as a technique be improved with new
technology or training approaches?
5. How do students practice when they are not in class?

29. RESEARCH QUESTIONS
1. Do tutors teach analytic or synthetic training methods?
2. What do tutors consider to be the teachable unit of
lipreading?
3. What technology if any is currently used in teaching?
4. Could lipreading as a technique be improved with new
technology or training approaches?
5. How do students practice when they are not in class?

30. TEACHING
METHODS

31. Synthetic
Basic Unit:
Sentence
(CONTEXT)

32. Analytic
Basic Unit:
Syllable or Phoneme
(EYE TRAINING)

33. RESEARCH QUESTIONS
1. Do tutors teach analytic or synthetic training methods?
2. What do tutors consider to be the teachable unit of
lipreading?
3. What technology if any is currently used in teaching?
4. Could lipreading as a technique be improved with new
technology or training approaches?
5. How do students practice when they are not in class?

34. RESEARCH QUESTIONS
1. Do tutors teach analytic or synthetic training methods?
2. What do tutors consider to be the teachable unit of
lipreading?
3. What technology if any is currently used in teaching?
4. Could lipreading as a technique be improved with new
technology or training approaches?
5. How do students practice when they are not in class?

35. RESEARCH QUESTIONS
1. Do tutors teach analytic or synthetic training methods?
2. What do tutors consider to be the teachable unit of
lipreading?
3. What technology if any is currently used in teaching?
4. Could lipreading as a technique be improved with new
technology or training approaches?
5. How do students practice when they are not in class?

36. We interviewed 7 speechreading tutors
to see how acquisition could be improved
with technology.

37. All Female
7 Tutors
Aged between 42-78
4/7 with hearing loss
Teaching Experience:
6 months - 32 years

38. All Female
7 Tutors
Aged between 42-78
4/7 with hearing loss
Teaching Experience:
6 months - 32 years

39. All Female
7 Tutors
Aged between 42-78
4/7 with hearing loss
Teaching Experience:
6 months - 32 years

40. All Female
7 Tutors
Aged between 42-78
4/7 with hearing loss
Teaching Experience:
6 months - 32 years

41. All Female
7 Tutors
Aged between 42-78
4/7 with hearing loss
Teaching Experience:
6 months - 32 years

42. THEMATIC
ANALYSIS

43. Thematic Map
Speechreading
Classes
Speechreading
as a Skill
Access To
Speechreading
Continual
vs Fixed
Attitudes to
Technology
Open to New
Approaches
Critical of Current
Approaches
Teaching
Practices
Varied
Techniques
Synthetic
vs Analytic

44. FINDINGS

45. FRAMEWORK
DESIGN

46. FRAMEWORK
DESIGN

47. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW

48. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW

49. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW

50. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW

51. FRAMEWORK
EVALUATION

52. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Teaching Techniques

53. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Mirror Practice
QREs
Word Quizzes
(name the animals)
Cue Recognition
(body language)
Pair Work
Framed Sentence
Excercises
Scenarios
Mystery Object
Stories
Speech Movements
Lip Shapes
Finger Spelling
(initial letter)
Teaching Techniques

54. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Mirror Practice
QREs
Word Quizzes
(name the animals)
Cue Recognition
(body language)
Pair Work
Framed Sentence
Excercises
Scenarios
Mystery Object
Stories
Speech Movements
Lip Shapes
Finger Spelling
(initial letter)
Teaching Techniques

55. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Mirror Practice
QREs
Word Quizzes
(name the animals)
Cue Recognition
(body language)
Pair Work
Framed Sentence
Excercises
Scenarios
Mystery Object
Stories
Speech Movements
Lip Shapes
Finger Spelling
(initial letter)
Teaching Techniques

56. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Mirror Practice
QREs
Word Quizzes
(name the animals)
Cue Recognition
(body language)
Pair Work
Framed Sentence
Excercises
Scenarios
Mystery Object
Stories
Speech Movements
Lip Shapes
Finger Spelling
(initial letter)
Teaching Techniques

57. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Mirror Practice
QREs
Word Quizzes
(name the animals)
Cue Recognition
(body language)
Pair Work
Framed Sentence
Exercises
Scenarios
Mystery Object
Stories
Speech Movements
Lip Shapes
Finger Spelling
(initial letter)
Teaching Techniques

58. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Mirror Practice
QREs
Word Quizzes
(name the animals)
Cue Recognition
(body language)
Pair Work
Framed Sentence
Exercises
Scenarios
Mystery Object
Stories
Speech Movements
Lip Shapes
Finger Spelling
(initial letter)
Teaching Techniques

59. FRAMEWORK
EVALUATION

60. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Cued Speech,
Spectrograms
Subtitles,
Signed Languages
LipreadingPractice CME (C)
CME (B)
DAVID
Upton, Ebrahimi
TactileSATs, CME(A)
Lip Assistant, iBaldi
Related Work

61. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Cued Speech,
Spectrograms
Subtitles,
Signed Languages
LipreadingPractice CME (C)
CME (B)
DAVID
Upton, Ebrahimi
TactileSATs, CME(A)
Lip Assistant, iBaldi
Related Work

62. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Cued Speech,
Spectrograms
Subtitles,
Signed Languages
LipreadingPractice CME (C)
CME (B)
DAVID
Upton, Ebrahimi
TactileSATs, CME(A)
Lip Assistant, iBaldi
Related Work

63. FRAMEWORK
EVALUATION

64. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Cued Speech,
Spectrograms
Subtitles,
Signed Languages
LipreadingPractice CME (C)
CME (B)
DAVID
Upton, Ebrahimi
TactileSATs, CME(A)
Lip Assistant, iBaldi

65. PhonemeViz

66. PhonemeViz

67. Skill Type
Amountof
Information
LipreadingPractice
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Cued Speech,
Spectrograms
Subtitles,
Signed Languages
PhonemeViz
MirrorMirror
ContextCueView
CME (C)
CME (B)
DAVID
Upton, Ebrahimi
TactileSATs, CME(A)
Lip Assistant, iBaldi

68. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Cued Speech,
Spectrograms
Subtitles,
Signed Languages
LipreadingPractice CME (C)
CME (B)
DAVID
Upton, Ebrahimi
TactileSATs, CME(A)
Lip Assistant, iBaldi

69. MirrorMirror

70. Skill Type
Amountof
Information
LipreadingPractice
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Cued Speech,
Spectrograms
Subtitles,
Signed Languages
PhonemeViz
MirrorMirror
ContextCueView
CME (C)
CME (B)
DAVID
Upton, Ebrahimi
TactileSATs, CME(A)
Lip Assistant, iBaldi

71. Skill Type
Amountof
Information
LipreadingPractice
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Cued Speech,
Spectrograms
Subtitles,
Signed Languages
PhonemeViz
MirrorMirror
ContextCueView
CME (C)
CME (B)
DAVID
Upton, Ebrahimi
TactileSATs, CME(A)
Lip Assistant, iBaldi

72. Speechreading: A Way to Improve Understanding
Kaplan et al. (1985)
Constellation Diagram

73. ContextCueView

74. ContextCueView

75. Skill Type
Amountof
Information
LipreadingPractice
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Cued Speech,
Spectrograms
Subtitles,
Signed Languages
PhonemeViz
MirrorMirror
ContextCueView
CME (C)
CME (B)
DAVID
Upton, Ebrahimi
TactileSATs, CME(A)
Lip Assistant, iBaldi

76. Speechreading can dramatically
improve conversational
understanding.
However, speechreading is a skill
that can be challenging to learn.
We want to design technology
to help address this problem.

77. Speechreading can dramatically
improve conversational
understanding.
However, speechreading is a skill
that can be challenging to learn.
We want to design technology
to help address this problem.

78. Speechreading can dramatically
improve conversational
understanding.
However, speechreading is a skill
that can be challenging to learn.
We want to design technology
to help address this problem.

79. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
SAT Framework

80. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Mirror Practice
QREs
Word Quizzes
(name the animals)
Cue Recognition
(body language)
Pair Work
Framed Sentence
Excercises
Scenarios
Mystery Object
Stories
Speech Movements
Lip Shapes
Finger Spelling
(initial letter)
Teaching Techniques

81. Skill Type
Amountof
Information
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Cued Speech,
Spectrograms
Subtitles,
Signed Languages
LipreadingPractice CME (C)
CME (B)
DAVID
Upton, Ebrahimi
TactileSATs, CME(A)
Lip Assistant, iBaldi
Related Work

82. Skill Type
Amountof
Information
LipreadingPractice
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Cued Speech,
Spectrograms
Subtitles,
Signed Languages
PhonemeViz
MirrorMirror
ContextCueView
CME (C)
CME (B)
DAVID
Upton, Ebrahimi
TactileSATs, CME(A)
Lip Assistant, iBaldi
Designing new SATs

83. Skill Type
Amountof
Information
LipreadingPractice
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Cued Speech,
Spectrograms
Subtitles,
Signed Languages
PhonemeViz
MirrorMirror
ContextCueView
CME (C)
CME (B)
DAVID
Upton, Ebrahimi
TactileSATs, CME(A)
Lip Assistant, iBaldi
Summary & Conclusion
b.gorman@dundee.ac.uk
@benjgorman

84. EXTENSIONS

85. Skill Type
Amountof
Information
LipreadingPractice
ANALYTIC HYBRID SYNTHETIC
HIGHMEDLOW
Cued Speech,
Spectrograms
Subtitles,
Signed Languages
PhonemeViz
MirrorMirror
ContextCueView
CME (C)
CME (B)
DAVID
Upton, Ebrahimi
TactileSATs, CME(A)
Lip Assistant, iBaldi
Summary & Conclusion
b.gorman@dundee.ac.uk
@benjgorman