As presented by Emily Vinson (University of Houston) at the FIAT/IFTA World Conference 2019 in Dubrovnik, Croatia

1. ACCURACY & COST ASSESSMENT
for
ARCHIVAL VIDEO
TRANSCRIPTION METHODS
Emily Vinson
University of Houston

2. INTRODUCTION | University of Houston

3. INTRODUCTION | Collections

4. PURPOSE |
• Accessibility
• Research-ability
• Collection as Data

5. METHODOLOGY | Sample Videos
• Single speaker, scripted speech, clear recording
• Interview b-roll, crosstalk, off-mic speech
• Multi-person news segment, off-mic speech, unclear
speech
• Bilingual scripted speech
• Background music, repetitive language

6. STUDY DESIGN | Methods
• Professional transcriptionist services
• Online Artificial Intelligence-based services
• Free “hacks”
• Untrained study participants

7. STUDY DESIGN | Professional Services
• $0.80 - $2.95 per minute
• Average 1-2 week turnaround
• Promise 99% accuracy *
• Include timestamps, frequency varies
• Usually include some sort of speaker
identification, specificity varies
• Transcript file formats available vary by service

8. STUDY DESIGN | Online AI
• $0.10 - $0.25 per minute
• Minutes-long turnaround
• Some make no accuracy claims, others 90-95%*
• Include timestamps
• May or may not include indication of speaker
change
• Option to add “training” vocabulary terms
• Large variety of file formats available, including
TXT, DOCX, PDF, VTT, SRT

9. STUDY DESIGN | Free Hacks
Google Docs
• Requires set up to feed audio back into computer
• No punctuation
• Times out after long silences (or tones)
YouTube
• Requires upload to YouTube platform
• Seems to work only with short video files

10. STUDY DESIGN | Study Participants
• All participants were student or staff employees of
UH Libraries
• Most had little or no background in transcription
• Required to work at dedicated workstation with
transcription software and foot pedal
• Allowed to complete transcripts over as many
visits as they wanted, but asked to keep track of
time
• Only given minimal instructions

11. METHODOLOGY | Considerations
• Vocal pauses
• Disfluencies / False Starts
• Repetition
• Unfamiliar pronunciations
• Cross-talk
• Soft or off-camera speech
• Punctuation

12. METHODOLOGY | What is an error?
• Added or omitted words
• Misspellings/Typos
• Misused homonym
• principal vs. principle
• Contractions
discrepancy
• they’re vs. they are
• Exclusion of Disfluencies
• Alternate Spellings
• sarape vs. serape
• Numbers or Numerals
• twenty vs. 20
• Punctuation, capitalization
• “too close to call” words
Counted as Errors Not Counted as Errors

13. METHODOLOGY | Comparison
I was thinking especially of young people like little Will
Benson. His future and that of all his young peers will be
brighter for the efforts of men like his grandfather. Cornel
Bates has given the best that's in him, so that many younger
Americans will have he chance to realize the most that is in
him. Participant 6
I was thinking especially of young people like little Will
Benson. His future and that of all his young peers will be
brighter for the efforts of men like his grandfather. Colonel
Bates has given the best that's in him, so that many younger
Americans will have the chance to realize the most that is in
them. Service 3
I was thinking especially of young people like Little will
Benson his future and that of all his young peers will be
brighter for the efforts of men like his grandfather . Colonel
Bates is given the best that's in him so that many younger
Americans will have the chance to realize the most that is in
them. AI 3

14. METHODOLOGY | Comparison
McElroy: I don't think it has fallen off, uh, too much.
Interviewer: Uh sorry can you incorporate the question.
McElroy: Ok, I'm gonna make a F today, uh. The city services
in the community have not fallen of, uh, too much. I would
say there have been some neglect but uh, measurably I don't
think that it has fallen off too much.
Participant 1
I don’t think it had fallen off too much…
Sir, can you incorporate the question?
[Ok, I'm gonna make a F today, huh?] The city services in the
community have not fallen off too much. I would say there
have been some neglect, but measurably I don’t think that it
has fallen off too much.
Professional Service 1
Speaker 1: I don't think it has fallen off, uh, too much [Sorry
can you incorporate the question?] Okay. I'm gonna make
half today. The city services in the community have not fallen
off. Uh. Uh, too much. I would say there have been some
neglect, but uh, measurably I don't think that it has fallen
off too much AI 2

15. METHODOLOGY | Comparison
At first, I didn't know what was happening. I didn’t I just
knew I was real sick. And like after I came home from my
treatment, for the first treatment that they give you at, MD
Anderson and my dad took me into my room and told me
that I that I had cancer. And that if I wanted to live I had to
do every I had to do all my medicine and listen to him.
Participant 4
At first, I didn’t know what was happening. I didn’t I just
knew I was real sick. And like after I came home from my
treatment, from the first treatment that they give you, at
Andy Anderson and my dad took me into my room and told
me that I, that I had cancer. And that if I wanted to live, I had
to do every – I had to do all my medicine and listen to them.
Professional Service 2
At first I didn't know what was happening. I didn't, I just
knew I was real sick. And like after I came home for my
treatment for him the first treatment that they give you at
MD Anderson and my dad took me into my room and told
me that I um, what I did I had cancer and that if I wanted to
live, I had to do every everything. I had to do all my medicine
and listen to them. AI 2

16. METHODOLOGY | Comparison
The word canasta is one of three words that we
learned. Canast and Rio, river, and aqua, the water in the
river. Notice that the canasta is held on the back with a
raybosto, which is also a wool product that is woven locally
and is worn by women to keep them warm and also to carry
things. Participant 3
The word canasta is one of three words that we learned:
Canasta and rio, river and agua, the water in the river. Notice
that the canasta is held on the back with a reboso, which is
also a wool product that is woven locally and is worn by
women to give them warmth and also to carry things.
Professional Service 5
They weren't canasta is one of three words that we learned
canasta a real river. And I watched the water in the river
noticed that the canasta is held on the back with a Rainbow
Soul which is also a water product that is woven locally and
is worn by women to give them warmth and also to carry
things. AI 1

17. METHODOLOGY | Comparison
You can also do it moving sidewards. Suppose you try it that
way and move into the center of the circle first so you will go
in with step-step-step-hop and back with step-step-step-
hop. All right? And step-step-step-hop step-step step hop
step-step-step-hop step-step-step-hop step-step-step-hop
step-step-step-hop Yes! That’s fine.
Participant 5
Dr. Margaret P.: You can also do it moving sidewards.
Suppose you try it that way and move to the center of the
circle first. So you would go in with step, step, step, hop and
back with step, step, step, pop. All right. And step, step, step,
hop. Step, step, step, hop. Step, step, step, hop. Step, step,
step, hop. Step, step, step, hop. Step, step, step. Yes. That's
fine. Professional Service 4
You can also do it. Moving sideways. Suppose you try it that
way and move to the center of the circle first so you will go
in with Step step step hop and backward with step step step
Pop.
All right. And Yep. Yep. Yep. Yep. Yep. Yep. Yep. Yep. Yep. Yep.
Yep. Yep. Yep. Yep. Yes. That’s fine. AI 3

18. FINDINGS |Error By Sample
0
100
200
300
400
500
600
Bentsen McElroy Polo Textil Schottische

19. FINDINGS |Average Error Rates
0
50
100
150
200
250
300
350

20. FINDINGS |Accuracy vs. Cost
0
100
200
300
400
500
600
$0.00
$0.50
$1.00
$1.50
$2.00
$2.50
$3.00
Errors Price per minute

21. FINDINGS | Common Errors
• Proper nouns
• Spelling
• Tense changes
• Articles (such as swapping “a” and “the”)
• Repeated words and false starts
• “Correcting” grammar used by speaker

22. FINDINGS |Conclusions
• “Hacks” are not an option
• AI Services are worse than expected
• QC will always be important
• Professional services can be more affordable
than expected
• Should always keep the goal in mind, converting
a text-based file to SRT or WebVTT format for
Closed Captions will take additional time

23. NEXT STEPS|
• Consider quantifying errors by type
• Expand sample size, including lengthier samples
• Add participant QC of AI transcripts as method
• Include Speech to Text API methods

24. GET IN TOUCH| evinson@uh.edu