Online collections of video material are fast becoming a staple feature of the Internet and a key educational resource. What we are working on at transLectures is a set of easy-to-use tools that will allow users to add multilingual subtitles to these videos. In doing so, they will make the content of these videos available to a much wider audience in a way that is cost-effective and sustainable over the vast collections of online video lectures being generated. Automatic transcription tools will provide verbatim subtitles of the talks recorded on video, thereby allowing the hard-of-hearing to access this content. Language learners and other non-native speakers will also benefit from these monolingual subtitles. Meanwhile, machine translation tools will make these subtitles available in languages other than that in which the video was recorded. Specifically, we will be developing tools for use on VideoLectures.NET, a collection of videos recorded at various academic events set up by JSI’s Centre for Knowledge Transfer, and for poliMedia, a lecture capture system designed and implemented at the UPVLC. Our tools will also be fully compatible with Matterhorn, a free, open-source platform for the management of educational audio and video content. The language pairs being targeted in this project are English, Spanish and Slovenian for transcription, and English<>Spanish, English<>Slovenian, English>French and English>German for translation. http://www.translectures.eu

1. FP7-ICT-2011-7 - Language technologies (STREP)
Duration: 36 months (1 Nov 2011 – 31 Oct 2014)
No Name Short Country
1 UNIVERSITAT POLITECNICA DE VALENCIA UPVLC Spain
2 XEROX SAS XEROX France
3 INSTITUT JOZEF STEFAN JSI Slovenia
3+ KNOWLEDGE FOR ALL FOUNDATION K4A UK
4 RHEINISCH-WESTFAELISCHE
TECHNISCHE HOCHSCHULE AACHEN
RWTH Germany
5 EUROPEAN MEDIA LABORATORY GMBH EML Germany
6 Deluxe Digital Studios Ltd DDS UK

2. Outline of the talk
• Motivation
• Goal and technology challenges
• Demo videos: project overview, intelligent
interaction and Matterhorn integration
• Transcriptions and translations at the pilots
• Massive adaptation
• Intelligent interaction with users
• Integration and evaluation

3. • Large on-line repositories of video lectures are being
established: VideoLectures.NET, poliMedia, OCW and
Opencast Matterhorn-based repositories, etc.
• Video lectures are neither transcribed nor translated
due the lack of cost-effective tools.
• Transcriptions and translations are needed to make
them accessible to speakers of different languages
and people with disabilities. They would also
facilitate further search and analysis functions.
• Starting hypothesis: current ASR and MT techniques
are not far from achieving acceptable results in
educational repositories.
Motivation

4. Goal and technology challenges
Goal:
To develop innovative, cost-effective solutions to produce
accurate transcriptions and translations in VideoLectures.NET,
with generality across other Matterhorn-related repositories.
Technology challenges:
• Improvement of transcription and translation quality by
massive adaptation.
• Improvement of transcription and translation quality by
intelligent interaction.
• Integration into Matterhorn to enable real-life evaluation.

5. Demo videos
• Project overview
https://www.translectures.eu/project-summary
• Intelligent interaction with users
https://www.translectures.eu/web/progress
• Matterhorn integration
https://www.translectures.eu/web/progress

6. Transcriptions and translations at the pilots
Language Nb. of lectures Nb. of hours
VideoLectures.NET
English 9148 5900
Slovenian 741 490
Total 9889 6390
poliMedia
(live)
Spanish 7554 996
English 420 62
Catalan 91 11
Total 8065 1069
VideoLectures.NET: EnEs, EnFr, EnDe, EnSl, SlEn
poliMedia: EsEn

7. Massive adaptation
• Massive adaptation of acoustic models
• Massive adaptation of language models
• Massive adaptation of translation models

8. 10
15
20
25
30
35
40
45
50
M06 M12 M18 M24 M30
WER (ASR)
Sl
Ca
En
Es
15
20
25
30
35
40
M12 M18 M24 M30
BLEU (MT)
EnEs
EnFr
EsEn
EnDe
SlEn
EnSl

9. Intelligent interaction with users
• Fast constrained search
• Intelligent interaction for transcription
• Intelligent interaction for translation
• Incremental training for translation

10. Transcription of 23 videos (3h) with no interaction
1
1 2
2
1 2 3
3
1 2 3 4 5
4
1 2 3 4 5
5
1 2 3 4 5
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER

11. Transcription of 23 videos (3h) with no interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
18.8

12. Transcription of 23 videos (3h) with batch interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
18.4 BI

13. Transcription of 23 videos (3h) with batch interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
18.0 BI

14. Transcription of 23 videos (3h) with batch interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
17.6 BI

15. Transcription of 23 videos (3h) with batch interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
17.2 BI

16. Transcription of 23 videos (3h) with batch interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.8 BI

17. Transcription of 23 videos (3h) with batch interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.8 BI

18. Transcription of 23 videos (3h) with batch interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI

19. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
18.5 II

20. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
18.1 II

21. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
18.0 II

22. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
17.5 II

23. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
17.3 II

24. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
17.3 II

25. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
17.2 II

26. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
17.0 II

27. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI16.5 II

28. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI16.4 II

29. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
16.0 II

30. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
15.7 II

31. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
15.7 II

32. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
15.6 II

33. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
15.4 II

34. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
15.2 II

35. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
15.1 II

36. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
14.5 II

37. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
14.5 II

38. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
14.3 II

39. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
14.3 II

40. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
14.0 II

41. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
13.6 II

42. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
13.6 II

43. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
13.4 II

44. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
13.1 II

45. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
13.0 II

46. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
12.6 II

47. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
12.6 II

48. Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18
Supervised minutes
WER
16.5 BI
12.4 II

49. Integration and evaluation
• Integration and evaluation at the pilots
• Try our tools and transcribe your videos
www.translectures.eu
Free service opened by UPV on 13 March 2014