Talk at 6th NLP Dublin meetup (https://www.meetup.com/NLP-Dublin/events/237998517/) by Emer Gilmartin, PhD student at Trinity College Dublin.

1. SPOKEN DIALOGUE SYSTEMS AND
SOCIAL TALK
Emer Gilmartin
Speech Communication Lab
Trinity College Dublin

2. What?
• Spoken dialogue systems attempt to create a spoken
interaction with a user
• Dialogue systems, Intelligent Virtual Agents (IVA’s),
Embodied Conversational Agents (ECA’s), Chatbots
• Dream (Turing, 1950 ) vs Practical Progress (Allen, 2000)
• AI – early chat – pattern matching – ELIZA
• Practical Dialogues – task to be performed - Practical Dialogue
Hypothesis (Allen, 2000)

3. What’s out there?
• Command and Control – voice commands
• Information Retrieval – Siri
• Interactive Voice Response – IVR
• Chatbots
• Embodied Conversational Agents (ECA)
• Intelligent Virtual Agents

4. Casual conversation – the unmarked case
• Ordering a pizza (transactional)
• performing a well-defined task
• content (‘What?’) vital for success
• Chat with neighbour (interactional)
• building/maintaining social bonds
• social (‘How?’) very important
• ‘continuing state of incipient talk’

5. What is a Spoken Dialogue System?

6. Multimodality
• Expression and Recognition
• Audio, visual, verbal, vocal, non-verbal, facial expression,
gesture, posture…
• Presence, affect, attitude...

7. • The Problem: Building social dialogue systems entails
understanding of casual social dialogue but…
• Much linguistic theory is based on language similar to writing but
highly unlike talk
• regards spoken interaction as debased, chaotic
• SDS technology based on
• Practical Dialogue Hypothesis (Allen, 2000)
• Constraint introduced to make dialogue modelling tractable
• Much corpus study of spoken interaction based on Task-based
Dialogue
• Information gap activities – MapTask (HCRC), DiaPix (Lucid)
• Meetings – AMI, ICSI
• These are not corpora of casual or social talk

8. Social Talk
• Spoken interaction as social activity
• Malinowski, Dunbar, Jakobsen, Brown and Yule
• Structure and Content
• Smalltalk at the margins (Laver)
• Chat and chunks (Slade & Eggins)
• Bouts – gossip, narrative
• Bouts end with ‘idling’ (Schneider)
• Phases – greetings, approach, centre, leavetaking (Ventola)
• Multiparty (Slade)
• Problems:
• much of this is theory, analysis by example
• based on orthographical transcriptions
• corpus based studies on transactional dyadic interaction,
phonecalls…
January 15, 2016 IWSDS 2016

9. Genre differences in spoken interaction?
• Spoken interaction is situated
• ‘speech-exchange systems’ (SSJ),
• communicative activities (Allwood)
• Some low level mechanisms may follow universal
patterns
• It is also possible that even basic interaction
mechanisms such as turn-taking vary with the type
and parameters of different interactions
• What might vary?
• Utterance/turn characteristics
• Distribution of pauses/gaps/overlaps
• ‘Disfluencies’, VSU’s, laughter…
• Explore different genres and use knowledge to
inform design of interfaces

10. Anatomy of casual conversation
January 15, 2016 IWSDS 2016
L
A
C
G

11. 10 minutes from a 5-party casual conversation showing chat (240s-480s and chunk 480 –
end) phases
Red-speech, yellow-laughter, grey-silence
January 15, 2016 IWSDS 2016

12. Chunk to Chunk Transition – more
interaction and laughter at end of chunks
January 15, 2016 IWSDS 2016

13. Chat/Chunk
• Significant differences in
• Length – chat very variable, chunk ~ 30s
• Phrase final prosody
• Gap lengths
• Important because;
• Need different timing modules for different phases
• Useful to know which phase we’re in
• Current Work
• Stochastic model – simple bigram HMM can classify chat/chunk
• Goal - online classifier – knowledge additional to ASR to inform
dialogue management
January 15, 2016 IWSDS 2016

14. How do I make a spoken dialogue
system?
• Virtual Human Toolkit (https://vhtoolkit.ict.usc.edu/)
• Pandorabots (www.pandorabots.com)
• CSLU Toolkit (http://www.cslu.ogi.edu/toolkit/)
• Voxeo Prophecy (http://voxeo.com/prophecy/)
• AT&T Speech
Mashupshttps://service.research.att.com/smm/login.jsp

15. SPOKEN AND
MULTIMODAL DIALOGUE
APPLICATIONS IN
HEALTHCARE
Emer Gilmartin
Speech Communication Lab
SCSS

16. ELIZA – text-based Rogerian therapist
• Weizenbaum - 1966
• http://www.masswerk.at/elizabot/eliza_test.html

17. NWU - Relational Agents Group
• Relational agents group
• https://www.youtube.com/watch?v=Jx5Tsn9wFw
• Presentation
• https://www.youtube.com/watch?v=lcb_rMhJQTI
• RAISE
• https://www.youtube.com/watch?v=ttBMG-F1HS0

18. ICT – Virtual Humans
• Simsensei
• https://www.youtube.com/watch?v=I2aBJ6LjzMw
• https://www.youtube.com/watch?v=ejczMs6b1Q4

19. Simulations and Virtual Reality
• ICT – MILES – Training for Therapists
• https://www.youtube.com/watch?v=KNGVRePdEL8
• ICT - Standard Patient Hospital

20. The future
• ICTnarrative
• https://www.youtube.com/watch?v=4IBoO2IKFMI
• Bickmore’s warning
• https://www.youtube.com/watch?v=V20KEjIjFl8

21. • http://relationalagents.com/index.html