3. Introduction
• Field study
- Describe an example of how a computer
can be used as a medium for engagement
- Understand why would this be a promising
field for HCI applications
2
4. Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
3
5. Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
4
8. The system: Euclide
• Has been successfully used for engagement
in the Science museum in Naples, Italy
• Why? How could it be improved? What can
we learn for future puppetry systems?
7
33. The system: Euclide
• Compromise between
- Museum’s requirements for cheap, robust,
generic and flexible hardware
- Puppeteers’ requirements for large number
of functionalities
- Usability
· e.g. mouth control moved from index to thumb
to avoid tiredness
32
34. Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
33
35. Data Collection
• Video+audio recordings
- Over several months
- Four viewpoints
- Three different pupeteers
• Interview of head pupeteer
34
36. Data Analysis
• Constant comparison analysis
• Top-down and Bottom-up approach
- Findings originates from both the data and
our research questions
• Coding scheme built through three
steps: open, axial, selective coding
• Clips selected together, then divided
between coders
• Discussion and revision of coding until
agreements > 0.6 35
37. Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
36
42. Findings: Multimodal Puppetry
Speech durations
Silence durations
• Lot of very short speeches and silences
in order to maintain engagement
• Puppet active even when silent
41
43. Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
42
44. Findings: Emergence of
Performative Structure
• Engaging the Audience Throughout the
Sessions: Evidences
- People in the audience:
· Don’t talk to each other 99% of the time
· Don’t pay attention to the outside 98% of the time
· Talk to the puppet 65% of the time
· Pleasure and Arousal annotated as positive all the
time
· Number increase, then decrease
43
45. Findings: Emergence of
Performative Structure
Number and length of sessions (mins)
10
8
6
4
2
0
< 1 min 1 min < < 2 min 2 min < < 9 min > 9 min
• Rather short, but:
- Sessions interrupted by teachers
- Group of children continue interacting
44
from the next puppet station
46. Findings: Emergence of
Performative Structure
• Structure of sessions
- Approach: enter the interaction area;
observe
- Testing: Try to interact to find out which
actions have an effect
- Playing: Interact in an aware, active,
involved way; climax
- Ending: attention diverted; leave
45
47. Findings: Emergence of
Performative Structure
• Structure of sessions
- Approach, Testing, Playing, Ending
- Long playing phases, with scenery animations
46
49. Findings: Emergence of
Performative Structure
• the audience starts interacting happily but calm,
• then gets excited,
• finishes the interaction happily and calm again
48
50. Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
49
51. Findings: Construction of
Engagement
• The Puppet’s Multimodal Resources:
Effects (face, body and scenery
animations), like natural gestures, in
relation to speech
- Pure special effects
- Non-verbal turns
- Verbal accompaniements
- Virtual gesturing
50
52. Findings: Construction of
Engagement
• Pure special effects
- Non verbal ressources are predominant
modality for interacting with the audience
- Used to attract visitors that cannot be reached
by speech (babies, far away visitors)
51
53. Findings: Construction of
Engagement
• Non-verbal turns: Contribution in an
interaction that includes speech, e.g.
- wearing accessory like glasses on special
request from the audience and subsequently
commenting on it
- Explaining a fact by displaying a picture of it
52
54. Findings: Construction of
Engagement
• Verbal accompaniements: Non-verbal
actions are accompagnied by speech, e.g.
- Acrobatic movement with onomatopeic sound
→ Communication relies on the nonverbal
resource, which dictates its structure and
meaning
53
55. Findings: Construction of
Engagement
• Verbal accompaniements: Example
Puppet: Ho una molla
I’ve got a spring
Mi serve per saltare
I need it to jump
⎡ Dong:: dong:: ⎤
Dong dong
⎣((Starts jumping))⎦
54
56. Findings: Construction of
Engagement
• Virtual gesturing: priority is given to
speech; visual features are used in
synch with speech, following its
structure and duration, e.g.
- Head movement, like pointing gesture
55
57. Findings: Construction of
Engagement
• Virtual gesturing: Example
Visitor: Dove sei?
Where are you?
Puppet: Sono⎡::((pause))⎤
I’m
⎣((Looking down))⎦
⎡qui dentro,⎤
inside here
⎣((Looking around))⎦
non mi vedi?
can’t you see me 56
58. Findings: Construction of
Engagement
• Multimodal interaction for puppeteer
is essential
- Helps avoiding the low status of a
conversational agent, e.g. verbal abuse
57
59. Findings: Construction of
Engagement
• Response to verbal abuse: Example
Child: (Scemo)
Dumb
Puppet: Vabbeh adesso basta.
All right; let’s stop this
Child: ((turns back, surprised))
Puppet: Ogni volta che scemo, ogni volta che scemo.
All the times dumb, all the times dumb.
E tu invece come sei?
And what about you instead, what are you?
Child: ((goes away))
58
60. Findings: Construction of
Engagement
• Talk = Main ressource,
Multimodal animation = instrument
• Combination attracts visitors effectively
59
61. Outline
1. The system studied
2. Data collection and analysis
3. Findings
a. Multimodality
b. Performative structures
c. Construction of engagement
4. Conclusion
60
62. Conclusion
• Why is digital puppetry a promising application
field for HCI?
- On the contrary to a fully computer-driven system
· Engagement stems from combination of multimodal
ressources by puppeteer
· Users from the audience inspire the narrative by the
puppeteer and feel like they have an active role
61
63. Conclusion
• Why is digital puppetry a promising application field
for HCI?
- Performative Structures for Brief Interactions repetitive
element of an entertainment interface could be further
taken into account
- Multimodal Use of the System by the Puppeteer
possibility to introduce advanced interface techniques
· Easing lip/speech sync,
· Mapping expressive hand gestures to puppet expressions,
· Etc.
62