Pedagogical Conversational Agents (PCAs) have the advantage of offering to students not only task-oriented support but also the possibility to interact with the computer media at a social level. This form of intelligence is particularly important when the character is employed in an educational setting. This paper reports our initial results on the recognition of users' social response to a pedagogical agent from the linguistic, acoustic and gestural analysis of the student communicative act.
What's New in Teams Calling, Meetings and Devices March 2024
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
1. Università degli studi di Bari “Aldo Moro”
Dipartimento di Informatica
Recognising the Social Attitude in Natural
Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella
{decarolis, ferilli, novielli}@di.uniba.it, {fabio.leuzzi, fulvio.rotella}@uniba.it
DIDAMATICA, Informatica per la Didattica
Taranto, Italy, May 14-16, 2012
2. Overview
● Introduction
● Objective
● The proposed model
● The proposed approach
● Signs of social attitude
● Evaluation
● Conclusions
● Future works
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 2
3. Introduction
Pedagogical Conversational Agent (PCA)
● fulfil pedagogical goals
● interact with the user through a natural dialog by
appropriately mixing verbal and non verbal expressions:
● recognize verbal and non-verbal inputs
● generate verbal and non-verbal outputs
● handle typical functions of human conversations,
with particular emphasis on social aspects
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 3
4. Objective
Aim: building a multimodal framework for the recognition
of the social response of users to a PCA.
In particular: building a framework that integrates the
analysis of the linguistic component of the user's
communicative act with the analysis of the acoustic
features of the spoken sentence and of the gestures.
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 4
5. Objective
The intuition
The combination of these different input modalities may
improve the recognition of multimodal behaviours that
may denote the openness attitude of the users towards
the embodied agent.
Steps:
● Recognize signs of social attitude
● Build a model to infer the user attitude toward the PCA
● Adapt the dialog strategies accordingly
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 5
6. The proposed model
Dynamic Belief Network (DBN):
● handling uncertainty and incompleteness of data
● representing situations which gradually evolve from a
dialog step to the next one
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 6
7. The proposed approach
● The model is initialized
● At every dialog step:
● Knowledge about the evidence is produced
● The produced knowledge is entered and propagated in the
network
● The model revises the probabilities of the social attitude node
● The new probabilities of the signs of social attitude are used
for planning the next agent move
● The probability of the social attitude node supports revising
high-level planning of the agent behaviour
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 7
8. Signs of social attitude
in the language
● Sense of intimacy (use of common jargon)
● Friendly self-introduction
● Familiar style
● Attempt to establish a common ground
● Talk about self
● Personal questions about the agent
● Irony and humour
● Benevolent/polemic attitude towards the system failures
● Favourable/negative comments
● Interest to protract interaction
● Friendly farewell
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 8
9. Signs of social attitude
in the prosody
Praat functions to extract features related to:
● variation of the fundamental frequency
● variation of energy
● variation of harmonicity
● Spectrum Central Moment, Standard Deviation, Gravity
centre, Skeweness and Kurtosis
● speech rate
Classification (using NNge algorithm) of user's spoken
sentence into 3 classes: positive, negative and neutral.
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 9
10. Signs of social attitude
in the gestures
Gesture recognition performed using
Microsoft Kinect + KinectDTW
need to consider only a subset of gestures compatible
with the nodes in
the skeleton that
the Kinect SDK
can detect.
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 10
11. Signs of social attitude
in the gestures
Signal Possible meaning(s)
Crossed arms Defensiveness, closure
Gripping own upper Insecurity, closure
arms
Adjusting cuff, Nervouseness,
watchstrap, tic, using negative attitude
an arm across the
body
touching or scratching Nervouseness,
shoulder using arm negative attitude
across body
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 11
12. Evaluation
Collecting a corpus
We collected moltimodal dialog moves, consisting in
linguistic, acoustic and gesture data.
Participants: 2 groups of 5 italian students aged between
16 and 25 (equally distributed by gender).
Goal: getting information about a correct diet in order to
stay in shape.
PCA role: nutrition expert.
Collected: about 300 moves.
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 12
13. Evaluation
Collecting a corpus
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 13
14. Evaluation
Results
Move U6 Move U7
Move U8
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 14
15. Conclusions
Existing model for recognising social attitude enriched
with the analysis of signals regarding non-verbal
communication: prosody and gesture.
We propose an extension of the multimodal analysis to
gesture modeling, according to the meanings that
psycholinguistic researchers attach to gestures in
conversations.
Preliminary experiments show promising results.
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 15
16. Future works
● Improving gesture recognition since the new Kinect
should allow for a better hand recognition
● extending the social attitude analysis with facial
expressions
Carrying out more evaluation studies in order to test the
robustness of our framework:
● for social attitude recognition in different scenarios
● with respect to different interaction modalities with both
ECAs and Robots
Recognising the Social Attitude in Natural Interaction with Pedagogical Agents
B. De Carolis, S. Ferilli, N. Novielli, F. Leuzzi, F. Rotella 16