Keynote talk at INCOS 2010 Analysis of interaction in collaborative activities: the Synergo trail It provides background information on Synergo a collaborative learning environment more at hci,ece,upatras.gr/synergo

1. INCoS 2010 – Thessaloniki November 24th
Analysis of interaction in
collaborative activities:
the Synergo trail
Nikolaos Avouris
University of Patras, GR
Keynote Talk
1/60

2. outline
- on analysis of collaboration
- the synergo testbed
- synergo studies
- models from synergo data
2/60

3. On analysis of
collaborative
activities
3/60

4. Typical analysis objectives
method focus
Usability evaluation Collaborative technology
Pre-post testing Learning outcomes
Quantitative, qualitative,
sequential methods Interaction process
Inquiry methods Participant’s perceptions
4/60

5. Focus on the
interaction process
– Dillenbourg: “the basic instrument for
understanding collaborative learning is
understanding the interaction that takes
place during a learning process”
– Koschmann: “CSCL research is not focused
on instructional efficacy, but it is studying
instruction as enacted practice”
5/60

6. Quantitative analysis
• Frequency counts of events such as:
- messages posted per student per period of time
- hits on particular discussion forum pages
- actions taken on objects of a shared workspace
- number of files read in a shared file system etc.
• Defining metrics (indicators) that combine
different kinds of frequency counts
• Suitable for all kinds of collaborative learning
• They can lead to models of interaction (e.g.
Social Networks etc.)
6/60

7. Qualitative content analysis
• “Content analysis refers to any process
that is a systematic replicable technique
for compressing many words of text into
fewer content categories based on explicit
rules of coding” (Kripendorf, 1980)
• Suitable for every means of dialogue
oriented collaborative learning
(synchronous & asynchronous, collocated
& distant)
7/60

8. Content analysis models
• Henri’s scheme
• Garrison’s model
• Gunawardena’s Interaction
Analysis Model
• Language/action OCAF
8/60

9. Content analysis resources
• The content analysis
guidebook
http://academic.csuohio.e
du/kneuendorf/content/
9/60

10. Small group synchronous
interaction: Integration of
dialogue and action
• Treats language acts and actions taken to objects
in an integrated way
• Uniform annotation (eg. the OCAF framework)
• Shifts the focus to the objects of a shared
workspace
• Objects have an ‘owner’ just like language acts
• Can visualize uptaking actions (Suthers 05)
10/60

11. Dialogue: Chat tool affordances
• Visual and/or auditory cues are not available
• No production blocking->overlapping exchanges
• Persistence of messages – substantiation of conversation
• Loose inter-turn connectedness - but possibility of
simultaneous engagement in multiple threads
• Verbal deixis spans throughout the whole history of
dialogue (no restricted time window is adequate for
analysis)
• Posters may reply rapidly, using short messages and split
long messages to increase referent/message coherency
(Garcia and Jacobs 1999)
• Participants begin new topics fairly much at will in a
manner that would not happen in a formal face-to-face
group discussion (O’Neil & Martin, 2003)
11/60

12. Action: Shared Activity space
affordances
• Feedthrough (Dix et. al., 1993)
• Various degrees of coupling (Salvador
et. al., 1996)
• Workspace can be used as an external
representation of the task that allows
efficient nonverbal communication
• Workspace artefacts act as
conversational props (Hutchins, 1990)
12/60

13. Types of communication acts /
gestures in shared workspace
• Deictic references
• Demonstrations
• Manifesting actions
• Visual evidence
(Gutwin, Greenberg, 2002)
13/60

14. Grounding through actions on a
workspace representation
(Suthers, 2006)
Sequences of actions :
(1) one participant’s action in a
medium…
(2) is taken up by another participant
in a manner that indicates
understanding of its meaning, and
(3) the first participant signals
acceptance
14/60

15. Merging Action and dialogue
Annotated model=collection of
objects (OCAF Avouris et al. 2003)
MEF = {
Entities= E (ABC) = 1/EP, FA , EI
E (VELO) = 2/ EP, FA , EI
E (TRUCK) = 3/FP, FI
E (STOREHOUSE) = 4/FP EC, FA, FI
E (STORE) = 5/FP EC, FA, FI
Ε(DELIVERY)= 11/ FP, EX, FI
Relations= R (VELO-owns-SH) = 9/FPI
R (VELO-owns-ST) = 10/FPI
R(TRUCK-transports- DELIVERY)=17/ EP, FI, EC A (storehouse) A (store)
R(SH-are-suppplied-by-TR) = 18/ FIM A(id)
7/EP, FC
R (ABC-owns-TR) = 25/ FPI 6/EP, FC 24/ FI
A (truck)
R(ST-owns-SH) = 24/ EP FP FI EC, EM
8/FP, EX
R (ABC-owns-TR) = 25/ FPI R
E(STORE-
E(VELO) HOUSE)
Attributes= A (DEL.id) = 13/FIM
2/EP, FA , EI 9/FPI 4/FP EC, FI
A (DEL.volume) = 14/FIM
A (DEL.Weight) = 15/FI E(ABC)
A (DEL.Destination) = 16/FI R R R
1/EP, FA , EI
A (TR.Max_Weight ) = 19/FI 18/FIM
R
10/FPI 24/EP FPI, EM R
A (TR.id ) = 21/EP , FI 25/FPI
12/EP, FR
A (TR.Journey_id ) = 23/FI
A (TR.volume ) = 20FIM E(DE-
E(STORE) E(TRUCK) R LIVERY)
A (SH.id ) = 24/FI 5/FP , EC, FAI 17/EP,F I,EC
EP, FI 11/FP, EX, FI
Items not in the final solution
-R (SH-DEL) = 12/EP , FR , A(Max_ A(Journey A(Id) A(volume)
-A(VELO.Storehouse)=6/ EP , FC weight) _id)
19/FI 23/FI 13/FIM 14/FIM
-A(VELO.Store)= 7/ EP , FC
-A(ABC.Truck)= 8/ FP , EX A (max-
A(volu
journeys/week A(id) A(Weight A(destina
-A (TR.max_journeys_per_week) = 22/EP , FR } me) tion)
22/EP, FR 20/FI,M 21/EP, FI 15/FI 16/FI
15/60

16. Synergo
Chat
Avouris et al. 2004 Act
hci.ece.upatras.gr/synergo 16/60

17. Synergo Partner
selection
tool
Drawing
objects
libraries
Chat
tool
Shared
Activity
Space
17/60

18. Synergo Drawing libraries
Concept maps
Entity-Relationship Diagrams
Flow charts
Free Drawing
18/60

19. Activity logging
used for :
• Build a history of interaction at server
• support latecomers during synchronous
collaboration
• analysis and playback of the activity
•Support replication/ reduce bandwidth
requirements
19/60

20. Analysis tools
20
20/60

21. Log Data Preprocessor
Analysis
tools
21
21/60

22. Typed events automatically
annotate the diagram
E i = (t i , Aa , [O o ], [Tt ])i
Object A
I C M R
Actor A
Actor B
Actor C
Types of events
I (Insert),
M (Modify),
D (Delete)
C (Contest)
22/60

23. Playback of annotated view
23/60

24. What about the chat? Can we
annotate chat automatically?
One approach is to ask the user
to do it - open sentences
(e.g. Epsilon (Soller et al. 97)
24/60

25. Annotation of chat events
Deleted objects
(b)
Model objects
Dialogue messages
Abstract objects
25/60

26. Define types of actions
(annotation scheme)
26/60

27. Overview: Visualization of
logged actions
27/60

28. Teachers view and tool support
• E. Voyiatzaki, M. Margaritis, N. Avouris,
Collaborative Interaction Analysis: The teachers'
perspective, Proc.ICALT 2006 - The 6th IEEE
International Conference on Advanced Learning
Technologies. July 5-7, 2006 – Kerkrade ,
Netherlands, pp. 345-349.
28/60

29. Teacher support (supervisor tools)
29/60

30. Study of the use of tools by teachers
Level of Education Computer Engineering University degree
program (A’ Semester)
Teachers involved 1 Teacher + 5 Teaching Assistants
Learners involved 80 students
(46 students 2004-2005,
34 students 2005-2006)
Collaborative Activity Problem solving activity: Development and
Exploration of an Algorithm
Students in Dyads , no roles assigned
Typical Laboratory lesson (2 didactic hours)
Collaborative Tools SYNERGO Collaborative Environment
SYNERGO Analysis Tools
30/60

31. The Teachers Used the proposed
views and gave feedback…
Quantified
Overview:
Class and
Group
teacher
The Process Teachers: “The
View
(Playback
process view is
of the the most
activity) important tool
for in depth
insight .”
Qualitative
view
researcher
Row
data
31/60

32. studies
Vrachneika Gymnasio-
3rd year
UnivPatras Algorithms
32/60

33. Typical tasks
- Collaborative Cognitive Walkthrough
of an interactive system
- Designing Data bases (ER-D)
- Building and exploring Flow Charts
33/60

34. Joint Univ Patras -
UnivDuisburg croos-national
collaborative activities
(2004-2005)
• A. Harrer, G. Kahrimanis, S. Zeini, L. Bollen, N. Avouris, Is there a
way to e-Bologna? Cross-National Collaborative Activities in
University Courses, Proceedings EC-TEL, Crete, October 1-4,
2006, LNCS vol. 4227/2006, pp. 140-154, Springer Berlin
34/60

35. Similar models with different
tools (Synergo, Freestyler)
35/60

36. Findings of the Patras-Duisburg study
• Mixture of synchronous and asynchronous
approaches.
• Only partly use of the provided tools
• Engaging activities - examples of sessions
of many hours (4-5 h) in joint activity and
discussion
• Innovative use of media and coordination
mechanisms
• Good strategies for division of labor
• Excellent social dynamics and group spirit.
36/60

37. A distance learning course
of Hellenic Open University
(HOU) (2003-2004)
M. Xenos, N. Avouris, D. Stavrinoudis, and M. Margaritis,
Introduction of synchronous peer collaboration activities in
a distance learning course, IEEE Transactions in
Education, vol. 52 ( 3), Aug. 2009, pp. 305 - 311,
37/60

38. Mixed media and collaboration approaches
Asynchronous group activity
Post
assignments, Respond to technical and
form groups organizational problems –
follow activity
Tutor ODL Server
(forum, exchange of material, ODL
repository
help desk)
Asynchronous interaction
Submit final Facilitator
Activity Record
solution Synergo activity
server
logging
Synchronous
Synchronous
activity
interaction (share
Synergo drawing / chat Synergo
client communication) client
Student #1 Student #2
Arrangements on
sessions plan-
direct contact Group
38/60

39. Synergo- Discussion forum
39/60

40. Findings of the HOU study
• Infrastructure overhead higher than
expected (unforeseen technical
problems)
• Peer tutoring patterns emerged in
higher degree than younger students
• Multiple media engaged
• Strong social aspects of community
building
40/60

41. Study on Mecahnics of
Collaboration:
Coordination protocol
200
180
160 GROUP A (with key)
GROUP Β (without key)
Number of events
140
120
100
Group A Explicit floor Group B No floor control: 80
control: Only the key all partners can act in the 60
40
owner can act in the shared shared work space
20
work space
0
Critical Insert Delete Move Chats
T+ T- Type of events
41/60

42. Findings of the study
§ Explicit floor control of the shared activity area
did not inhibit problem solving
§ Similar patterns of activity in both groups.
§ group T- was more active than T+
§ T+ students have been obliged to negotiate on
possession of the activity enabling key and thus
argue at the meta-cognitive level of the activity
and externalise their strategies, a fact that helped
them deepen their collaboration
42/60

43. models
43/60

44. #1 Support for Group
Awareness through a Machine
Learning Approach
Train a classifier to be used for estimation of
the quality of collaboration using historical
data of problem solving activities of
students engaged in building concept maps
and flow-chart diagrams in Hellenic Open
University and University of Patras
M. Margaritis, N. Avouris, G. Kahrimanis, On Supporting Users’
Reflection during Small Groups Synchronous Collaboration, 12th
International Workshop on Groupware, CRIWG 2006 Valladolid,
Spain, September 17-21, 2006, LNCS 4154
44/60

45. Logfile segmentation
L={S1, S2, … Sk}
NE
quality of
collaboration
per segment
(bad, average,
good)
45/60

46. Correlation based feature selection
(CFS) for different segment sizes
NE=60 NE=80 NE=100 NE=200
(2) num_chat (2) num_chat (2) num_chat (2) num_chat
(3)symmetry_chat (3)symmetry_chat
(4) altern_chat (4) altern_chat (4) altern_chat (4) altern_chat
(5) avg_words (5) avg_words (5) avg_words (5) avg_words
(6) num_quest (6) num_quest (6) num_quest
(7) num_draw (7) num_draw (7) num_draw (7) num_draw
Correlation based Feature Selection (CFS)
NE= number of
technique:
events per segment
makes use of a heuristic algorithm along
with a gain function to validate the
effectiveness of feature subsets.
46/60

47. Performance of classification
algorithms
• Naïve Bayesian
Network 90
• Logistic Regression
Success rate (%)
85
• Bagging NaiveBayes
Logistic
80 Bagging
• Decision Trees SimpleLogistic
RandomForest
NNge
• Nearest Neighbor 75
60 80 100
Fragmentation factor NE
200
47/60

48. Visualization of group
awareness indicator
State of
Collaboration
48/60

49. Evaluation study
• 11 groups of 3 students each were given a
collaborative task.
• 6 of these groups were provided with the group
awareness mechanibsm.
• 5 groups did not have that facility
• The mean values of collaboration symmetry
were significanlty different between the two sets
(p=0,0423).
49/60

50. Side-effect
• in four (4) out of the six (6) groups there
was an explicit discussion about the group
awareness mechanism.
• A side-effect:
50/60

51. #2 Measuring quality of
collaboration in Synergo
activities using a rating scheme
and an automatic rating model
Based on:
Meier, A., Spada, H., & Rummel, N. (2007). A rating
scheme for assessing the quality of computer-supported
collaboration processes. International Journal of
Computer-Supported Collaborative Learning, 2, 63–86.
51/60

52. Meier et al. (2007) rating scheme
Original setting New setting
Desktop-videoconferencing Synergo: shared whiteboard
CSCL tool system with shared text and chat
editor
Domain Medical decision making Computer programming
(algorithm building)
Collaborators Intermediates; Beginners;
complementary prior similar prior knowledge
knowledge (psychology and
medicine)
52/60

53. Meier et al (2007) rating scheme
dimensions
53/60

54. Kahrimanis et al. (2009) adapted
collaboration rating scheme
Aspect of Dimensions
collaboration
Communication 1. Collaboration Flow
2. Sustaining Mutual Understanding
Joint information 3. Knowledge Exchange
processing
4. Argumentation
Coordination 5 .Structuring the Problem Solving Process
Interpersonal 6 .Cooperative Orientation
Relationship
Motivation 7. Individual Task Orientation (for dyad mean
or absolute difference) 54/60

55. Development of a Collaboration
Quality Estimation Model
Data set used
• 350 students of 1st year working in
dyads to solve an algorithmic problem
using Synergo (academic year 2007-
2008) duration of activity 45’ to 75’
• 260 collaborative sessions
• Grading according to the quality of
solution and quality of collaboration
55/60

56. 36 derived metrics used
(Kahrimanis et al. 2010)
56/60

57. Quality of Collaboration Estimator
(Kahrimanis et al. 2010)
Partial Least Squares Regression Model
collaboration_quality_avg =
0.460 + 0.004*C4 - 0.005*C6 + 0.011*C8_17.5 - 0.012*C7
+ 0.602*EV3 + 0.447*STC - 0.001*MW1 + 0.008*MW6
Observed vs. Estimated CQ average
VIPs (1 Comp / 95% conf. interval)
3
1.8
Observed (collaboration quality avg)
1.6
1.4 2
1.2
1 1
VIP
0.8
0.6
0
0.4 -2 -1 0 1 2 3
0.2
-1
Stone & Geisser 0
Coefficient
-2
Variable
(cross validation) Estim ated(collaboration quality avg)
57/60

58. Use of Quality of Collaboration
Estimator as discriminator
between cases of good and bad
collaboration
• The model scored between 76.6% to
79.2%, with the exception of one dimension
of lower quality.
58/60

59. Use of Quality of Collaboration
Estimator as automatic rater
• The model had acceptable performance
as rater as the inter-rater reliability with
human raters had the following values:
ICC=.54, Cronbach’s α=.70, Spearman’s
ρ=.62 acceptable for α και ρ (George, &
Mallery, 2003; Garson, 2009), not for ICC
(.7) (Wirtz & Caspar, 2002) . This applies
both for the average collaboration quality
value and the individual dimensions.
59/60

60. Current developments
• Study of tablet-based collaboration patterns
(synergo v. 5)
• Study of Attention mechanisms (Chounta et al.
2010)
60/60

61. More on Synergo:
hci.ece.upatras.gr/synergo
61/60

62. Some more key references
• Avouris N., Margaritis M., & Komis V. (2004). Modelling interaction
during small-group synchronous problem-solving activities: The
Synergo approach, 2nd Int. Workshop on Designing Computational
Models of Collaborative Learning Interaction, ITS2004, Maceio,
Brasil, September 2004.
• Κahrimanis, G., Meier, A., Chounta, I.A., Voyiatzaki, E., Spada, H.,
Rummel, N., & Avouris, N. (2009). Assessing collaboration quality in
synchronous CSCL problem-solving activities: Adaptation and
empirical evaluation of a rating scheme. Lecture Notes in Computer
Science, 5794/2009, 267-272, Berlin: Springer-Verlag.
• Kahrimanis G., Chounta I.A., Avouris N., (2010) Determining
relations between core dimensions of collaboration quality - A
multidimensional scaling approach, In the 2nd International
Conference on Intelligent Networking and Collaborative Systems
(INCoS 2010)
62/60