Slides CapTechTalks Webinar March 2024 Joshua Sinai.pptx
Orchestrating collaborative technology-enhanced ecosystems: How to support teachers with conceptual and technological tools
1. University of Sydney
COCO-CHAI-LATTE Joint Seminar
Orchestrating collaborative
technology-enhanced ecosystems:
How to support teachers with conceptual and
technological tools
Prof. Yannis Dimitriadis
GSIC/EMIC research group
University of Valladolid
May 30, 2012
Based on work by L.P. Prieto – S. Villagrá
3. Outline
s The two hats of this seminar
s Hat 1: Conceptual tools
s A framework on orchestration
s Multi-level patterns as mediating tools
s Hat 2: Technological tools for
s Learning design
s Flexible deployment
3
4. The two hats of this seminar
s Personal
s Engineering background + Interest in education
s Audience (and associated drinks …)
s CHAI-LATTE and CoCo
s TEL research
s sound pedagogy + efficient engineering
s Quest for sustainable innovation
s good research for understanding, abstraction and
evidence
s effective support to actors – teachers, learners or
even service and technology providers 4
5. The two hats of GSIC/EMIC
s “Dark and bright side” in a group since 1994
s Focus on CSCL: complex and innovative
s Support real-world actors and own action-research
s Inquiry cycle (design-enactment-evaluation) for
s teaching/learning and technological support
5
6. Hat 1: Conceptual tools
s Analyze and understand the field of orchestration in
complex TEL ecosystems
s The “5+3 aspects” framework for orchestration
s Collect and use knowledge in terms of good
practices during the CSCL cycle
s Multi-level patterns
6
7. Orchestration: Moving target
s Increasing presence in literature, but what is it
about? Is there a new focus?
s Efforts to define the field/concept/metaphor:
s Stellar NoE deliverable, CSCL 2011 and ICLS
2012 workshops, collective paper
s New proposal for “5+3” orchestration framework
and validation through expert panel 7
8. Focus of orchestration on …
s Acknowledge complex ecosystems for innovative
TEL/CSCL environments
s Support teachers as orchestrators (and think of how
to minimize-distribute the orchestration load on
teachers, learners and systems)
s Build on pragmatism and minimalism as keys for
sustainability
s Explicit emphasis on how to flexibly manage
(learning time) enactment
s Design for adequate orchestrating and orchestrable
technologies (and conceptual tools)
8
9. The “5+3” framework
A diagramatic view
http://prezi.com/aa2vighak7hh/orchestration-in-tel-cscl-as-easy-as-53/
9
11. Evaluating the framework
The setup
s Research questions on
s Completeness of common understanding
s Usefulness for researchers
s Two panels (March – May 2012)
s Pilot phase: 22 Spanish researchers of low-
medium expertise (2.58) and more technological
profile (16T-7E-1M)
s International panel: 24 researchers of medium-
high expertise (4,46), high research experience
in TEL (15,8 years), more balanced profile
(6T-11E-7M)
11
12. Completeness of framework
s Quantitative and qualitative data
s Non-normative and inclusive
s Need to clarify role of actors, technology, theory
s More than 4,5 (Lickert scale of 1-6) for all items
s Logical: 4,78 –Comprehensive: 4,52 –Relevant: 5,13
12
13. Usefulness of framework
s To be referenced in papers, checklist and didactic
tool for young researchers
s But need for normative advice and real-world
examples
s Will be used in future (4.43)
s Provided new insights (4.17)
13
14. Multi-level patterns
s But the “5+3” Orchestration framework is a
conceptual tool mainly for researchers
s Tools (conceptual or technological) for orchestration
should also support “teachers”
s Exploiting articulated (design) knowledge
s (Pedagogical) patterns as mediating tools regarding
various orchestration aspects
s Design and Management of learning flow,
Assessment, Adaptation …
s Related fields of
s Learning Design, Educational Modeling, Scripting
14
17. Usefulness of patterns
s Are these patterns useful for practitioners?
s Evidence for learning flow patterns for OER
repurposing to collaborative activities
s Appropriation in University (Higher Education –
HE) teachers (Professional Development - PD)
workshops
s But not always
s If not included in appropriate technological tools
s Too complex (abstract) for (Primary Education –
PE) practitioners
s Do not take into account enactment issues
17
18. The need for atomic patterns
s Two independent studies by GSIC and SRI using
GroupScribbles in Primary and Secondary Education
schools (2008-2011)
s Studying enactment and disciplined improvisation
s Showed that other small-scale, informal, contextual,
actionable patterns emerged
18
21. Atomic patterns
as mediating conceptual tools
s And we need to “fill the gap”, “flesh the bones” with
appropriate mediating artifacts (conceptual tools)
21
22. Evaluation of atomic patterns
in PD workshops
s Four (2 in PE, 2 in HE) PD workshops (2010-2011)
s WS4 data quantitative data (scale 1 to 7):
s Enactment patterns useful (5,31)
s Feasibility (5,22) and nearness (6,06)
s 19 out of 22 teachers enriched design
s Useful but not revolutionary
s Especially important for non-experienced or pre-
service teachers
s Need to condense and classify long list of AP
s What about inclusion in technological tools?
22
24. Hat 1: Wrap-up
s Knowledge regarding orchestration for researchers:
s “3+5 Aspects” framework
s Approach for multi-level patterns
s Atomic pattern catalogue and LD representation
s Support teachers too with conceptual tools:
s Classroom norms for participation, discourse
moves for discussion, decision rules for
contingent teaching (SRI)
s Multi-level patterns
s Use in PD workshops Role playing and learning
design enrichment using cards (GSIC) 24
25. Hat 2: Technological tools
s But TEL is not only about
s Conceptual tools (patterns, moves, …)
s It is also about
s enacting learning in TEL complex ecosystems
s especially in HE distributed learning
environments where VLE/PLE are combined with
third-party tools
s And also support teachers (and learners) with
technological tools when
s orchestrating TEL/CSCL ecosystems (less
orchestration load for teachers – more for tools)
25
26. Learning Design tools
s Articulate/exploit the knowledge of patterns and
support the learning design process
s Create a model or representation of a learning
design (unit of learning, script, scenario, etc.)
s That allows understanding and reflecting on the
pedagogy, making the learning design explicit,
sharing it with others
s Through the Web Collage tool
s Or other Learning tools by the Stellar Learning
Design Grid Theme Team
http://www.ld-grid.org/home
26
27. Web Collage:
Learning & Assessment flows
Students Teacher
Select and work
Support individual
individually in a
work
subproblem
Write activity input for… Review
report report
Work in expert
Work expert
groups of students
groups students provide Support expert
with the same
with same feedback… group work
subproblem
subproblem
Form Jigsaw groups
and explain each Support jigsaw
other the different group work
subproblems
27
http://pandora.tel.uva.es/wic2
29. Web Collage:
Evolution and evaluation
s Evolution of the Collage tool since 2005
s Towards a more stable and user-friendly Web-
based tool that includes learning and assessment
patterns, as well as support for instantiation
s Extensive use and evaluation using mixed methods
s Multi-case studies in authentic settings and PD
workshops with HE teachers
s Wide acceptance of the pattern-based design
process
29
30. The deployment gap
s But teachers wanted to complete the cycle
s deploy their design in their TEL environment,
especially their institutional VLE (e.g. Moodle)
30
31. Alternatives for deployment
s Have the learning design “in mind” and then go
through the VLE/PLE interface and struggle to
“translate” the design to a course/lesson plan
s Use a single system (LAMS) to do both design and
deployment in the same environment
s Use an LD Tool (LDSE) to create the interpretable
script and then deploy it to one environment
s But there are many LD tools and VLEs:
s Provide intermediary to connect them: Glue!-PS
31
34. Glue!-PS
Validation
s Proof of concept validations using the Planet Game
(ICALT 2006) scenario
s Two LD languages: LDL + IMS-LD
s Two Learning Environments: Moodle +
MediaWiki
s Design and deployment
s by GSIC researchers of 36 learning designs
created using pen and paper at two HE PD
workshops (Sep-Nov 2011)
s by 24 teachers in a follow-up PD workshop (Feb
2012)
34
35. Glue!-PS and atomic patterns
s Atomic patterns are especially important for flexible
orchestration
s University teachers suggested embedding atomic
patterns in technological tools
s Initial evidence of use in Glue!-PS
s Implementation of a subset of atomic patterns
without architectural changes as e.g.
s AP: Spontaneous use of additional IT
s Actions: Add and synchronize in real-time
resources in an activity
s Feasibility evidence in 2 cases (Mar-Apr 2012)
35
36. Hat 2: Wrap-up
s Technological tools to support orchestration
s Patterns (that reflect good educational practices)
guide a flexible design and deployment process
s Glue!-PS allows to bridge the deployment gap in
distributed learning environments
s time-efficient and less error-prone
s allows sustainable migration to new VLE and
connection to various LD tools
s More naturalistic evaluation is necessary
s Important role of PD workshops for innovation and
feedback to research 36
37. Hat 2 (cont.): More tools to …
s Regulate through monitoring and the detection of
critical aspects of the learning designs (GLUE!-CAS)
s Find and select appropriate tools and include them
in learning designs (SEEK)
s Integrate third-party tools in learning designs
directly through the VLE interface (GLUE!)
s Design evaluation of CSCL case studies and
generate multimedia reports (CSCL-EREM)
s ….
37
38. Hats 1 and 2: wrap-up
s Support bits and pieces but also the complete cycle
of orchestration
s flexibly and sustainably
s Design for orchestration to show added value of
TEL
s innovative pedagogies using old and new
technologies
s Integrate conceptual and technological tools for
s researchers, teachers and students
38
39. And many open questions
s Do researchers need an orchestration theory and
framework and should it be normative?
s Is the proposed approach for pattern elicitation
sustainable? (crowd-sourcing ….)
s How can we support co-orchestration with students
and integrate efficiently PLE?
s How various (Web, 3D worlds, augmented physical)
spaces can be orchestrated?
s Is there a way to solve the ethical and security
problems that are involved?
s …
39
40. Some links
s Publications
s http://www.gsic.uva.es/public.php?lang=en&list_pu
s GSIC/EMIC channel at YouTube:
s http://www.youtube.com/user/gsicemic?
feature=watch
s WS on orchestration at ICLS2012 (July 3)
s https://sites.google.com/site/iclsorchestration2012/
40
Notes de l'éditeur
Special thanks to Luis Palblo Prieto (doctoral student) and Juan Ignacio Asensio (co-director), Sara Villagrá (doctoral student) and Iván Manuel Jorrín (director) from the Technology and Education “divisions” of GSIC respectively.
Complex TEL/CSCL ecosystems Multiple tasks activities, multiple social levels, multiple tools How can we manage them and survive (ina sustainable way!), especially the teachers?
Technology-enhanced classrooms or distributed learning environments are becoming increasingly complex ecosystems, which may include VLEs, Web 2.0 tools, 3D worlds, or tangible artifacts, either digital or not. In CSCL environments, multiple social planes are involved in multiple teaching and learning activities that may take place using multiple devices or tools. Orchestration of such collaborative ecosystems is challenging, since it may require creation and flexible deployment of learning designs or real-time class monitoring and management. This seminar presents a view of conceptual and technological tools that may support teachers in orchestrating CSCL classes. Different aspects of orchestration will be discussed focusing on design and atomic enactment patterns, as mediating conceptual artifacts. On the other hand, various technology-based tools will be presented that enable the teachers perform the complete orchestration life cycle. The presentation will draw on data from two years of field studies in a primary classroom, as well as professional development workshops and field studies with university teachers, where the conceptual and technological tools have been employed.
Cite main people and proposals, especially Dillenbourg Mention here and in the beginning the work by Luis Pablo and his publications and thesis (and Sara, etc.)
Cite main people and proposals, especially Dillenbourg Mention here and in the beginning the work by Luis Pablo and his publications and thesis (and Sara, etc.)
Will be used in future: 4.43 (international) – 4.40 (spanish) Provided new insights: 4.17 (international) – 3.36 (spanish)
CLFP and assessment patterns
Related and structured according to the models, maps or pattern languages
Learning Design composed of JIGSAW learning flow pattern And the REPORT REVIEW and OBSERVATIONS OF COLABORATIVE WORK assessment patterns
From 2 LD formalizations: one widely used (IMS-LD) and one specialized/higly expressive (LDL) ... ... to 2 different LEs: one widely used (Moodle) and one peculiar/lo-fi (MediaWiki) GLUE!-PS WORKSHOP EVALUATION * 24 teachers, multi-disciplinary, most had experience with CL, patterns, routines... from a previous workshop * Tried both deployment and (some) real-time flexibility (but Moodle is limited in this regard!) * Questionnaire responses indicated overwhelmingly positive feedback from respondents (21/24): all were able to deploy, only 2 could not deploy their own design (because of lack of time) ** Very good ratings for usability (in a 1-8 scale, WebCollage 6.76, GLUEPS 7, Moodle 6.47!), usefulness (7.38), time-efficiency (7.14) ** Good ratings for immediate future use (in a 1-8 scale, 6.19), still some reluctance due to beliefs (do not like CL, or prefer pen/paper), or lack of adequate support * Mini-focus groups (debates) indicated very positive feedback, with a few "buts": ** using CL (independent of CS) would depend very much on the context (subject, number of students, own expertise), sp. since current education/evaluation paradigm is very individualistic ** positive feedback on GLUEPS (intuitive, flexible), but they need more technical support (human resources, manuals, courses...) and reliability of ICT tools ** Some other well-known issues: some teachers still think that ICT is more complex than social/pen&paper, to do basically the same things, students sometimes prefer the individual/IRE mode
ATOMIC PATTERNS AND GLUE!-PS MAIN IDEAS * Of the original 169 atomic patterns in the catalogue, 21 were selected for eventual implementation (on the basis of: the AP being implementable through technology, having APs from all orchestration aspects, APs that appear frequently) * From these 21, GLUEPS functionalities have been abstracted, and generated an implementation roadmap * The implementation has shown certain architecture/system issues (e.g. having an API favors flexibility in real-time, synchronizing the VLE and GLUEPS/GLUE is difficult) IMPLEMENTATION * 8 of them were supported &quot;by design&quot; Monitoring the task On -the- fly assessment (Monitoring + Formative Assessment) Use results of a task in a different task / Reuse generated artifacts Teacher as participant in student groups Teacher chooses tools Use Moodle to automate submissions Use wiki to do collaborative writing Use wiki to structure activities * the next 5 most common have been implemented Strategies to group formation / Form groups Reform groups in face of the current attendants (only in MW) Spontaneous use of additional ICT (only in MW) Successive deployment of activities (only in Moodle, for now) Use pre-existing groups (only in Moodle) * 1-2 more are being implemented till the end of the thesis Peer review Adding improvised activities TRIED WITH SARA AND BEA * Authentic experiences, one with Moodle, one with MW * TICs-Sara2 (WebCollage -> MediaWiki, jigsaw pattern) ** Real-time changes in groups (many!) were needed. The teacher chose to make changes manually (it was feasible because it had no external tool), because the changes would have overwritten manual changes that she did after deploying ** Underlying issue to take into account in DLEs: state synchronization among elements, in both ways (VLE <-> GLUEPS, GLUElet Manager...), which is not trivial without modifying the VLE ** The teacher tried the real-time changes afterwards and valued GLUEPS very positively ** Still, a lot of time was needed to make sense of the (large) jigsaw and seeing if all instrinsic constraints were complied with. GLUEPS does not help with that * OyT-Bea (Pattern Collector -> Moodle, non-CLFP role playing) ** Integrate the design into an existing, ongoing Moodle course. Large number of groups/students (16 groups, around 60 students) ** Slight usability problems, resolved as the experiment went on ** Incremental deployments, to help with contingent elements of the design (i.e. materials/resources that could not be known in design-time) ** Teacher valued highly the usability enhancements, but... ** ... instantiation of groups and resources still took a lot of time. More automation/intelligence needed! (e.g. we will implement the peer review atomic pattern, and tell Bea to do it again) *** social coordination is still a desirable option, if you are willing to decrease usability for students a little bit (e.g. put all links to resources in the moodle, and tell socially who reviews what - instead of hardwiring the data flow as GLUEPS does). However, in larger groups social coordination might not be feasible...
We begin to bridge the deployment gap in DLEs - time is saved and the process is less error-prone ** ... but the real time savings is in sharing/reusing (e.g. deploy a design in 8 minutes when reusing, vs. in 48 min without GLUEPS) GLUEPS makes possible the deployment of LDs in DLEs which teachers thought were not feasible ** But still...*** ... average teachers (e.g. in our PD workshops) are reluctant to rely so much on ICT (they need support, help, tutorials, time to try...) also a hint for managers/policy-makers *** ... many operations in instantiation are repetitive (e.g. peer reviews con Bea) - there is still room for improvement by automating * More naturalistic evaluation needed (up to now, all cases were &quot;special teachers&quot;)! ** Highlight the role of PD workshops to disseminate and get feedback from &quot;more real&quot; people... and &quot;capture&quot; them for experiments ;)
From 5+3 ** do we (or do researchers) really want orchestration to be defined? do we want an &quot;orchestration theory&quot;? from our study, not everybody does! ** descriptive frameworks are fine, but we also need a more normative frame, for concrete advice: is it possible to give advice, regardless of context? can atomic patterns be the beginning of an answer? Are we really sustainable? ** Atomic patterns are &quot;expensive&quot; to elicit, and are quite contextual... could we &quot;crowdsource&quot; it to e.g. teachers? (but let us remember the failure of Crea-TIC - what is in it for the (average) teacher?) ** The GSIC technological tools currently put a big orchestration burden/responsibility on the teacher... could we co-orchestrate with students (e.g. teacher defines high-level structure, students choose the tools and other fine-grained stuff) - but still maintaining teacher control somehow (e.g. for monitoring)? * Security, ethical questions... where is the data? is it protected? (e.g. CSCL-EREM tool does not want your data, only links to it) permissions across this complex service ecosystem... (e.g. see the problems GLUE is having with implementing supposedly standard stuff like Oauth) * To which extent our current CSCL focus can be extended to more integrated curriculum (which includes, IBL, even more IRE stuff...)?