Up2U Data from Past Platform

1. DATA FROM PAST PILOTS
Part 2: Learning Analytics
(UROMA Sapienza - G. Toffoli and S. Lariccia)

2. A DEFINITION OF LEARNING ANALYTICS
Learning analytics​ (LA) is the measurement, collection, analysis and reporting
● of data about learners in their context,
● for purposes of understanding and optimizing
(a) learning
(b) the environment in which it occurs.
(definition adopted at the First International Conference on Learning Analytics and Knowledge, 2011)

3. SOME FEATURES OF LEARNING ANALYTICS
LA is concerned mainly with e-learning.
In particular, with online or hybrid environments, such as LMSs and MOOCs.
These can generate large amounts of data related to learning and teaching processes.
LA is also considered an extension and an application of web analytics.
LA addresses learning in formal and informal scenarios.

4. WHO IS LEARNING ANALYTICS FOR?
Today LA is one of the most popular keywords in the literature on education.
It can mean different things for different people categories.
Possible stakeholders: administrators, teachers, students, parents.
Macro, meso and micro levels of analysis.
Possibly we are interested to meso and micro levels.

5. WHAT IS LEARNING ANALYTICS FOR?
Anticipation of learner difficulties (predictive analytics)
Feedback on the individual learning process (assessment, recommendation, tutoring)
Feedback on the efficacy of teaching (at different levels)
● methodology, tools and materials
● teaching process
● context
Monitoring of compliance with planning

6. THE xAPI PROTOCOL
xAPI ​(the ​Experience API​ - old name: ​TinCan AP​I) is a protocol for the exchange of LA data. Its main
objective is interoperability
.
xAPI specifies how to
● send and store ​statements ​tracking learning related activity
● retrieve a collection of statements by performing a rough query.
Both functions are implemented by a specialized tool, being called ​Learning Record Store​ (LRS).

7. xAPI AS A SUCCESSOR OF SCORM
The xAPI protocol has been proposed as the successor of ​SCORM ​by ​ADL ​(​The Advanced Distributed
Learning Initiative​).
Like SCORM, xAPI is a protocol for tracking learning related activity.
The key difference is that xAPI allows you to track learning in almost any context, not just in the LMS.
For example, in informal learning and in team-based learning.
Besides specifying the “syntax” (general format) for tracing experiences, xAPI aims also to promote the
reuse of the terminology.

8. THE xAPI STATEMENT MODEL - THE BASIC TRIPLE
Statements are the vehicle by which “experiences” are captured.
The core of an xAPI statement is the ​actor​-​verb​-​object​ "triple".
I - did - this
Andy - Listened to - Great Expectations on Ebook
Where the ​actor ​could be an individual or a group.
Together, ​verb ​and ​object ​specify the “action” being performed.
Usually the verb is an English word identifying the action type in a very generic way, while the object can
represent an educational content, an artifact under construction, a physical object or place, a
performance, a social event, or even an individual or a group.

9. THE xAPI STATEMENT MODEL - OTHER COMPONENTS
A statement can include other attributes, like a timestamp and the accrediting authority.
The most interesting ones are ​context ​and ​results​.
The results could represent things like​ ​answers to quizzes and grading scores.
The context can put together a lot of data; currently we exploit mainly those modeling the ​Context
activities​, and specifically the ​parent ​and ​grouping ​sub-properties.
(see the paper “Anatomy of a Tin Can API Statement”:
https://xapi.com/wp-content/assets/e-book/Anatomy-of-a-Tin-Can-Statement-e-Book.pdf​)

10. THE POPULARITY OF xAPI
Under the pressure of some major actors in the LA field, xAPI has become almost a standard.
The main “competitor” of xAPI is ​Caliper​, from the ​IMS Global Learning Consortium​.
A working group bringing together people from ADL and IMS should study the possible “convergence” of
xAPI and Caliper, but they don’t seem very optimistic.
High level of interest for the compliance with xAPI can be found in the producers of tools for job training
and in large corporations.
Lower level of support of xAPI can be found in the developer and user communities of popular open
source VLEs such as Moodle and Open EdX.
We notice scarce awareness in the school world of the need for the interoperability of learning tools and
learning data; this may in turn depend on scarce attention being devoted to the exposition of the learners
to a variety of formal and informal learning situations, which could benefit most of interoperability.

11. LA and xAPI IN THE UP2U DOCUMENTS
Quotations from the Project proposal
● specific objective 1​ – “real-time assessment of interactive learning paths supported by learning
analytics”
● specific objective 2​ – “… Teachers will be able to assess the progress of the learner in real-time
and share data with the Learning Record Store, LMS ... via rich standard protocols (such as
OneRoster or the Tin Can API)”
● specific objective 4​ – “In addition, the student portfolio should be interoperable and should enable
novel analytics and visualisation of learning paths”.
The ​Workplan ​references learning data and TinCan API wrt: Task 4.1. (Application toolbox framework),
Task 4.3. (Tools for informal learning) and WP5, mainly Task 5.2. (Teacher skills development).
Deliverable 5.2.​ references LA and xAPI, in particular in sections: 3.6, 4.1.2, 4.2, Annex D, Annex E.

12. EXPECTED USE OF LA DATA FOR IMPACT EVALUATION
2 impact levels?
● impact of the whole project
● impact of the Pilot activities.
The use of LA data should complement other methods of assessment, including
● surveys answered by teachers and learners at different time steps
● focus groups.
(see Deliverable 5.4)

13. LEVELS OF ANALYSIS THAT SHOULD BE SUPPORTED BY LA
Monitoring of activity going on at different observation levels:
● class
● school
● district, ...
Analysis of the data according to different aggregation dimensions; e.g.
● formal and non-formal context
● more passive or active behaviour, ...
Identification of some activity patterns, especially in the case of collaborative creation of artifacts.
Relating user activities to learning progress.

14. LA SUPPORT IN THE NGDLE - THE LEARNING RECORD STORE
A common Learning Record Store (LRS) is provided by the NGDLE of Up2U.
Currently, it is an instance of Learning Locker (LL), “the world's most installed Learning Record Store”.
LL is a commercial product, but its community edition, which is open source, has been deployed.
A few shortcomings of LL have been pointed out, but it is quite a valuable tool and its deployment has
proved reliable.
(see the learning path "Learning Analytics in Up2U" in CS:
https://cs.up2university.eu/lp/learning-analytics-in-up2u/)

15. LA SUPPORT IN THE NGDLE - POSSIBLE TOOLS’ EXTENSIONS
Currently only Moodle, CommonSpaces and DSpace include xAPI interfaces.
The xAPI interface of Moodle (the xapi_logstore plugin) is largely incomplete and must be revised
substantially; this is important since it seems that, for several national pilots, we get LA data only through
Moodle.
Probably H5P is the Moodle extension whose xAPI support would add most value.
The OpenEdX MOOC platform, which is the other general-purpose VLS added to the NGDLE, currently
lacks an xAPI interface.
Moodle central and Moodle-Lithuania use two different xAPI "vocabularies": not an ideal situation.
(see Deliverable 5.4)

16. SUPPORT FOR LA IN THE NGDLE - MAJOR SHORTCOMING
Two major shortcomings in the xAPI support provided by the NGDLE have been posed to the attention
of the other Up2U work packages:
● the lack of a common way of uniquely identifying the user in the “actor” slot of the xAPI statements;
this makes very difficult to correlate activities done by the same user on different platforms; a
possible solution for this issue has been suggested
● the absence of information allowing to aggregate xAPI statements by user country and affiliation
and even to discriminate between users acting as teachers and learners; the best approach and
the cost of a solution for this issue are not clear; currently, this problem can be only partially
circumvented in very twisted ways.
Unfortunately, we cannot expect these problems to be radically solved in time for the next pilots.

17. LA SUPPORT IN THE NGDLE - VARIETY OF TOOLS FOR ANALYSIS
Learning Locker (LL) not only acts as a store of xAPI statements, but also provides tools for filtering
them and aggregating the data.
Analytics dashboards can be defined interactively in LL, through a web interface, by composing filters,
and a repertoire of visualization widgets.
LL dashboards can be published on the web and parameterized dynamically, to provide custom and
always up-to-date monitoring views.
LL supports also HTTP API allowing to perform more advanced and efficient data processing, using a
query and aggregation language based on MongoDB.

18. LA DATA COLLECTED FROM THE PILOTS - MAJOR SOURCES
Hundreds of schools in Up2U are networked based on xAPI and a central LRS, which is an open source
version of Learning Locker.
Sources of data include 2 instances of Moodle, Moodle plugins, and external Moodle extensions.
They include also CommonSpaces, a platform supporting project-oriented and constructive learning.
The picture of the situation is quite inhomogeneous. In short, currently we get, through an xAPI LRS, a
large and systematic set of LA data coming from Lithuania; some data of difficult interpretation, unevenly
distributed among schools and courses, from Greece, Italy, Portugal and Spain; no data at all from
Germany, Poland and Hungary.

19. LA DATA FROM THE PILOTS - THREE DISTINCT STORES
3 distinct stores (databases) have been created inside the unique installation of LL:
● learn.up2university.eu for the central Moodle instance
● vma.lm.lt for the Lithuanian Moodle instances
● cs.up2unuversity.eu for CommonSpaces
Since the beginning of this year, more than 1 million xAPI statements have been generated by the
Lithuanian pilots.They are making the most systematic use of Moodle: the number of statements being
sent daily by them to LL exceeds by more than one order of magnitude the total number of statements
being sent by all the other sources.

20. LA DATA COLLECTED FROM THE PILOTS - SOME EXAMPLES
For examples of "shared dashboards" published on web pages, outside the interactive interface of LL
itself, see:
● the homepage of "The community of the Up2U H2020 project" in CS (3 dashboards)
https://cs.up2university.eu/project/up2u/
● the homepage of the "Lithuanian Schools" community in CS (1 dashboards - see image below -
and 1 map)
https://cs.up2university.eu/project/lithuanian-schools/

21. FROM THE PILOTS - RECENT ACTIVITY IN LITHUANIAN SCHOOLS
In the Lithuanian pilot, the “actor” identifier includes a web address (the homepage of a virtual Moodle
instance) which is different for each school. This is just one of the many ways one could envisage to
trace the user affiliation; maybe, it wouldn’t be practical to extend a similar scheme to Moodle central.

22. FROM THE PILOTS - ACTIVITY IN SOME OTHER COUNTRIES
Here, the attribution of activities to different countries is only presumptive, based on a hand-made table
mapping courses to countries! We could share said table to let you amend and update it.

23. PARAMETERIZABLE DASHBOARD -
BAR CHARTS FOR A LITHUANIAN COURSE
For examples of "shared parameterizable dashboards" published on web pages, see:
● the unit "A few live examples of Dashboards" inside the learning path "Learning Analytics in Up2U"
​https://cs.up2university.eu/lp/learning-analytics-in-up2u
These 2 Visualizations include examples of verbs and activity types in statements generated by the xAPI
interface plugin in the Moodle instances of the Lithuanian pilots.

24. PARAMETERIZABLE DASHBOARD -
VERB CATEGORIES FOR A LITHUANIAN COURSE
In a tentative (arbitrary) way we defined 4 groups of xAPI verbs to categorize the actions being traced.
Here we see that in the Lithuanian pilots a significant share of the statements are related to assessment.

25. PARAMETERIZABLE DASHBOARD -
CORRELATION CHART FOR A LITHUANIAN COURSE
Only in this example we have been able to exploit a Visualization of type “correlation”. For less simple
cases computing correlations requires a more advanced query tool.

26. LL PARAMETERIZABLE DASHBOARD -
BAR CHARTS FOR A CS PROJECT
These 2 Visualizations include examples of verbs and activity types in statements generated by the xAPI
interface of CommonSpaces; this platform is more addressed to collaborative work.

27. LL PARAMETERIZABLE DASHBOARD -
VERB CATEGORIES FOR A CS PROJECT
We have categorized the actions being traced by CommonSpaces using the same mapping table as for
Moodle. Here we see that a significant share of the statements are related to user-generated content.

28. A VARIETY OF APPROACHES TO THE EXPLOITATION OF xAPI
DATA
The entry level for analysis of data is the definition and execution of queries, visualizations and
dashboards from the interactive browser interface of LL. Almost all charts shown above have been taken
from LL dashboards.
Pre-filtered and reformatted data can be exported in CSV for further processing with data spreadsheets
or more powerful applications dedicated to data analysis (please, ask Gytis for details).
Besides defining the structure and the semantics of a statement, the xAPI Specifications state that data
are transferred via HTTP requests and responses; they define several interfaces.
The ​Statements interface​ deals with the storage and retrieval of the learning records (from-to the LRS),
without addressing filtering strategies and performance
LL implements also the more advanced ​Aggregation interface​, which allows you to access ​MongoDB​’s
powerful ​Aggregation API​ for more custom filtration of statements; this is intended to provide a
communication channel of greater flexibility and higher performance.

29. EXPLOITATION OF THE DATA WITH JUPYTER
A wholly original experimentation being done in Up2U is the development of ​Jupyter Notebooks​ aimed at
analyzing the learning data by accessing the HTTP Aggregation API of Learning Locker.
Jupyter Notebooks, are supported in Up2U by the ​SWAN ​service, being provided by ​CERN​.
Jupyter is an ideal environment for training and self-study, since notebooks integrate text, code and
graphics.
Jupyter Notebooks are also a flexible solution for accessing LL through the HTTP API.

30. MY PERSONAL VIEW OF LA IN UP2U
In the Up2U project we had/have, to some extent, the opportunity to collect a consistent set of data from
an extended experimentation.
We evaluated a set of techniques for analyzing them and to present the results of the analysis.
We also planned a range of services that can be offered to school principals and teachers in order to
familiarize them with LA concepts and practices.
Since LA is a relatively new field - at least it was for us -, I think that we should follow in part a bottom-up
approach for introducing it in Up2U, in the belief that until you don’t have a variety of available data and
are able to experiment with analytics tools, it can be difficult to get insights on how to exploit them.
At the same time, we need to elaborate the methodology for using LA in effective ways, by matching it to
the design of learning scenarios and to the assessment needs, as far as this is permitted by the
limitations of the data collected and of the analysis tools.

31. DOING LA AS A SERVICE
In Up2U, LL queries and visualizations and, even more, Jupyter Notebooks for doing LA are developed
by expert users and are made available to others for monitoring and assessment tasks.
The service team can provide support for the definition of LL Analytics Dashboards on request of Pilot
managers, teachers' trainers and even willing and curious teachers.
Clever design of Analytic Dashboards can reduce their number, since they can be parameterized to
focus on specific courses or learners.
In general, Jupyter Notebooks are an ideal means to share ideas and experiences.
They have been used in the educational practice as a bi-directional channel for communication between
teachers and students, e.g. for delivering, discussing and evaluating assignments).
Besides being a flexible solution for accessing LL through the HTTP APIs, they can be used to share
questions, proposed solutions and results.

32. NOTEBOOK “ACTIVE COURSES ON MOODLE-PSNC” -
SORTED BY ACTIVITY LEVELS
This is the partial output of a notebook enumerating the courses defined in the Moodle instance shared
by Greece, Italy, Portugal and Spain, sorted by activity level; two partial counts are computed: ​blue​ bars
count traces of actions referring directly to a course, while ​orange ​bars refer to resources, forums, aso.
We aren’t yet clever in doing graphics in Jupyter; however this demonstrates that with notebooks it is
possible to do analyses that its is impossible or impractical to do with from the LL interactive interface.

33. NOTEBOOK “ACTIVE COURSES ON MOODLE-LITHUANIA” -
SORTED BY ACTIVITY LEVELS
This is the partial output of a notebook enumerating the courses defined in the Moodle instances of
Moodle-Lithuania. As in the previous case, the activity levels are made-up by 2 partial counts; our guess
is that entirely ​blue ​courses are “​sc​hool courses” (sc - maybe they support enrollment and
communication), while the others are “Disciplinary Courses” (DC).

34. NOTEBOOK “ACTIVE COURSES ON MOODLE-LITHUANIA” -
ANNOTATED WITH MORE DATA
This printout, from the same notebook as above, lists most “active” courses in the Lithuanian pilots,
being identified with their full “object ids” and their names; the other columns show the course type (first
11 are all “disciplinary courses”), number of actors involved, number of statements referring directly to
the object and total number of statements related to the course.

35. “ACTIVE COURSES ON MOODLE-LITHUANIA” -
SELECTED COURSES
This line chart, from the same notebook as above, shows the individual activity trends for the 3 most
active “​Disciplinary Courses​” and the 2 most active “​school courses​” of the Lithuanian pilots.
Obviously, the same tools used in this and in the previous slides could be used to analyze the activity of
individual learners and teachers, when this would appear meaningful on the basis of the available data.

36. “ACTIVE COURSES” AND “ACTIVITY LEVEL” -
WHAT THESE TERMS MEAN?
As in all this presentation, the notion of “activeness” is the result of many types of simplification, since
(just to mention a few of them):
● only one platform (e.g.: Moodle) with some plugins records actions in the LRS
● only a small subset of the action types are traced (e.g.: forum posts are not traced by Moodle)
● actions of different types could be traced at very different granularity, so that some types could be
weighted much less or much more than others; the effect of this could be alleviated by defining a
table of correction factors.

37. POSSIBLE OBJECTIVES TO BE PURSUED
● correlate individual activity with measured outcomes (concerning mainly “technical skills”):
need to obtain data from tests and surveys being carried out before and after a course;
better if this happens online, using tools xAPI-enabled
● correlate group activity with measured outcomes (concerning mainly “transversal skills”):
need also to know group roles and to analyze activity patterns, including interpersonal
communications
● assess content quality:
automatic assessment of essays and of other textual “artefacts”?
● ...

38. SOME REFERENCES
● Giovanni Toffoli, "Learning Analytics in Up2U”, CS learning path
​https://cs.up2university.eu/lp/learning-analytics-in-up2u
● S. Lariccia, G. Toffoli, “Shared Learning Data …”, paper presented at EduLearn 2019
https://docs.google.com/document/d/1YAQEtNHswRs3dOK_sB2Sv3hwbK2XQaY6N5qev7EJMjc/edit?usp=sharing
● Charles LANG, George SIEMENS, Alyssa WISE, and Dragan GAŠEVIĆ (Eds), “Handbook of
Learning Analytics”, the Society for Research on Learning Analytics (SoLAR) (2017)
https://solaresearch.org/wp-content/uploads/2017/05/hla17.pdf
● Brian Miller, “Anatomy of a Tin Can API Statement”, Rustici Software
https://xapi.com/wp-content/assets/e-book/Anatomy-of-a-Tin-Can-Statement-e-Book.pdf
● “Dashboards & Widgets”, in user documentation of Learning Locker
https://ht2ltd.zendesk.com/hc/en-us/categories/115000129989-Learning-Locker
● “H5P and xAPI”, in H5P documentation
https://h5p.org/documentation/x-api
● L.A. Barba et al., “Teaching and Learning with Jupyter”, 2019
https://jupyter4edu.github.io/jupyter-edu-book/
● homepages of the communities "Up2U H2020 project" and "Lithuanian Schools" in CS
https://cs.up2university.eu/project/up2u/​ , ​https://cs.up2university.eu/project/lithuanian-schools/