Presentation at TEA 2016 conference, Tallinn University

1. Evidence-Centered Approach to
Online Assessment of Students’
Digital Competence
Mart Laanpere, Kai Pata (Tallinn University)
Mario Mäeots , Leo Siiman (University of Tartu)

2. Problem space
• Need to assess digital competence: a newly
defined key competences in Estonian national
curriculum for primary and secondary schools
• Requirements:
– Compliant with DigComp framework
– Implemented online using e-exam system EIS
– Scalability
– Automated assessment
– Reliability, validity

3. Digital competence
• Three competing approaches to digital
competence/competencies:
– Theory-based knowledge, derived from the
academic discipline of computer science
(computational thinking)
– Pragmatic skills, needed at the workplace (ECDL)
– Key competence, contextualised, closer to
“learning to learn” skill
• DigComp follows the second approach,
Estonian curriculum combines 2nd and 3rd

4. Teaching and assessing ICT skills in
Estonia
• 1986: Computer science as a compulsory
theoretical subject, no exam, teacher-created
tests
• 1996: Informatics as an elective subject, learning
generic ICT skills for office jobs
• 2000: ICT and media as cross-curricular theme,
pilot exam on national level (test + practical task)
• 2011: informatics as an elective subject, digital
competence as a key competence

5. Research design and organisation
• Two teams of researchers from two different
universities (University of Tartu & Tallinn
University)
• Combination of two approaches:
– Inductive/descriptive, bottom-up (Tartu team):
collecting and analysing the best practice from
large-scale comparative and national studies
– Deductive/prescriptive, top-own (Tallinn team):
building a new framework from scratch, deducing
the new type of practice

6. ECD framework (Almond,
Mislevy et al. 2015)
inspired by e-exams of
Cisco Network Academy;
Has been applied in
various contexts:
Assessing language
competence,
Inquiry skills in STEM
Etc.

7. Layered assessment architecture of ECD
Layer Role Key entities Examples
Domain
analysis
Collect info about
domain, teaching
practice, frames
Concepts; terminology;
tools; representation
forms
Curricula, rubrics, test
Domain
modelling
Express assessment
argument in narrative
form
KSAs; potential work
products;
potential observations
Design patterns
Assessment
framework
CAF
Express assessment
argument as blueprints
for tasks or items
Student, evidence, and
task models; student-
model, rubrics etc
Task templates
Imple-
mentation
Implement assessment,
presenting tasks or
items and gathering and
analyzing responses
Task materials (incl.all
materials, tools,
affordances); work
products
Rendering protocols
for tasks; IMS/QTI
representation of
materials & scores;
Delivery Coordinate interactions
of students and tasks;
task and test-level
scoring; reporting.
Tasks as presented; work
products as created;
scores as evaluated.
summaries for
individual and group-
level reports;

8. Assessment framework (CAF)
What?
How?
Where?
Extent?

9. Domain analysis: example

10. Performance taxonomy
Fact Concept Procedure Rule
Knows
Understands
Applies
HOTS
Based on: Merrill ja Feisel-Schmitz

11. Bayesian network
Information &
Data Literacy
Digital content
creation
Communication
& collaboration
Digital safety
Problem solving

12. KSA (Knowledge-Skills-Abilities)
P_Info P_Comm P_Content P_Safety P_Probl
T_Info
T_Comm
T_Content
T_Safety
T_Probl

13. Example: practical task
• Create a presentation that compares two
countries: Estonia and Slovenia
• Criteria:
– At least 4 slides/screens
– Diagrams based on reliable and recent data
– Images (IPR need to be followed, licensing)

14. Piloting
• Not completed, planned in 6-8 schools this
October
• Bayesian data analysis, evidence accumulation
• Scenario-based design of new task templates
• Exploring the solutions for simulation tasks
(inspired by examples collected by Tartu
group)

15. Questions for discussion
• Any experiences with ECD?
• How to combine inductive and deductive
approaches?
• Semi-automated assessment of practical
tasks?