IEEE EDUCON 2013 Conference Berlin

1. A cluster-based analysis to diagnose students’
learning achievements
Luz Stella Robles Pedrozo, U. Tecnológica (Cartagena, Colombia)
Miguel Rodríguez Artacho, UNED University (Madrid)
IEEE EDUCON 2013 (Berlin)

2. Content
1. General Objectives
2. Background and Motivation
3. Proposed Diagnostic Test Methodology
4. Conclusions
5. Future Work
IEEE EDUCON 2013 (Berlin)

3. General
Objec,ves
Scope
Recognizing Learning Disabilities trough Testing: Clustering Based Methodology and Reliability
General Objective
The design and implementation of a methodology for learning disabilities diagnosis and
assessment based on:
û Adaptive feedback to the students in order to individually identify learning weaknesses and
misconceptions about a topic right after assessment through testing.
û Classification of the students via clustering of the detected learning disabilities, as a support
for the design of feedback strategies and activities for improving their academic performance.
IEEE EDUCON 2013 (Berlin)

4. Background
and
Mo,va,on
û Problems with prior knowledge diagnostic assessment using standardized tests with manual
scoring: Type I ICFES multiple choice questions with only one correct answer. This kind
of questions are used for: Midterm exams, SABER 5th, 9th, 11th and SABER PRO mandatory
state tests in Colombia.
û The traditional education system uses pass/fail scoring scale based written exams for
assessment à The score does not provide enough information about learning that
can be used for performance improving.
û The recognition of learning disabilities and misconceptions is key and complex process that
has to be manually performed.
IEEE EDUCON 2013 (Berlin)

5. Background
and
Mo,va,on:
tests
û Disadvantages of traditional tests : The same test, with a fixed
number of items, is given to all test takers. They have limited answer
choices. The test is long in order to make it more accurate.
û The assessment uses traditional methodologies which do not allow :
− Identification of systematic misconceptions and weak
understanding of concepts in order to plan strategies to improve
their academic performance.
− The classification and grouping of the students to undertake a reorientation of the reinforcement activities.
− The individual recognition of the level of learning disabilities and
misconceptions.
IEEE EDUCON 2013 (Berlin)

6. Background
and
Mo,va,on:
feedback
A diagnostic assessment methodology that provides a classification score, identifies
learning disabilities, misconceptions and weak understanding of concepts, allowing to
group the students with similar problems in clusters, is required.
Structure of the proposed diagnostic assessment methodology:
û Item Response Theory (IRT) is used as the method to obtain the skill level of
each concept.
û The use of a system of interrelated concepts and dependences to identify
cognitive disabilities (misconceptions and weak understanding of concepts)
û The use of Clustering to classify the students in groups with similar disabilities
IEEE EDUCON 2013 (Berlin)

7. Proposed
Diagnos,c
Assesment
Methodology
Item Response Theory (ITR) [Thurstone, 1925], [Lord, 1952, 1968]
ITR allows invariant measured variables that are independent with respect to the examinees
and the used test instruments.
CTT
ITR
Lack of invariance in the properties of the
tests with respect to the test subjects. The
characteristics of the items depend on the
group of persons.
Different tests can be comparable, as
the skill level trend to be the same
between different item sets
Asumes the same error level for all subjects,
Similar level of assessment accuracy
or the test liability is the same for all the
for all different participants.
participants (as a property of the test)
IEEE EDUCON 2013 (Berlin)

8. Proposed
Diagnos,c
Assesment
Methodology
ITR Models
û 1, 2 and 3 parameters unidimensional logistic models
û Dichotomous answer format (only one answer)
û Performance and skills assessment
ITR – Model proofing
The test instrument, with the items containing the object variable, is applied to
û Validate the ITR assumptions
û Select the optimum models based on statistical analysis
ITR – Once the model is selected …
û Estimate the parameters of the selected model
û Calculate the skill or proficiency level of the test subjects
û Identify learning disabilities in the test subjects
IEEE EDUCON 2013 (Berlin)

9. Proposed
Diagnos,c
Assesment
Methodology
Diagnostic Methodology : Item selection
û At least one assessment item assigned to each node of the framework.
û The knowledge domain to be evaluated, categorized into sub-topics and pre-requisites.
û The dependences between the items and the concepts (concepts for the assessment in
each item).
û The weight of the concepts in each item.
IEEE EDUCON 2013 (Berlin)

10. Proposed
Diagnos,c
Assesment
Methodology
An inference example (probability and statistics)
IEEE EDUCON 2013 (Berlin)

11. Proposed
Diagnos,c
Assesment
Methodology
IEEE EDUCON 2013 (Berlin)

12. Proposed
Diagnos,c
Assesment
Methodology
Diagnostic Methodology
Tool
used:
R
h,p://www.r-­‐project.org/
IEEE EDUCON 2013 (Berlin)

13. Proposed
Diagnos,c:
Learning
Paths
Diagnostic Methodology
IEEE EDUCON 2013 (Berlin)

14. Clustering
Cluster Generation
IEEE EDUCON 2013 (Berlin)

15. Clustering
Cluster Generation
û List of weakly-understood concepts per each examinee
û Total weight of each weakly-understood concept in the test (TP CI d)
û Calculate the total weight of the weakly-understood concepts in the test (PTcd) per each
examinee, as in :
IEEE EDUCON 2013 (Berlin)

16. Clustering
Cluster Generation
IEEE EDUCON 2013 (Berlin)

17. Clustering
Cluster Generation
IEEE EDUCON 2013 (Berlin)

18. Conclusions
Psychometric aspects
û The Item Response Theory (IRT) was selected for this work after a proper understanding of its
advantages with respect to the Classical Test Theory (CTT).
û An statistical procedure was proposed to select and validate the optimum model to use with the
obtained data from the tests used in this work.
A computer program was designed on the R
language for analysis purposes .
û
A comparative studied was performed between the score for the skills level of a group of
examinees obtained with the classical test theory (TCT, average score) and that obtained with the
IRT model (unidimensional 3 parameters logistic model 3PL)
IEEE EDUCON 2013 (Berlin)

19. Conclusions
Regarding the Diagnostic Methodology
A software for diagnostic was implemented:
• Process answers of the examinees ( Deficient and Minimum) to generate the weaklyunderstood concepts per student
• Represent the suggested leaning paths for each examinee.
• An index representing the total weight (or total sum of weigths) of the weakly-understood
concepts in the test per examinee is generated.
Regarding the Cluster
A computer program was implemented in R in order to generate a list classifying the examinees in
groups with similar misconceptions or learning disabilities.
à Data mining tools can be as useful as Intelligent Systems in certain domains to diagnose
student models.
IEEE EDUCON 2013 (Berlin)

20. Conclusions
û This work is useful for public education institutions in Colombia because it serves as a solution
for the efficient diagnostic of the learning disabilities in students by using a test.
û The design and implementation of the diagnostic procedure, suppported with IRT and
clustering procedures, allow to perform a comprehensive diagnostic of the learning disabilities,
misconceptions and weak understanding of concepts in students.
û The work provides the students with a tool for the easy identification of their learning and
cognitive disabilities, and the suggested self-learning path to improve their academic
performance
à Provide feedback
IEEE EDUCON 2013 (Berlin)

21. A cluster-based analysis to diagnose students’
learning achievements
THANKS!
Luz Stella Robles Pedrozo, U. Tecnológica (Cartagena, Colombia)
Miguel Rodríguez Artacho, UNED University (Madrid)
Learning Technologies and Collaborative Systems
http://ltcs.uned.es
IEEE EDUCON 2013 (Berlin)