1. An Adaptive Assessment System with Knowledge
Representation and Visual Feedback
Joaquim Fernando Silva, Francisco José Restivo
Porto University
Portugal
Abstract— Assessment is essential for certification, regulation Our research is focused on how to use technology, for
and feedback within the education process. Our research effort formative assessment, in formal learning contexts to
focus on the last two features, and tries to identify the strengths individually infer which knowledge each student is acquiring
and weaknesses of the students, in order to adapt assessment and identify the learning strengths and weaknesses. Our
activities for better learning results. hypothesis is that students desire immediate results during the
learning process and quantitative measures to differentiate
The approach consists in designing a technological artefact for themselves. Also there is an appealing interest in visual
formative assessment with visual feedback during the learning
representations, in line with visual thinkers, who use the
process, to meet students’ demands for personalised tests in order
emotional and creative part of the brain to organize information
to effectively assess the learning outcomes.
or thoughts in an intuitive manner [1].
While many of the assessment tools are oriented to quantitatively This paper has three contributions for AAS. Firstly, it
measure the student’s knowledge acquisition, the idea behind this presents a general architecture with interrelated modules.
work is to use assessment as an analysis tool to measure, optimize Secondly, it provides a bottom-up knowledge representation
the learning process, personalise assessment and involve students, technique for teachers structure a domain knowledge. Thirdly,
through an adaptive approach, where each student can visualize
it presents a visual feedback solution for students’ enhance
the dynamic process of knowledge acquisition.
learning.
Keywords- Adaptive Assessment System, AAS Architecture, After we have introduced the problem, the remaining part
Knowledge Representation, Visual Feedback. of this paper: section 2 describes several adaptive assessment
systems analysed during the design of our technological
I. INTRODUCTION
solution; section 3 refers to our approach solution starting with
Many students, who have repeated learning difficulties, a general architecture and the use case for the adaptive
make small gains in school performance, while their academic assessment system; section 4 refers to the proposed proof of
achievement gap widens year after year. There is a missed concept; and section 5 presents the paper conclusions and the
guiding in the assessment role in education. Instead of ongoing work.
identifying the literacy or numeracy skills, that should be
worked out reflecting achievements, the effort seems to II. ADAPTIVE ASSESSMENT SYSTEMS
serialize students. During the learning process, formative assessment gives
Looking at the players, students usually are not motivated, continuous feedback about students’ current state of knowledge
or engaged in learning activities, and teachers aspire to find the acquisition. Our focus is on formative adaptive assessment.
average student. In classes, with many students, individual There are different approaches for formative adaptive
teaching is unviable. Teachers claim they do not have enough assessment systems. Some authors associate adaptation with
time to deliver individual tests, correct them and identify the presentation, when the environment is adjusted with different
underlying difficulties each student has. colour schemes, layouts or amounts of functionality, whereas
The need for individualised learning with personalised tests others associate it with curriculum sequencing, when learning
is a consequence of the raising differentiation in education. In is tailored to individual preferences, objectives, learning styles
fact, there is an increasing heterogeneity in students’ profiles or prior knowledge; and also associate it with problem-solving,
which demands for adaptive actions in education. Formative when feedback is provided with hints or solutions during the
assessment can achieve this when both players: teachers and problem solving process [2, 3]. Other authors even consider
students infer which knowledge each student is acquiring and adaptive assessment a place to obtain any evidence about
identify their learning difficulties. Also Adaptive Assessment knowledge, skills, or attitudes. It is possible to use adaptive
Systems (AAS), a technological solution used for personalised processes in tests, when assessment is item based, or in
tests and evaluate the student’s progress, can assist teachers in questions experiences when assessment is the curriculum itself.
formative assessment .

2. Several adaptive assessment systems are presented next, The iAdaptTest [8] is a desktop-based tool for adaptive
classified as computer adaptive tests, or adaptive questions question based on IMS QTI, IMS LIP and XML Topic Maps
tests. These systems are a reference for our solution proposal. standards. It has the possibility of reusability and
interoperability of data.
A. Computer Adaptive Tests
Computer adaptive assessment test uses engines to select The LEO [9] is a software program that provides a concept-
questions according to the students’ abilities, eliminating map with extra features to design a curriculum. It has a graph
questions that are too easy or too difficult. The questions can like structure with two types of nodes: instructional and
be multiple choice or true and false or even matching questions explanation nodes that explain the topics.
and tests assess student’s strengths and weaknesses on an item- All these systems and considerations led us to design a
by-item basis. With this assessment approach students proposal, presented next, for the initial problem of using a
accomplish excellent results where they are proficient and technological tool for formative assessment, tailored by the
identify the flaws in their learning. teacher and adapted to each student in a continuous and
Usually these systems use a statistical model, mostly the interactive manner.
Item Response Theory (IRT) or the Rasch Modelling. There III. SOLUTION PROPOSAL
are several adaptive systems in literature. Some systems are
worth mentioning identifying its advantages and disadvantages A. General Architecture of an Adaptive Assessment System
in order to help us designing an improved adaptive assessment Based on the analysis of several adaptive assessment
system. systems (AAS), and on our own findings, we propose an
The SIETTE [4] is a web-based system that mainly uses architecture with three sections and six modules.
multiple-choice questions. The question selection is based on a 1) Domain and Activities Module
function that estimates the probability of correctness to The Activities Module defines, classifies and stores the
particular questions based on a temporary learner model, available activities for the assessment tests, and the Domain
leading to an estimation of the students’ level of knowledge. Module supports the concept structure for representing the
The COMPASS [5] is also a web-based system that uses a domain knowledge to be assessed by the student. Basically it
concept map assessment tool for helping students understand, uses a hierarchical tree structure with the general assessment
as well as to support the assessment process. It provides outcomes in the parent node, and at the child nodes the
different informative and tutoring individualized feedback. concepts.
Students reconsider their beliefs, reflecting on them, and
2) Pedagogic and Student Module
redefine their knowledge structure.
The Pedagogical Module is responsible for the instructional
The PASS is a web-based assessment module, which can method used, representing several sequence of concepts and/or
be integrated into an Adaptive Educational Hypermedia activities for the assessment outcomes.
System (AEHS). AEHS are learning environments which
The Student Module stores information about the students’
address personalization with learning models based on goals,
profile and performance when doing the tests. It introduces a
preferences and knowledge of each student, adapting learning
dynamic approach in choosing the test items not only with the
content to the needs of each student. The PASS system
assessment outcomes, but also with the students’ previous
estimates students’ performance based on multiple assessment
assessment results. The adaptive process, finding which
options (pre-test, self-assessment and summative assessment).
concepts have been accomplished and the ones which are
It re-estimates the difficulty level of each question at any time
getting behind, uses an artificial neural network for calculating
and relates with the importance of the educational material
the probability of success for each concept with chosen
available [6].
activities. Also the ratio of right/wrong answers for a particular
B. Adaptive Question Tests set of activities, within a concept is considered.
The adaptive questions approach starts with the premise The adaptive assessment has two dimensions. A static
that a dynamic sequence of questions depend on students’ dimension based on the structured domain knowledge and a
responses. It is supported with rules about the student given dynamic dimension which models the student behaviour when
responses and overlay the student model to knowledge doing test assessments. Both dimensions complement each
representation based on topics or concepts. In comparison with other.
adaptive testing this approach presents more flexibility to
teachers in order to include didactical and personal methods To model the student’s behaviour, the system takes into
through the creation of appropriate rules. account personal characteristics like age, sex, learning style or
even the past and present class that he or she belongs to. Also
The CosyQTI [7] is a web-based tool for authoring and the reaction towards failure and subsequent commitment is also
adaptive questions test based on IMS QTI, IMS LIP and IEEE important. Some students give up when facing obstacles while
LTSC PAPI standards. It has the advantages of using standard- others enrol more when find difficult questions. This issue is
complaint and open information policy, which makes it related with the learning pattern each student has or the
interoperable and reusable in other learning environments. learning curve. For instance, the level of confidence is
measured with the amount of questions which are postponed

3. and actually answered correctly, along with the questions the Figure 1 represents the general components of the AAS
student did not answered. artefact.
3) Feedback and Environment Module Module
Teacher
The Feedback Module is responsible for transmitting, using
text and graphics, the test results and gives hints or clues to
students for future actions. Finally the Environment Module is
related with the assessment presentation which can be Web
based, desktop based, game based or mobile based. Activities Module Domain Module
Pedagogical
Module
The assessment feedback, about the achieved results is
essential for each student get conscious of his or her
performance. Also feedback can provide hints or advices to
outcome the identified difficulties. The dynamic assessment Environment Feedback
Student Module
behaviour of the student is also essential to understand the Module Module
learning evolution and increase the interaction levels.
A personalized feedback, a text and visual representation
about student’s performance, helps students and teachers to
better understand the knowledge acquisition and change the
interactions in the classroom for effective learning. Student
Our focus is on visual representations for personalized
feedback assessment. Visual representations based for Figure 1. Components of a general AAS
representing hierarchical or semi-structured data [10] to
visualize the students results [11] are not an easy business and B. Use Case
usually the approaches are oriented to a particular problem The In this section the essentials of the use case used with the
time dimension is also essential for students understand the solution proposal are presented. The AAS solution is planned
learning progress, which is a dynamic and ongoing process. In to be used in a high school context with selected teachers and
fact, learning is a starting process that never ends. students. The selected teachers, from different domains just
create their classes. The students are the ones of the selected
A visual representation to express the temporal dimension teachers and just choose their class.
has several approaches that might be used for visual
assessment feedback. During our research we analysed several Teachers can author the activities, import them or build
approaches and decided to analyse our problem, with a them, can structure the concept layer of the domain with a
systematic approach, to select the right visual tool with a graph tool and define the desired abilities. The activities are
categorization criteria of time, data and representation [12]. classified accordingly to the three lower Bloom’s Taxonomies
[13] (remembering, understanding, applying). The cognitive
We consider assessment, within the AAS system, a cyclic taxonomy of remembering should take place before
time dimension and the students achieved results an abstract understanding and then applying the concept in new situations.
data where there is no spatial mapping. The other three upper taxonomies (analysing, evaluating and
The Feedback Module, uses both visual representations: a creating) should take place only after the first three taxonomies
static visualization for the knowledge domain and a dynamic are mastered. For that reason the teacher also plays an
time-oriented visual representation for the student performance. important role in the adaptive process because he or she can
use the feedback results for enhancing education in the
Along with visualization, user interaction is also very classroom [14].
important. The student has a global image of the assessment
feedback, but those who wish to acquire insight with the The students have the opportunity to do online assessment
picture can explore interactively, with pop-up texts appearing tests with feedback results. The feedback result is both a
on mouse over the petals, with additional information. For Treemap visualization for all concepts under study and a
instances the name of the concepts or the abilities and some PeopleGarden visualization for the ongoing achievements.
recommendations associated with the red concepts, in order to Several other services can be associated, but the essential
orient the student in future class assignments. services are defined in Figure 2 with a UML use case diagram.
In order to modify the student behaviours the social IV. PROOF OF CONCEPT
dimension is very important. The student can compare his or The concept is applied to a high school context. In this
her result with their peers. experimental work all students are connected with their
The Environment Module includes several channels, from teachers, based on the class and school. The experiment will be
web based to mobile based, promoting ubiquity. Besides monitored with weekly meetings with teachers and students to
mobile solutions are growing in importance with the identify the positive aspects and negative aspects of the system.
appearance of Smartphone’s with capabilities more than Finally all the players will answer the online questionnaire: in
sufficient for enhancing learning. the end of system evaluation.

4. After the first experiment, the case study will continue till The feedback visualization of results underpins the research
the end of the year, measuring the full impact of the tool in effort to reach higher levels of students’ commitment and we
terms of interest, changing attitudes towards learning and are currently designing the visual feedback, using a visual time
improving knowledge acquisition. based representation.
Adaptive Assessment System ACKNOWLEDGMENTS
*
Create Class
«extends»
Class
«uses»
Choose Class
*
The authors would like to thank the LIACC Laboratory for
-End37
«extends»
-End40
excellent working conditions. The first author would like to
*
*
* -End33
*
-End34 -End35
-End36
-End28
-End32
-End30
* thank Portuguese FCT for the support provided through
-End38
-End22
*
-End24
-End26
*
Choose Domain
*
-End39
*
*
* *
scholarship SFRH/BD/36206/2007.
* -End27
Teacher n
-End21
Register
*
Student n
REFERENCES
* -End29
-End23
Profile
* [1] E. Copperman, C. Beeri, and N. Ben-Zvi, "Visual modelling of learning
* -End31
Global Reports
processes," Innovations in Education and Teaching International, vol. 44,
-End25 *
+End2 pp. 257-272, 2007.
Author Activities
*
«uses» -End8 [2] P. d. Bra, Adaptive hypermedia. In Handbook on Information
-End4
-End3
+End1
Classify Activities
«uses» * -End10
Technologies for Education and Training Berlin: Springer, 2008.
-End5 *
-End5 -End7 *
-End5
**
*
-End6
Structure Concepts
«uses»
On Line Assessment
-End9
-End11
Student 1 -End12 [3] T. L. Kinshuk and A. Patel, User Adaptation in Supporting Exploration
* Test *
* * *
-End6
«uses»
*
Student 2
* Tasks in Virtual Learning Environments. Netherlands: Springer, 2006.
«uses»
Teacher n
*
Define Rules
«uses» Student n
[4] R. Conejo, E. Guzmán, E. Millán, M. Trella, Pérez-De-La-Cruz, and R. J.
-End6
*
Define Abilities * L., "SIETTE: A Web- based Tool for Adaptive Testing," International
-End18
* * * -End17
-End19
-End15
-End20
Journal of Artificial Intelligence in Education, vol. 14, pp. 29-61, 2004.
*
-End14 -End13
Feedback Results
-End16
**
Student 1
*
[5] E. Gouli, A. Gogoulou, K. Papanikolaou, and M. Grigoriadou,
Student 2
"COMPASS: AN ADAPTIVE WEB-BASED CONCEPT MAP
Teacher n Student n
ASSESSMENT TOOL," presented at International Conference on
Figure 2. Use Case for the Proposed AAS Concept Mapping, Pamplona Spain, 2004.
[6] E. Gouli, K. Papanikolaou, and M. Grigoriadou, "Personalizing
Also some interpellations were made to students, trying to Assessment in Adaptive Educational Hypermedia Systems," presented at
delve into their motivations. The system will be used by some Second International Conference on Adaptive Hypermedia and Adaptive
invited teachers with their students. We have decided not to use Web-Based Systems, 2002.
a test pilot approach, but rather verify the system impact in the [7] P. Lalos, S. Retalis, and Y. Psaromiligkos, "Creating personalised quizzes
players. That is, we are planning to use it with several classes
both to the learner and to the access device characteristics: the Case of
and compare it with results of previous years, and more
important, in the end of the case study a questionnaire will be CosyQTI," presented at Workshop on Authoring of Adaptive and
delivered. The questionnaire will give emphasis for the Adaptable Educational Hypermedia, 2005.
qualitative approach decision, in order to validate our research [8] F. Lazarinis, S. Green, and E. Pearson, 257-278., "Focusing on content
that using technology for formative assessment in formal reusability and interoperability in a personalized hypermedia assessment
learning contexts, the players can infer which knowledge each tool," Multimedia Tools and Applications, vol. 47, pp. 257-278, 2009.
student is acquiring and identify the learning strengths and [9] J. W. Coffey, "LEO: A Concept Map Based Course Visualization Tool
weaknesses. for Instructors and Students," presented at Knowledge and Information
Visualization, LNCS, 2005.
V. CONCLUSIONS AND ONGOING WORK
[10] M. Lanzenberger, J. Sampson, and M. Rester, "Ontology Visualization:
The model presented, for an adaptive assessment system, Tools and Techniques for Visual Representation of Semi-Structured
starts with the premise the teacher provides knowledge domain
Meta-Data," Universal Computer Science, vol. 16, pp. 1036-1054, 2010.
structure with abilities and concepts attached with several
[11] P. Brusilovsky, I.-H. Hsiao, and Y. Folajimi, "QuizMap: Open Social
activities. The adaptive assessment algorithm, based on
artificial neural network (ANN), predicts the true knowledge Student Modeling and Adaptive Navigation Support with TreeMaps,"
each student has, identifying the strengths and weaknesses, presented at Towards Ubiquitous Learning, 6th European Conference of
suggesting the next activity. The system delivers a visual Technology Enhanced Learning, EC-TEL 2011, LNCS, 2011.
feedback, for each student, about the ongoing assessment. To [12] Wolfgang Aigner, Silvia Miksch, Wolfgang Muller, Heidrun Schumann,
validate this approach solution a use case will be carried out in and Christian Tominski, "Visualizing Time-Oriented Data – A Systematic
a high school context in several knowledge domains. View," Computers and Graphic, vol. 31, 2007.
The student, using this system, can cement knowledge, [13] L. W. Anderson, D. R. Krathwohl, and A taxonomy for learning, teaching
deepening the concepts acquisition or exploring knowledge. and assessing: A revision of Bloom's Taxonomy of educational objectives:
Also the teacher can introduce, in the system, at any time, new Longman, 2001.
and different activities in order to reinforce one or both [14] M. Pohl, Learning to Think, Thinking to Learn: Models and Strategies to
assessment outcomes. Develop a Classroom Culture of Thinking, 2000.