Competency Modeling for E-Learning

PAWS Lab Work on
Competencies and
Student Modeling
Peter Brusilovsky
School of Information Sciences
University of Pittsburgh, USA
peterb@pitt.edu
http://www.sis.pitt.edu/~peterb

Agenda
• Overview
• ADAPT2 architecture
• Original student modeling in CUMULATE
– Example, DB Exploratorium
• Problems and solutions
– Multi-ontology issue – introduce ontology server
– Efficiency – pull to push switch
• Cross-systems, cross-ontology, and cross-
domain modeling

Main Stages of Our Work
• Centralized user modeling (1990-1998)
• Multi-system personalization based on ADAPT2 (2003-2007)
– CUMULATE 1: Single domain model (one system, one model)
(2003-2006)
– CUMULATE 2: Parallel independent modeling using 2 models
(2004-2014)
• Cross-domain mapping for cold start (2007)
– C to Java
• Single domain guided evidence mapping (2008-2010)
– Topic to concept mapping for Java
– Constraints to concepts mapping for SQL
• Single domain automatic mapping (2010-2012)
University of Pittsburgh - PAWS Lab 3

User Model
Collects information
about individual user
Provides
adaptation effect
Adaptive
System
User Modeling side
Adaptation side
Centralized Single System Modeling
Classic loop user modeling - adaptation in adaptive systems
University of Pittsburgh - PAWS Lab

KT Architecture
• Learning experiences are delivered by various [adaptive,
smart] re-usable activities residing on distributed activity
servers
• A portal provides single log-in and singe access point to all
content
• A student modeling server maintains a centralized student
model
• A value-added service could work as intermediary
between “dumb” learning content and portal
• Brusilovsky, P. (2004) KnowledgeTree: A distributed
architecture for adaptive e-learning. In: Proceedings of
13th International World Wide Web Conference, WWW
2004, New York, NY, 17-22 May, 2004, ACM Press, pp.
104-113

KT Architecture
Portal
Activity
Server
Student Modeling Server
Value-added
Service

Making it Open
• There are no other requirements to the
components than an ability to support
standard protocols
• Any new activity server can be used as long as
it complies to the protocols
• The architecture allows for different portals
and value added services to co-exist as long as
they support protocols
• Multiple student model servers allowed

Protocols
• Portal/service  activity server/service
– Request activities, respond with a list of relevant
activities, start activity
• Portal/service/activity server student
model server
– Report information about student, request
information about student
• Student model server portal  service
activity server
– Transparent chain of authentication

A student model server CUMULATE
Event Storage
Inferenced UM
UM requests
Application External
Inference Agent
Internal
Inference Agent
UM updates
Event reports
Event requests

Competencies-Based Modeling
• Lower level of student model has a flow of content-level
events
– Which content was used, who used, results (0-1)
• Each content item is connected to knowledge units
– Topic-based modeling: coarse grain units, each content
“belongs” to topic (1->N), based on topic network
– Concept-based modeling: fine grain units, each content is
indexed with related concepts, based on ontology
• An inference agent processes events in the context of KU
connections and maintains up-to-date KU-Level model
• Cumulate allows multiple independent inference agents
– Agents for different modeling approaches (i.e, BMA, BKT)
– Agents that model content on different levels

Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N10
3
0
2
7
4
Concept-Level Knowledge Model

Example: Database Exploratorium
• Knowledge Tree portal
for content access
• Three kinds of activities
– Examples
– Problems
– SQL Lab
• Central user
model server
CUMULATE
• Two levels of modeling
– Topics (teacher)
– Concepts (ontology)
• Both levels are used
independently for
adaptation
Brusilovsky, P., Sosnovsky, S., Lee, D., Yudelson, M., Zadorozhny, V.,
and Zhou, X. (2010) Learning SQL programming with interactive
tools: from integration to personalization. ACM Transactions on
Computing Education 9 (4), Article No. 19, pp. 1-15.

SQLOntology
We created C, SQL
and Java Ontologies

Moving to many systems and ontologies

Problems with KT
• We started the integration of adaptive systems
produced by other groups…
• Multiple ontologies (domain models)
– Two systems complement each other, but use
different domain models for content indexing
• Complex user modeling mechanisms
– User modeling server can’t replicate same level of
inference student models from events

Cross-System Knowledge Modeling
http://adapt2.sis.pitt.edu/kt/
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyes
no
no
no
yes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyes
no
no
no
yes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyes
no
no
no
yes
yes

Missing links
The Approach: Ontology-Based Cross-
System Personalization
Connect DM
(ontologies)

UM of C
knowledge
Java
C
UM of
Java
knowledge
How we started – from C to Java
• Manual vs. Automatic
ontology mapping
• Knowledge mapping using
ontology mapping
• Compare predicted and
demonstrated knowledge
• Automatic mapping is
comparable with manual
• Overall gain for translated
knowledge is not high
• We got concerned about
model to model mapping
• Started exploring evidence
mapping
Sosnovsky, S., Dolog, P., Henze, N., Brusilovsky, P., and Nejdl, W. (2007) Translation of overlay models of student
knowledge for relative domains based on domain ontology mapping. 13th International Conference on Artificial Intelligent in
Education, AI-ED 2007, Marina Del Rey, CA, July 9-13, 2007, IOS, pp. 289-296

How we can deal with multiple
competency organizations?
• Content should be separated from its content-
metadata, i.e., ontology indexing or topic
categorization
• The same smart content item could be classified
under different topic networks or indexed using
different ontologies
• We need to maintain and use multiple
descriptions for the same item and multiple user
models!

Solution: Ontology Server
• Ontology Server as a new component in the new ADAPT2
architecture
• Ontology server maintains one specific domain ontology
• Ontology Server collects metadata about everything
related to this ontology
– Content-level metadata for all resources indexed with this
ontology
– Overlay student models for all students that are modeled with
this ontology
• A Student modeling server can use several ontology
servers in parallel to perform modeling in different
ontologies
• Brusilovsky, P., Sosnovsky, S., and Yudelson, M. (2005)
Ontology-based framework for user model interoperability
in distributed learning environments. In: World
Conference on E-Learning, E-Learn 2005, pp. 2851-2855.

Multiple Ontologies in ADAPT2
• The new architecture ADAPT2 allows the use
of multiple ontologies for content and student
modeling
• Each ontology is maintained by a dedicated
ontology server
• Ontology server is handling all requests
related with the ontology - about the ontology
itself, learning activities, and users

Summary
• Learning activities are separated from its
content metadata
• An activity server’s duty is to maintain and
serve an activity (URI invocation)
• Each activity can be indexed in terms of
several ontologies
• An ontology server (not activity server!)
stores content metadata for all activities
indexed in terms of this ontology

Ontology server
An ontology server support inference level of UM server

SEDONA: UM exchange with ontology
servers
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyes
no
no
no
yes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyes
no
no
no
yes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyes
no
no
no
yes
yes
Ontology A
Ontology B

Practical Experience
• Implemented first version of an Ontology server
Sedona
• Addressed more urgent student model efficiency
issue
• Fully redesigned CUMULATE server, moved from
pull to push, very efficient
• Ontology server as a unit has never been adapted to
new CUMULATE, instead CUMULATE started to
perform some of its functions
• Decided to collect more cross-ontology experience to
redesign all Sedona functions properly
• Continued with a series of cross-ontology modeling
experiments

SEDONA: UM Exchange
• Ontology server is an exchange point for concept-
level overlay student models that are based on the
stored ontology
• Each UM server or adaptive system that can deduce
student knowledge in terms of this ontology reports
it to the server
• Each adaptive system that need to know the level of
student knowledge for concepts of this ontology can
query the ontology server

Lightweight event-based centralized
user modeling
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyes
no
no
no
yes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyes
no
no
no
yes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyes
no
no
no
yes
yes
Central UM
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyes
no
no
no
yes
yes
Sosnovsky, S., Brusilovsky, P., Yudelson,
M., Mitrovic, A., Mathews, M., and
Kumar, A. (2009) Semantic Integration
of Adaptive Educational Systems. In: T.
Kuflik, S. Berkovsky, F. Carmagnola, D.
Heckmann and A. Krüger (eds.):
Advances in Ubiquitous User Modelling.
Lecture Notes in Computer Science, Vol.
5830, pp. 134-158.

• Student side:
–Use systems in parallel (any order, any
combination)
–No extra overhead (single sign-on, single
place to access)
• System side:
–Integrated environment > (system1 +
system2)
–Each system should try to increase the
quality of user modeling and adaptation
What we Consider as True Integration

Explored Cases
• QuizJet integration with Problets in Java domain
– One source KI to many target KI mapping
– Brusilovsky, P., Sosnovsky, S., Yudelson, M., Kumar, A., and
Hsiao, I.-H. (2008) User Model Integration in a Distributed
Adaptive E-Learning System. Workshop on User Model
Integration at the 5th International Conference on Adaptive
Hypermedia and Adaptive Web-Based Systems.
• SQL Exploratorium integration with SQL tutor in SQL
domain
– Many to many KI mapping from source to target domain
– Sosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., and
Yudelson, M. (2008) Ontology-based integration of adaptive
educational systems. 16th International Conference on
Computers in Education (ICCE’2008), Taipei, Taiwan, October,
27-31, 2008, pp. 11-18

Java Problets: The Interface
Sample
program
Student’s
answer
Help
Question
text
System’s
feedback

Java Problets: Domain Model
• Problets implement traditional overlay user modeling to
adapt to student’s performance
 The domain model
of a problet is a
concept map
enhanced with
learning objectives,
that combine
pedagogical and
domain knowledge

QuizJET (1):
System Description
• QuizJet (Java Evaluation Toolkit) is a system for authoring
and delivery of online self-assessment quizzes for Java
programming language
• A typical QuizJET problem is a sample program (consisting
of one or several classes), that a student needs to evaluate
and provide an answer a follow-up question
• QuizJET generates problems by substituting a numerical
value in the program template with a randomized
parameter
• Upon receiving a student’s answer QuizJET provides a
feedback indicating the correctness of the answer and the
right answer (if the student’s attempt was not successful)

QuizJET (2):
Student Interface
• Students can access QuizJET problems through the
KnowledgeTree portal
Topics in the
course
Activities
available for the
current topic
Problem
text
Problem's
classes
QuizJET’s
feedback

QuizJET (3): Domain Model
• Java Ontology
specifies about 500
classes connected
with 3 types of
relations: subClassOf,
partOf/hasPart, and
related
• About 300 classes are
available for indexing
• A class can play one of
two roles in the problem
index: prerequisite or
outcome

Domain Model Integration
• Main problem: different modeling paradigms
– A learning objective models application of a concepts in the certain
context
– Extra classes from the Java ontology have been used for context
modeling
– Weights are assigned to prevent too aggressive propagation of
classes responsible for context modeling
• Example:
– This learning objective models a situation when the conditional part
of the if-else statement is a relational expression evaluated into true
value

Evidence-based UM integration in
CUMULATE

• An example of semantic integration of two working
adaptive systems relaying on very different domain
models
• Many to many KI mapping from source to target
domain
– Topology constructed by domain experts
– Data could be used to improve the mapping
Integrating SQL Tutor and SQL
Exploratorium

SQL-Exploratorium

http://www.sis.pitt.edu/~paws/ont/SQL.owl
SQL Explorer: SQL Ontology

SQL-Tutor: Constraints

• Constraints and Concepts are too difficult
to map them
• A typical constraint models syntactic or
semantic relation between several concepts
• Manual connect constraint to concepts
with some
degree (small-1,
medium-2, or large-3)
Domain Model Mapping

• Solution to SQL-Tutor problem, triggers a
number of constraints satisfied and or
violated
• Mapping model calculates knowledge
update for every concepts related to every
triggered constrained:
• The updates are reported to SQL-
Exploratorium’s user modeling server
Evidence-Based Modeling

• University of Pittsburgh,
2 courses: undergraduate and graduate
• ½ of semester
• 42 students tried SQL-KnoT, 18 – SQL-
Tutor
• Out of 103 sessions of using SQL-KnoT
66 co-located with SQL-Tutor usage
Evaluation

• Questionnaire (21 students)
– I1 / I2: Overall, I like the interface of SQL-
KnoT/SQL-Tutor.
– U1 / U2: SQL-KnoT/SQL-Tutor is a useful
learning tool.
– C1 / C2: SQL-KnoT/SQL-Tutor problems
challenged me intellectually.
Results

Evaluating and improving mapping:
SQL Exploratorium and SQL Tutor
• Authoring constraint mapping is time consuming
• How we can evaluate weights?
• How we can improve mapping?
49University of Pittsburgh - PAWS Lab

SQL KnoT and SQL-Tutor (2)
• 6 experts (2 teachers, 2 GSA, 2 practitioners)
• 1012 constraint-concept relations: strong (1/1),
medium (2/3), weak (1/3)
• Usage log of 3544 SQL-Tutor problem-solving
attempts of 38 users
• Dataset specific subset
– 282 constraints, 576 relations, 61 concepts

Fitting The Source
(Constraint) Model
• Experts only need to produce relations b/w KIs
– the rest is automatic
51University of Pittsburgh - PAWS Lab

References on cross-system modeling
Sosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., Yudelson, M., Brusilovsky, V., and Sharma, D.
(2008) Towards integration of adaptive educational systems: mapping domain models to
ontologies. Proceedings of 6th International Workshop on Ontologies and Semantic Web for E-
Learning (SWEL'2008) in conjunction with ITS'2008, Montreal, Canada, June 23, 2008.
Sosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., and Yudelson, M. (2008) Ontology-based
integration of adaptive educational systems. Proceedings of 16th International Conference on
Computers in Education (ICCE’2008), Taipei, Taiwan, October, 27-31, 2008, pp. 11-18.
Brusilovsky, P., Sosnovsky, S., Yudelson, M., Kumar, A., and Hsiao, I.-H. (2008) User Model
Integration in a Distributed Adaptive E-Learning Systems. Proceedings of Workshop on User
Model Integration at the 5th International Conference on Adaptive Hypermedia and Adaptive
Web-Based Systems (AH'2008), Hannover, Germany, July 29, 2008.
Brusilovsky, P., Mitrovic, A., Sosnovsky, S., Mathews, M., Yudelson, M., Lee, D., and Zadorozhny, V.
(2009) Database exploratorium: a semantically integrated adaptive educational system.
In: Proceedings of Ubiquitous User Modeling Workshop at the 17th International Conference on
User Modeling, Adaptation, and Personalization (UMAP 2009), Trento, Italy, June 22, 2009
Sosnovsky, S., Brusilovsky, P., Yudelson, M., Mitrovic, A., Mathews, M., and Kumar, A. (2009)
Semantic Integration of Adaptive Educational Systems. In: T. Kuflik, S. Berkovsky, F.
Carmagnola, D. Heckmann and A. Krüger (eds.): Advances in Ubiquitous User Modelling. Lecture
Notes in Computer Science, Vol. 5830, pp. 134-158
University of Pittsburgh - PAWS Lab 52

Automatic Ontology Mapping
• SQL Integration demonstrated using expert-authored and
automatically-tuned domain ontology mapping we can do
efficient cross-system personalization with two
conceptualizations (ontologies) in the same domain
• Expert labor is expensive. Could we do automatic mapping
between two ontologies in the same domain?
• The case is explored in
– Sosnovsky, S., Brusilovsky, P., and Hsiao, I.-H. (2012) Adaptation
"in the Wild": Ontology-based Personalization of Open-Corpus
Learning Material. In: Proceedings of 7th European Conference on
Technology Enhanced Learning (EC-TEL 2012), Saarbrücken,
Germany, pp. 425-431.
– Sosnovsky, S. (2011). Ontology-based Open-Corpus Personalization
for E-Learning PhD Thesis, University of Pittsburgh.
9/26/201053

What Happened with auto-mapping?
54University of Pittsburgh - PAWS Lab Sergey Sosnovsky PhD Thesis

OOPS Interface: Reading Phase
55
content of the
chosen topic
Navigation links to
the next and the
previous topics
Feedback/exit
buttons
University of Pittsburgh - PAWS Lab Sergey Sosnovsky PhD Thesis

Competency Modeling for E-Learning

Recommandé

Recommandé

Contenu connexe

Similaire à Competency Modeling for E-Learning

Similaire à Competency Modeling for E-Learning (20)

Plus de Peter Brusilovsky

Plus de Peter Brusilovsky (20)

Dernier

Dernier (20)

Competency Modeling for E-Learning