The researchers propose building an automated system to code discussion messages for levels of cognitive presence, which is one component of the Community of Inquiry framework. Existing Community of Inquiry coding schemes are too crude, labor-intensive, and cannot be used in real-time. The researchers have begun developing a support vector machine text classifier to code cognitive presence, achieving a Cohen's Kappa of 0.42. Their plan is to improve the classifier using additional features and evaluate it on a massive open online course dataset. The system would enable broader, faster, and cheaper adoption of the Community of Inquiry framework for research and learning analytics.

1. Automated System for
Cognitive Presence Coding
Vitomir Kovanović, Srećko Joksimović, and Dragan Gašević
THE UNIVERSITY of EDINBURGH
v.kovanovic@ed.ac.uk
@vkovanovicUK Learning Analytics Network
Nottingham,
24th June 2015
http://bit.ly/jisc_coi

2. Main Idea
Learning Analytics for Supporting
Traditional DE/OL & MOOCs
1

3. Overall Goal
Learning Analytics for Supporting
Traditional DE/OL & MOOCs
Through analytics:
1. Drive pedagogical interventions
2. Build models/frameworks of MOOC learning
3. Empowering instructors and learners
1

4. Study Approach
1. Build Learning Analytics research on existing
knowledge and models of Distance/Online education
2. Use Learning Analytics to validate and extend existing
educational theories
3. Automate as much as possible
2

5. Adopted Model:
Community of Inquiry (CoI)
Social
presence
Cognitive
presence
Teaching
presence
Educational
experience
Garrison, Anderson, and Archer (1999) 3

6. Focus:
Cognitive Presence
Social
presence
Cognitive
presence
Teaching
presence
Educational
experience
1. Triggering event:
Problem identification,
sense of
puzzlement
2. Exploration:
Brainstorming, Idea
exploration, divergence
3. Integration:
Synthesis of relevant
information
4. Resolution:
Problem resolution,
testing application
Garrison, Anderson, and Archer (1999) 3

7. CoI instruments
Quantitative coding schemes for three presences
● Very crude and high-level
● Very labour-intensive
● Require experienced coders
4

8. Two CoI instruments
Quantitative coding schemes for three presences
● Very crude and high-level
● Very labour-intensive
● Require experienced coders
4
Can’t be used in real-time for
driving instructional interventions

9. Two CoI instruments
Quantitative coding schemes for three presences
● Very crude and high-level
● Very labour-intensive
● Require experienced coders
Survey instrument
● 34 likert-scale items
○ 13 teaching presence
○ 9 social presence
○ 12 cognitive presence
4
Can’t be used in real-time for
driving instructional interventions

10. Two CoI instruments
Quantitative coding schemes for three presences
● Very crude and high-level
● Very labour-intensive
● Require experienced coders
Survey instrument
● 34 likert-scale items
○ 13 teaching presence
○ 9 social presence
○ 12 cognitive presence
4
Can’t be used in real-time for
driving instructional interventions
1. Administered after the course
2. Self-reported

11. Proposal:
Automated CoI Content Analysis
Build a system for automated
coding of discussion messages for
the levels of cognitive presence
5

12. Proposal:
Automated CoI Content Analysis
Build a system for automated
coding of discussion messages for
the levels of cognitive presence
5
Advantages:
● Enable for broader adoption of CoI model
● Faster and cheaper adoption in research
● Provide detailed operationalization of CoI coding scheme
● Real-time feedback of learning in discussions
● Enable for development of various analytics dashboards
Future work:
● Eventually support other presences/models

13. Current Progress (Kovanovic et al., 2014)
● One dataset fully coded (1,747 messages)
● First version of text classifier developed (SVM-based)
○ Uses Stanford NLP parser
○ Surface features (N-grams, POS N-grams)
○ Dependency triplets
○ Thread position features
● Cohen’s Kappa 0.42
Improvement: use of discussion context
6

14. Project Timeline
● July:
○ Implementation of lag features
○ Implementation of coh-metrix & LIWC features
● August - September:
○ Implementation of Conditional Random Field classifier
○ Coding of EDC (E-Learning and Digital Cultures) MOOC
● October:
○ Evaluation on E-Learning and Digital Cultures MOOC
● November - December:
○ Paper submission & Jisc reporting
7

15. Team
Vitomir Kovanović
1. Third year PhD student (School of Informatics, UoE)
2. Msc Software Engineering
3. Four years industry experience as software developer
4. Best paper award at LAK 2015
Srećko Joksimović
1. Second year PhD student (School of Education, UoE)
2. Msc Information systems
3. Six years industry experience as software developer
Dragan Gašević (supervisor)
1. Professor at the University of Edinburgh (Informatics & Education)
2. President of SOLAR - Society for Learning Analytics research
3. PhD Information Systems 8

16. Thank you

17. References
Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical Inquiry in a Text-
Based Environment: Computer Conferencing in Higher Education. The Internet
and Higher Education, 2(2–3), 87–105.
Kovanović, V., Joksimović, S., Gašević, D., & Hatala, M. (2014). Automated
Content Analysis of Online Discussion Transcripts. In Proceedings of the
Workshops at the LAK 2014 Conference. Indianapolis, IN. Retrieved from http:
//ceur-ws.org/Vol-1137/

18. Automated System for
Cognitive Presence Coding
Vitomir Kovanović, Srećko Joksimović, and Dragan Gašević
THE UNIVERSITY of EDINBURGH
v.kovanovic@ed.ac.uk
@vkovanovicUK Learning Analytics Network
Nottingham,
24th June 2015
http://bit.ly/jisc_coi