Staff and student evaluation of a new learning management system

1. Crossing the divide: Student and
staff responses to changing an
institutional learning management
system
Stuart Palmer (@s_palm ) and Dale Holt
Deakin University

2. … or:
Part past part (non-)fiction
Stuart Palmer (@s_palm ) and Dale Holt
Deakin University

3. At Deakin University
WebCT LMS trialled in 2003
All content migrated in by 2004
D2L selected as replacement in 2010
Phased cut-over in 2011
New LMS fully online 2012

4. Students 2004-2011
1
5
7
8
12
11 10
5.0
4.5
4.0
3.5
3.0
2.5
2.0
2.5 3.0 3.5 4.0 4.5
Importance
Satisfaction
♦ 2004
• 2005
2011
2
6
4
3
9
13
5.0
4.5
4.0
3.5
3.0
2.5
2.0
2.5 3.0 3.5 4.0 4.5
Importance
Satisfaction
♦ 2004
• 2005
2011
1 Accessing unit guide and other unit information 2 Accessing unit lecture, tutorial or lab notes etc.
3 Interacting with unit learning resources 4 Using the unit calendar
5 Contacting teachers via internal unit messaging 6 Contacting students via internal unit messaging
7 Reading contributions to online discussions 8 Contributing to online discussions
9 Completing online quizzes/tests 10 Submitting assignments
11 Receiving feedback on assignments 12 Working collaboratively in a group
13 Reviewing unit progress

5. Staff 2004-2011
1
4
3
7
11
12
9
6
5.0
4.5
4.0
3.5
3.0
2.5
2.0
1 Accessing unit guide and other unit information 2 Accessing unit lecture, tutorial or lab notes etc.
3 Interacting with unit learning resources 4 Using the unit calendar
5 Contacting teachers via internal unit messaging 6 Contacting students via internal unit messaging
7 Reading contributions to online discussions 8 Contributing to online discussions
9 Completing online quizzes/tests 10 Submitting assignments
11 Receiving feedback on assignments 12 Working collaboratively in a group
13 Reviewing unit progress
5.0
4.5
4.0
3.5
3.0
2.5
2.0
1.5
2.0 2.5 3.0 3.5 4.0 4.5
Importance
Satisfaction
♦ 2004
• 2005
2011
2
5
8
10
13
1.5
2.0 2.5 3.0 3.5 4.0 4.5
Importance
Satisfaction
♦ 2004
• 2005
2011

6. The LMS WILL change, b/c
Improving online learning and teaching
Competitive advantage via innovation
A new direction for learning and teaching
Adoption of open source IT systems
Expiry of current system vendor contract
Improving back-office efficiency
Consolidating institutional IT systems
Acquisition of the current vendor by a competitor

7. Students 2011-2012
Students 2011 Students 2012
Sample Population Sample Population
No. of respondents 1353 22760 1710 24805
Response rate 5.95% — 6.89% —
Gender
Female 67.3% 59.5% 70.5% 59.9%
Male 32.7% 40.5% 29.5% 40.1%
Faculty
Arts and Education 29.8% 30.4% 30.5% 31.3%
Business and Law 28.8% 36.9% 26.6% 35.1%
Health 22.8% 19.8% 26.3% 20.1%
Science & Technology 17.0% 12.9% 15.6% 13.5%
Other 1.6% — 1.1% —
Campus
Burwood 43.7% 50.3% 42.0% 49.8%
Waurn Ponds 14.3% 12.1% 14.3% 12.5%
Waterfront 7.4% 5.8% 7.3% 6.0%
Warrnambool 3.3% 3.2% 4.4% 3.0%
Off-campus 31.3% 28.6% 32.1% 28.7%
Mean enrolment duration 2.42 years 2.24 years 2.33 years 2.22 years

8. Staff 2011-2012 Staff 2011 Staff 2012
Sample Population Sample Population
No. of respondents 263 2126 330 2291
Response rate 12.37% — 14.40% —
Gender
Female 55.9 % 59.8 % 61.8 % 59.8 %
Male 44.1 % 40.2 % 38.2 % 40.2 %
Faculty
Arts and Education 30.3% — 33.0% 24.3%
Business and Law 26.3% — 20.3% 20.6%
Health 29.2% — 32.4% 37.8%
Science & Technology 12.4% — 13.3% 13.9%
Other 1.8% — 0.9% 3.3%
Age range
<25 3.0% 2.8% 1.5% 2.7%
25-29 4.9% 9.1% 7.0% 6.5%
30-34 5.7% 14.5% 9.7% 12.2%
35-39 13.3% 13.3% 9.4% 13.7%
40-44 16.0% 14.8% 15.8% 14.1%
45-49 15.6% 14.4% 19.1% 14.5%
50-54 17.9% 14.3% 14.5% 14.2%
55-59 10.6% 10.0% 11.8% 11.6%
60-64 9.1% 6.0% 5.8% 7.5%
>64 3.4% 0.9% 5.5% 2.9%
Not indicated 0.4% — — —

9. Students 2011-2012
1 Accessing unit guide and other unit information
2 Accessing unit lecture, tutorial or lab notes etc.
3 Interacting with unit learning resources
4 Using the unit calendar
5 Reading unit News announcements
6 Contacting teachers via internal unit messaging
7 Contacting students via internal unit messaging
8 Reading contributions to online discussions
9 Contributing to online discussions
10 Completing online quizzes/tests
11 Submitting assignments
12 Receiving feedback on assignments
13 Working collaboratively in a group
14 Reviewing unit progress
15 Using the e-portfolio
16 eLive synchronous communication session
17 iLecture class lecture recording
18 Other iLecture recording – podcasts, etc.
19 Deakin's social software – MediaWiki, etc
20 Turnitin plagiarism detection
1
2
3
4
5
6
7
8
11
10
9
12
13
14
15
16
17
18
19
20
5.0
4.5
4.0
3.5
3.0
2.5
3.00 3.25 3.50 3.75 4.00 4.25
Importance
Satisfaction
♦ No Sig. Difference
• 2011
2012

10. Students 2011-2012
1 Accessing unit guide and other unit information
2 Accessing unit lecture, tutorial or lab notes etc.
3 Interacting with unit learning resources
4 Using the unit calendar
5 Reading unit News announcements
6 Contacting teachers via internal unit messaging
7 Contacting students via internal unit messaging
8 Reading contributions to online discussions
9 Contributing to online discussions
10 Completing online quizzes/tests
11 Submitting assignments
12 Receiving feedback on assignments
13 Working collaboratively in a group
14 Reviewing unit progress
15 Using the e-portfolio
16 eLive synchronous communication session
17 iLecture class lecture recording
18 Other iLecture recording – podcasts, etc.
19 Deakin's social software – MediaWiki, etc
20 Turnitin plagiarism detection
1
2
3
4
10
9
16
17
18
19
5.0
4.5
4.0
3.5
3.0
2.5
3.00 3.25 3.50 3.75 4.00 4.25
Importance
Satisfaction
♦ No Sig. Difference
• 2011
2012

11. Students 2011-2012
1
2
3
4
5
7
10
9
14
15
16
17
18
19
5.0
4.5
4.0
3.5
3.0
2.5
3.00 3.25 3.50 3.75 4.00 4.25
Importance
Satisfaction
♦ No Sig. Difference
• 2011
2012
1 Accessing unit guide and other unit information
2 Accessing unit lecture, tutorial or lab notes etc.
3 Interacting with unit learning resources
4 Using the unit calendar
5 Reading unit News announcements
6 Contacting teachers via internal unit messaging
7 Contacting students via internal unit messaging
8 Reading contributions to online discussions
9 Contributing to online discussions
10 Completing online quizzes/tests
11 Submitting assignments
12 Receiving feedback on assignments
13 Working collaboratively in a group
14 Reviewing unit progress
15 Using the e-portfolio
16 eLive synchronous communication session
17 iLecture class lecture recording
18 Other iLecture recording – podcasts, etc.
19 Deakin's social software – MediaWiki, etc
20 Turnitin plagiarism detection

12. Students 2011-2012
1 Accessing unit guide and other unit information
2 Accessing unit lecture, tutorial or lab notes etc.
3 Interacting with unit learning resources
4 Using the unit calendar
5 Reading unit News announcements
6 Contacting teachers via internal unit messaging
7 Contacting students via internal unit messaging
8 Reading contributions to online discussions
9 Contributing to online discussions
10 Completing online quizzes/tests
11 Submitting assignments
12 Receiving feedback on assignments
13 Working collaboratively in a group
14 Reviewing unit progress
15 Using the e-portfolio
16 eLive synchronous communication session
17 iLecture class lecture recording
18 Other iLecture recording – podcasts, etc.
19 Deakin's social software – MediaWiki, etc
20 Turnitin plagiarism detection
1
2
3
4
5
6
7
8
11
10
9
12
13
14
15
16
17
18
19
5.0
4.5
4.0
3.5
3.0
2.5
3.00 3.25 3.50 3.75 4.00 4.25
Importance
Satisfaction
♦ No Sig. Difference
• 2011
2012

13. Students 2011-2012
1 Accessing unit guide and other unit information
2 Accessing unit lecture, tutorial or lab notes etc.
3 Interacting with unit learning resources
4 Using the unit calendar
5 Reading unit News announcements
6 Contacting teachers via internal unit messaging
7 Contacting students via internal unit messaging
8 Reading contributions to online discussions
9 Contributing to online discussions
10 Completing online quizzes/tests
11 Submitting assignments
12 Receiving feedback on assignments
13 Working collaboratively in a group
14 Reviewing unit progress
15 Using the e-portfolio
16 eLive synchronous communication session
17 iLecture class lecture recording
18 Other iLecture recording – podcasts, etc.
19 Deakin's social software – MediaWiki, etc
20 Turnitin plagiarism detection
1
2
3
4
5
6
7
8
11
10
9
12
13
14
15
16
17
18
19
20
5.0
4.5
4.0
3.5
3.0
2.5
3.00 3.25 3.50 3.75 4.00 4.25
Importance
Satisfaction
♦ No Sig. Difference
• 2011
2012

14. Staff 2011-2012
1 Accessing unit guide and other unit information
2 Accessing unit lecture, tutorial or lab notes etc.
3 Interacting with unit learning resources
4 Using the unit calendar
5 Reading unit News announcements
6 Contacting teachers via internal unit messaging
7 Contacting students via internal unit messaging
8 Reading contributions to online discussions
9 Contributing to online discussions
10 Completing online quizzes/tests
11 Submitting assignments
12 Receiving feedback on assignments
13 Working collaboratively in a group
14 Reviewing unit progress
15 Using the e-portfolio
16 eLive synchronous communication session
17 iLecture class lecture recording
18 Other iLecture recording – podcasts, etc.
19 Deakin's social software – MediaWiki, etc
20 Turnitin plagiarism detection
21 Deakin's Learning Repository – Equella
22 Respondus online quiz authoring tool
2
1
3
4
5
6
8 9
7
10
11
12
13 14
15
16
17
18
19
20
21
22
5.0
4.5
4.0
3.5
3.0
2.5
2.0
2.75 3.00 3.25 3.50 3.75 4.00 4.25
Importance
Satisfaction
♦ No Sig. Difference
• 2011
2012

15. Staff 2011-2012
1 Accessing unit guide and other unit information
2 Accessing unit lecture, tutorial or lab notes etc.
3 Interacting with unit learning resources
4 Using the unit calendar
5 Reading unit News announcements
6 Contacting teachers via internal unit messaging
7 Contacting students via internal unit messaging
8 Reading contributions to online discussions
9 Contributing to online discussions
10 Completing online quizzes/tests
11 Submitting assignments
12 Receiving feedback on assignments
13 Working collaboratively in a group
14 Reviewing unit progress
15 Using the e-portfolio
16 eLive synchronous communication session
17 iLecture class lecture recording
18 Other iLecture recording – podcasts, etc.
19 Deakin's social software – MediaWiki, etc
20 Turnitin plagiarism detection
21 Deakin's Learning Repository – Equella
22 Respondus online quiz authoring tool
2
1
3
11 8 9 5
12
13 14
15
16
17
18
19
21
22
5.0
4.5
4.0
3.5
3.0
2.5
2.0
2.75 3.00 3.25 3.50 3.75 4.00 4.25
Importance
Satisfaction
♦ No Sig. Difference
• 2011
2012

16. Staff 2011-2012
1 Accessing unit guide and other unit information
2 Accessing unit lecture, tutorial or lab notes etc.
3 Interacting with unit learning resources
4 Using the unit calendar
5 Reading unit News announcements
6 Contacting teachers via internal unit messaging
7 Contacting students via internal unit messaging
8 Reading contributions to online discussions
9 Contributing to online discussions
10 Completing online quizzes/tests
11 Submitting assignments
12 Receiving feedback on assignments
13 Working collaboratively in a group
14 Reviewing unit progress
15 Using the e-portfolio
16 eLive synchronous communication session
17 iLecture class lecture recording
18 Other iLecture recording – podcasts, etc.
19 Deakin's social software – MediaWiki, etc
20 Turnitin plagiarism detection
21 Deakin's Learning Repository – Equella
22 Respondus online quiz authoring tool
2
1
3
5
6
8 9
7
10
11
12
13 14
15
16
17
18
19
21
22
5.0
4.5
4.0
3.5
3.0
2.5
2.0
2.75 3.00 3.25 3.50 3.75 4.00 4.25
Importance
Satisfaction
♦ No Sig. Difference
• 2011
2012

17. Staff 2011-2012
1 Accessing unit guide and other unit information
2 Accessing unit lecture, tutorial or lab notes etc.
3 Interacting with unit learning resources
4 Using the unit calendar
5 Reading unit News announcements
6 Contacting teachers via internal unit messaging
7 Contacting students via internal unit messaging
8 Reading contributions to online discussions
9 Contributing to online discussions
10 Completing online quizzes/tests
11 Submitting assignments
12 Receiving feedback on assignments
13 Working collaboratively in a group
14 Reviewing unit progress
15 Using the e-portfolio
16 eLive synchronous communication session
17 iLecture class lecture recording
18 Other iLecture recording – podcasts, etc.
19 Deakin's social software – MediaWiki, etc
20 Turnitin plagiarism detection
21 Deakin's Learning Repository – Equella
22 Respondus online quiz authoring tool
2
1
3
4
5
6
8 9
7
10
11
12
13 14
15
16
17
18
19
20
21
22
5.0
4.5
4.0
3.5
3.0
2.5
2.0
2.75 3.00 3.25 3.50 3.75 4.00 4.25
Importance
Satisfaction
♦ No Sig. Difference
• 2011
2012

18. Students 2011-2012
1 Accessing unit guide and other unit information
2 Accessing unit lecture, tutorial or lab notes etc.
3 Interacting with unit learning resources
4 Using the unit calendar
5 Reading unit News announcements
6 Contacting teachers via internal unit messaging
7 Contacting students via internal unit messaging
8 Reading contributions to online discussions
9 Contributing to online discussions
10 Completing online quizzes/tests
11 Submitting assignments
12 Receiving feedback on assignments
13 Working collaboratively in a group
14 Reviewing unit progress
15 Using the e-portfolio
16 eLive synchronous communication session
17 iLecture class lecture recording
18 Other iLecture recording – podcasts, etc.
19 Deakin's social software – MediaWiki, etc
20 Turnitin plagiarism detection
1
2
3
4
5
6
7
8
11
10
9
12
13
14
15
16
17
18
19
20
5.0
4.5
4.0
3.5
3.0
2.5
3.00 3.25 3.50 3.75 4.00 4.25
Importance
Satisfaction
♦ No Sig. Difference
• 2011
2012

19. Students 2011-2012
1 Accessing unit guide and other unit information
2 Accessing unit lecture, tutorial or lab notes etc.
3 Interacting with unit learning resources
4 Using the unit calendar
5 Reading unit News announcements
6 Contacting teachers via internal unit messaging
7 Contacting students via internal unit messaging
8 Reading contributions to online discussions
9 Contributing to online discussions
10 Completing online quizzes/tests
11 Submitting assignments
12 Receiving feedback on assignments
13 Working collaboratively in a group
14 Reviewing unit progress
15 Using the e-portfolio
16 eLive synchronous communication session
17 iLecture class lecture recording
18 Other iLecture recording – podcasts, etc.
19 Deakin's social software – MediaWiki, etc
20 Turnitin plagiarism detection
1
2
3
4
5
6
7
8
11
10
9
12
13
14
15
16
17
18
19
20
5.0
4.5
4.0
3.5
3.0
2.5
3.00 3.25 3.50 3.75 4.00 4.25
Importance
Satisfaction
♦ No Sig. Difference
• 2011
2012

20. Students 2011-2012
1 Accessing unit guide and other unit information
2 Accessing unit lecture, tutorial or lab notes etc.
3 Interacting with unit learning resources
4 Using the unit calendar
5 Reading unit News announcements
6 Contacting teachers via internal unit messaging
7 Contacting students via internal unit messaging
8 Reading contributions to online discussions
9 Contributing to online discussions
10 Completing online quizzes/tests
11 Submitting assignments
12 Receiving feedback on assignments
13 Working collaboratively in a group
14 Reviewing unit progress
15 Using the e-portfolio
16 eLive synchronous communication session
17 iLecture class lecture recording
18 Other iLecture recording – podcasts, etc.
19 Deakin's social software – MediaWiki, etc
20 Turnitin plagiarism detection
1
2
3
4
5
6
7
8
11
10
9
12
13
14
15
16
17
18
19
20
5.0
4.5
4.0
3.5
3.0
2.5
3.00 3.25 3.50 3.75 4.00 4.25
Importance
Satisfaction
♦ No Sig. Difference
• 2011
2012

21. Students 2011-2012
1 Accessing unit guide and other unit information
2 Accessing unit lecture, tutorial or lab notes etc.
3 Interacting with unit learning resources
4 Using the unit calendar
5 Reading unit News announcements
6 Contacting teachers via internal unit messaging
7 Contacting students via internal unit messaging
8 Reading contributions to online discussions
9 Contributing to online discussions
10 Completing online quizzes/tests
11 Submitting assignments
12 Receiving feedback on assignments
13 Working collaboratively in a group
14 Reviewing unit progress
15 Using the e-portfolio
16 eLive synchronous communication session
17 iLecture class lecture recording
18 Other iLecture recording – podcasts, etc.
19 Deakin's social software – MediaWiki, etc
20 Turnitin plagiarism detection
1
2
3
4
5
6
7
8
11
10
9
12
13
14
15
16
17
18
19
20
5.0
4.5
4.0
3.5
3.0
2.5
3.00 3.25 3.50 3.75 4.00 4.25
Importance
Satisfaction
♦ No Sig. Difference
• 2011
2012

22. Key findings
Did the new LMS dramatically change the
student and staff experience of using the
system? To a large extent, the answer is, “No.”
Some observed significant differences were ‘as
expected’

23. Other observations
University LMS users should expect that their
current system will be retired at some point
There is a need for on-going monitoring
It can take many years before significant
increases in perceptions of LMS value emerge

24. In the doledrums
On the (d)O.L.E.
Thank for your time
ow.ly/CiYXm