Presenation from a Centre for Distance Education seminar 'Writing course materials and formative assessment for successful flexible learning', held at the University of London in June 2014. Conducted by Ormond Simpson, Education Consultant, Visiting CDE Fellow. Audio from the session is available at www.cde.london.ac.uk

1. Writing course materials
for successful flexible learning
Ormond Simpson
Visiting Fellow CDE
CDE seminar - 24 June 2014
1

2. Variations in OU course module retention
T302
T331
MU120
W300
K224
60
65
70
75
80
85
90
95
100
70
% getting to exam
%passingexam
40 50 60 80 90 100
2

3. 3
A successful distance course keeps
a student’s learning motivation switched on

4. Course Design
- possible influences on retention
1. Course Choice
- how students choose their course
2. Course workload
– how much is in the course
3. Course structure
– how the course is organised
4. Course writing
- the writing of the course
5. Course assessment strategies
- how students are assessed
6. Course evaluations
- how courses are assessed
4

5. 1. Course Choice and retention
The second biggest reason
students give for dropping
out (after time issues) is
that they were on the
wrong course.
5

6. 2. Course workload and retention
No link between students’ reported course
workload and dropout - Crooks, 2005
But that’s:
Counter intuitive?
Counter experience?
Possibly a methodology problem?
Or the variety of concepts and media?
6

7. 3. Course structure and retention
‘Flexibility’
- choice of material to study
- choice of time to study
- choice of assessment
Crooks 2005
7

8. John Sweller
‘Cognitive
Load Theory’
John Keller
‘ARCS’ theory John Hattie
‘Self –Reporting’
4. Course writing - theories
Maintaining a student’s motivation to learn

9. Keller’s ARCS theory
A = Get and keep their Attention
R = Ensure everything is Relevant to their needs
C = Ensure they have Confidence in what they’re
doing
S = And that they are continually Satisfied with
their learning experience
9

10. Getting
attention
- Empathy – eg stories
sharing personal thoughts,
acknowledging difficulties
Incongruity – eg humour
Readability – Flesch tests
Relevance
Confidence
Personal,
Approachable,
Using ‘I’
Satisfaction Ensuring students feel progress
Worked example
Ensuring relevance
Avoiding redundancy
Conclusions from theories
Keeping
attention

11. Keller – Keeping Attention – Readability
’Flesch Reading Ease’
Score depends on sentence length and number of syllables
per word
Readability score Interpretations
0 - 20 Very difficult
20 - 50 Difficult
50 – 60 Fairly difficult
60 – 70 Plain English
70 – 80 Fairly easy
80 – 90 Easy
90 - 100 Very easy
11

12. Open Poly of New Zealand course on communication
The field of communication studies runs wide. As a discipline, it
borders on academic specialities such as linguistics, psychology,
media studies, cultural studies, sociology, philosophy, marketing, and
business studies. Its diverse components include interpersonal
communication, intercultural communication, workplace writing,
organisational studies, and mass communication. It can, at times, be
difficult to limit the scope of communication studies: it seems to
involve pretty much most things human beings to together. This is
an indication of the obsession in modern times with communication.
As Peters (1999) notes, communication has been viewed as the
solution to humanity’s diverse and profound troubles. How many
times, for instance, during local or international conflicts have you
heard talk of communication breakdowns, or of the need to open
channels of communication?
In this optimistic view, communication entails connecting with others,
expressing our true selves, alleviating loneliness, preventing
misunderstandings and conflict. The other side of the attention to
communication in the modern world is a focus on the dilemmas and
apparently intractable problems of communication. Peters (1999)
points to the way that so much twentieth century humour focuses on
human miscommunication, where humans are left dazed and lonely
– gesticulating wildly, impotent, ridiculous.
12

13. Open Poly of New Zealand course on communication
The field of communication studies runs wide. As a discipline, it
borders on academic specialities such as linguistics, psychology,
media studies, cultural studies, sociology, philosophy, marketing, and
business studies. Its diverse components include interpersonal
communication, intercultural communication, workplace writing,
organisational studies, and mass communication. It can, at times, be
difficult to limit the scope of communication studies: it seems to
involve pretty much most things human beings do together. This is
an indication of the obsession in modern times with communication.
As Peters (1999) notes, communication has been viewed as the
solution to humanity’s diverse and profound troubles. How many
times, for instance, during local or international conflicts have you
heard talk of communication breakdowns, or of the need to open
channels of communication?
In this optimistic view, communication entails connecting with others,
expressing our true selves, alleviating loneliness, preventing
misunderstandings and conflict. The other side of the attention to
communication in the modern world is a focus on the dilemmas and
apparently intractable problems of communication. Peters (1999)
points to the way that so much twentieth century humour focuses on
human miscommunication, where humans are left dazed and lonely
– gesticulating wildly, impotent, ridiculous.
FRE = 13.5
= Very difficult
13

14. • The field of communication studies runs wide. As a discipline, it borders
on academic specialities such as:
– linguistics,
– psychology,
– media studies,
– cultural studies,
– sociology,
– marketing,
– business studies.
• Its diverse components include:
– interpersonal communication,
– intercultural communication,
– workplace writing,
– organisational studies,
– mass communication.
• It can sometimes be difficult to limit the scope of communication studies.
It seems to involve pretty much most things human beings do together.
This shows the obsession today with communication.
FRE = 62
= Plain English
14

15. 15
Evidence for Keller’s Theory?
‘Motivational measure of the instruction compared:
Instruction Based on the ARCS Motivation Theory
vs Traditional Instruction in Blended Courses’
Ozgur M. COLAKOGLU & Omur AKDEMIR Turkish
Online Journal of Distance Education (TOJDE) 2010

16. Cognitive Load Theory (Sweller, 1998)
Transfer of learning
Initial learning goes into the
working (short) memory and needs to be transferred
to long term memory
16
Learning Working
memory
Long term
memory

17. Three types of cognitive load
1. Intrinsic – due to inherent difficulty of subject.
Should be managed by e.g. ‘segmentation’ and using
worked examples
2. Extraneous – due to way information is presented.
Should be minimised by e.g. Ensuring relevance
avoiding redundancy
3. Germane – due to way info’ relates to previous info’
Should be maximised by making clear links between new
and old information
17

18. ‘Assessing Cognitive Load Theory to improve Student
Learning for Mechanical Engineers’
- Impelluso,V. American Journal of Distance
Education 23 (4) (2009)
- claimed increased retention and learning
18

19. 5. Course assessment strategies
‘Assessment drives learning’
But…
Does assessment also drive dropout?
19

20. 1
100
62 57 52
38
43
48
72
5
ASSIGNMENT RIVERGRAM
100 students start the course. At each assignment some drop out and enter
the ‘exit’ channel. A very few re-enter the ‘progress’ channel having skipped
the previous assignment
Assignment 1 Assignment 2 Assignment 3
Progress
Exit

21. Evidence - what does research say about
assessment and retention?
(1) Hattie (2009) most important is ‘self
reporting of grades’ – so a student knows how
well (s)he is doing
21

22. ‘Self reporting of grades’
Put ‘self-assessment questions’ in text
- but how to get students to do them?
1. Tell them
2. Nudging – “most students do the SAQ’s”
“research has shown that students
who do the SAQ’s do best in the course”
3. Electronic text – make it impossible to move
on unless the SAQ is attempted
22

23. E-teaching
- can be provided in two ways
Course texts in print.
Teaching / support online
Everything online
including course texts
Reading off a screen can be 30% less effective
than reading paper copy
http://www.scientificamerican.com/article/reading-
paper-screens/

24. 24
E-teaching
Advantages of e-teaching – can use many different kinds
of media – forums, podcasts, video/audio clips, blogs etc
Disadvantages of e-teaching – can use many different
kinds of media….
‘Course Exuberance Syndrome’

25. ‘Learning value’ vs. ‘learning time needed’
Learning value
poor
Learning value
rich
Learning time
needed short
Learning time
long
Paper
text
Online
text
Computer
forums
blogs
podcasts
Videoclips
Wikis
Facebook
‘Use line’
Secondlife
email
‘Learning
time
needed’
‘Learning
value’
pencasts
Audioclips
ebooks

26. 26
6. Course evaluations
1. Asking the students
2. Data analysis

27. 27
Asking the students
OU Courses survey

28. ‘Crash testing’
28

29. 29
Module title
Students
started
Students
passed
Predicted
pass
Actual
pass
Z-score
‘Creative writing’ 1,995 1,615 77.2% 81.0% + 4.38
‘Introducing
religions’
355 254 62.9% 71.5% + 3.71
‘Medicine and
society in Europe
1500-1930’
357 227 69.6% 63.6% - 2.80
‘Exploring the
classical world’
605 414 68.9% 68.4% - 0.29
‘Exploring
philosophy’
699 500 65.3% 71.5% + 3.84
‘Inside music’ 413 255 74.0% 61.7% - 6.34
Z-scores for a sample of OU modules
Z-scores outside the range +4 or -4 are considered significant

30. Variations in course retention
T302
T331
MU120
W300
K224
60
65
70
75
80
85
90
95
100
70
% getting to exam
%passingexam
Sifters
Heavy goings
Fair
Knock Backers
40 50 60 80 90 100
30

31. Barriers to ‘retention-friendly’ text?
31
Material focused
Exclusive
Highly challenging
Serious
Student focused
Inclusive
Aptly challenging
Enjoyable
Weeding out the unfit
– ‘Darwinistas’
Sink or swim
- ‘Fatalistas’
Help students be the
best they can be
- Retentioneer

32. Thank you!
More material on
www.ormondsimpson.com
32

33. 33

34. 4. Motivational and ungraded feedback
(i) Motivational feedback (Wigfield et al, 2009):
Learning motivation
= (assumed possibility of accomplishing task)
x (perceived value of task)
If either factor is zero then motivation is zero
So feedback tasks should be carefully graded for
difficulty
34

35. Formative assessment – the evidence
• Yorke and Longden (2004)
• Black and Wiliam (1998)
• Kluger (1996)
• Gibbs and Dunbar-Goddet (2007)
35

36. Without external formative assessment 1
100
62 57 52
38
43
48
7
2
5
ASSIGNMENT RIVERGRAM – FORMATIVE ASSESSMENT
100 students start the course. At each assignment some drop out and enter the ‘exit’ channel. A very few
re-enter the ‘progress’ channel having skipped the previous assignment
Assignment 1 Assignment 2 Assignment 3
Progress
Exit
36

37. With external formative assessment
100
72 70 65
28
30
35
11
9
5
ASSIGNMENT RIVERGRAM – FORMATIVE ASSESSMENT
100 students start the course. At each assignment some drop out and enter the ‘exit’ channel. A
very few re-enter the ‘progress’ channel having skipped the previous assignment.
Assignment 1 Assignment 2 Assignment 3
Progress
Exit
37

38. Keller’s A = Attention
1. Getting students attention
2. Keeping students attention
38

39. Keller - Getting Attention
Use:
- incongruity – eg humour*
- empathy – eg stories, sharing
personal thoughts
- authority – displaying expertise
But everything in a course text should be
relevant to learning.
*eg see ‘Flipnosis’ – Dutton (2011)
39

40. Keller - Keeping attention
- Reading skills and reading habits
• Researchers used ‘Cloze’ tests (replacing blanks in
text) on new students
– many would have significant difficulties in
understanding their course material.
- 42% new students had lower comprehension than
needed for courses
- Students’ previous reading was newspapers and
magazines
- Datta and Macdonald Ross (2002)
40

41. Readability Scales
• Automated Readability Index
• Flesch Reading Ease
• Flesch-Kincaid scores
• Gunning-Fog index
• SMOG index (simplified measure of
gobbledygook)
41

42. Newspapers Flesch Reading Ease score
Sun 62.8 – Plain English
Daily Mail 61.5 – Plain English
Mirror 60.5 – Plain English
Guardian 44.5 – Difficult
Telegraph 48.8 – Difficult
42

43. OU courses
initial pages
Flesch Reading Ease score
Arts Foundation 47.1 – difficult
Social science
Foundation
55.2 – Fairly difficult
Maths Foundation 39.9 - difficult
Science Foundation 53.7 – Fairly difficult
OU Access Course 58.1 – Fairly difficult
Significant differences in readability levels between
tabloid newspapers and UKOU courses - Moore (2004)
43

44. Fonts
• Times New Roman
• Arial
• Helvetica
• Courier
• Comic
• Calibri
• Cambria
• Brush
• Kumasi Pioneer
Two groups of students
were given exercise advice
in Arial and Brush fonts.
Students getting Arial
text were more likely to
change exercise behaviour
than students getting the
Brush text
Herbert - Sci. Amer.
Mind Feb 2009
44

45. Type layout - Justifying 1
There appears to have been relatively little work done on
course design and retention. Indeed designing a course
for better retention rates is a complex concept since
course design is intimately bound up with course
assessment. In recent years in the UK the annual festival
of punditry revolves around the question of whether
more students passing the ‘A’ level exams means that
‘standards’ have therefore fallen. Thus the retention
debate has tended to avoid questions of the difficulty of
content and concepts in the course and concentrated on
other possibilities for increasing retention such as course
workload, course readability and course design and
structure.
45

46. Justifying 2
There appears to have been relatively little work done on
course design and retention. Indeed designing a course
for better retention rates is a complex concept since
course design is intimately bound up with course
assessment.
In recent years in the UK an annual festival of punditry
revolves around the question of whether more students
passing the ‘A’ level exams means that ‘standards’ have
therefore fallen.
Thus the retention debate has tended to avoid questions
of the difficulty of content and concepts in the course and
concentrated on other possibilities for increasing retention
such as course workload, course readability and course
design and structure.
46

47. Using columns
There appears to have been
relatively little work done on
course design and retention.
Indeed designing a course
for better retention rates is a
complex concept since
course design is intimately
bound up with course
assessment.
In recent years in the UK an
annual festival of punditry
revolves around the
question of whether more
students passing the ‘A’ level
exams means that
‘standards’ have therefore
fallen.
Thus the retention debate
has tended to avoid
questions of the difficulty of
content and concepts in the
course and concentrated on
other possibilities for
increasing retention such as
course
47

48. Newspapers
o Different fonts
o Narrow columns
o Short paragraphs
o Justified – sometimes ragged right
o Many graphics
o Frequent subheadings and quote boxes
o Variable fonts
o Colour and shading
- all aimed at ‘Attention’
48

49. R = Relevance
• Avoid redundancy
•
Avoid ‘split attention’
• - see ‘Cognitive Load Theory’
49

50. C = Confidence
• Why are texts largely anonymous?
- could we use ‘I’ or ‘we’?
• Why little information about authors?
– should we ‘personalise’ texts?
50

51. S = Satisfaction
Feedback to student - Hattie - most
important is ‘self reporting’ – so a
student knows how well (s)he is doing’
So use self assessment questions?
• put answers straight after questions
• keep questions short so students can
see progress
51

52. Types of assessment
1. Formative - to enhance learning, not to
allocate grades
- usually ungraded
- feedback for student’s benefit
2. Summative – graded
- counts towards final grade
- continuous or final exam
3. Ipsative - iterative feedback
52

53. ‘Meta-cognitive awareness’
(3) Gibbs (2010)
– marking exercises
- enhanced worked examples
Some indirect evidence
for increased retention
53

54. Gibbs – (1) Marking exercises
Students given specimen student assignment
answers and asked to mark them
54

55. 55
Graham Gibbs - Marking exercise
F2F GROUP ACTIVITY 2: What do 'they' want? (25 mins)
Give students copies of a short student assignment.
1. Tell them, “Imagine you are a tutor - read this quickly and mark it.
Give it a grade (100 to 0) and helpful comments." (5 mins)
2. In pairs, “Compare your comments and grading together. What
have you picked out as important in your comments?“ (10 mins)
3. In plenary, “Right let's now try to draw out some good and bad
points from this to help us in our own writing". (10 mins)

56. Gibbs - (2) Enhanced worked examples
“Last week I found 50 New Zealand dollars in my
suitcase.
“So I rushed down to the Post Office where there was
one of those illuminated signs in the window.
Against NZ$ it said ‘We buy at NZ$2.72 to the £’.
So how much did the man behind the counter give me?”
56

57. Enhanced worked example - answer
“The statement ‘We buy at NZ$2.72 to the £’ means
that for every NZ$2.72 I give the man he’ll give me
£1.
I've got NZ$50 so we need to know how many times
2.72 goes into 50 to see how many £’s I'll get.
Now 50/2.72 = 18.51851852 on my calculator. So the
man gave me £18.51 as the post office doesn’t deal in
less than a penny of course.”
57

58. Enhanced worked example - answer
“The statement ‘We buy at NZ$2.72 to the £’ means that
for every NZ$2.72 I give the man he’ll give me £1. (It
helps to write out what the statement ‘We buy at NZ$2.72
to the £’ means in practice).
I've got NZ$50 so we need to know how many times 2.72
goes into 50 to see how many £’s I'll get. (remember to
explain each step rather than just write down the calculation
like 'NZ$ = £50/2.17' The more you explain what you're
doing, the better).
Now 50/2.72 = 18.51851852 on my calculator. So the man
gave me £18.51 as the post office doesn’t deal in less than
a penny of course.” (Be sure your answer makes sense – it's
very easy when using the calculator to write down whatever it
tells you!) 58

59. With external formative assessment
100
72 70 65
28
30
35
11
9
5
ASSIGNMENT RIVERGRAM – FORMATIVE ASSESSMENT
100 students start the course. At each assignment some drop out and enter the ‘exit’ channel. A
very few re-enter the ‘progress’ channel having skipped the previous assignment.
Assignment 1 Assignment 2 Assignment 3
Progress
Exit
59

60. ‘Ipsative assessment’
- a new approach?
• Assessment or feedback which compares a
student’s achievement not with an absolute
standard, but with their previous performance
(Hughes, 2011)
• For example students can repeat an assessment
with feedback at each attempt until they reach
the required standard
• No retention results as yet
60