Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Are assessment scores good proxies of estimating learning gains: a large-scale study amongst humanities and science students
1. Are assessment scores good
proxies of estimating learning
gains: a large-scale study
amongst humanities and
science students
Dr. Jekaterina Rogaten
Prof Bart Rienties
https://twitter.com/LearningGains
https://abclearninggains.com/
3. What students think they gain?
I think I am more openly critical
(in the positive sense)
Day to day when I have my book I have very different
approach from recording my notes for example
[in my new job], there will reports and
planning to be drawn and I think that this will
be an aspect of my job where I can say yes
the OU study and discipline I’ve received
from the OU has actually contributed to that.
I observe things better, work into deeper and
work on the whole picture rather than narrow.
I think more logically and more ‘why did that
happen, why did that happen’, there is more
questioning, instead of just to accept things.
I am much better at time management, I am much more
organised now and planning things in advance.
now I say, ‘you know what, I can do that in future’.
I feel more confident and I am happier
because I am doing something I have always
wanted to be doing and something that
interests me
I think I can go confidently to
speak what I learned. But
even to a job that isn’t directly
related to this subject area. I
could talk about my
experiences, my time
management, team working,
computer skills as I feel much
more confident, I can say,
‘actually I have done this’.
Which was one of the
reasons I wanted to a degree.
4. Do grades matter?
How well do your grades represent your progress?
probably in the same way that many other people when
they look at their own assignment results and exam results
…. I feel that I am doing fairly well but I’d always like to
improve myself to my results.
I get quite upset when I get around 70s
… because I am putting so much effort I want my grades to reflect it.
They usually go up. But it is Marginal. 5 marks across all the TMAs
that’s the variance, it just varies very slightly
Even if it is 1-2 marks I say what did I do differently and
I go back to tutor to see what did I do differently. What
happened, what caused it?
Well there are questions with the text books, exercises. So
if I get correct answer, I know I am doing fine. When I say
correct answer that’s not the end product that’s the whole
answer check through it
“I suppose you could say… the skills you learn, like group work, presenting and being able to talk to people…
I would say the main way that you think about [achievement], it’s just the grade because… that’s what is going
on your CV… and affect what job you get. … I’d say the skills you learn as well as becoming an all-rounded person
are quite important as well”.
5. Do students make learning gains, and
what is the power of LA to predict
learning gains?
Using assessment results for estimating learning gains has number of
advantages:
Assessment data readily available
Widely recognized as appropriate measure of learning
Relatively free from self-reported biases
Allows a direct comparison of research finding with the results of
other studies
Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, Estonia
8. 1st year STEM students
Model 1 S.E. Model 2 S.E.
Fixed Part
Intercept 74.52 4.22 75.14 4.21
B1 Slope 0.58 1.39 0.58 1.39
B2 Black -6.86** 0.97
B3 Asian -4.67** 0.79
B4 Mixed/Other ethnicity -2.64** 0.86
B5 Unknown ethnicity -1.05 1.02
B6 HE/PG qualification 1.81** 0.36
B7 Lower than A level / No
formal qualification
-2.08** 0.36
B8 Unknown qualification -1.80* 0.79
B9 Low SES -1.58** 0.42
B10 Unknown/Not
Applicable SES
0.94 0.58
9. How do STEM students compare to
Social Science students
Participants
5,791 Science students of whom 58.2% were females and 41.8% were
males with average age of M = 29.8, SD = 9.6.
11,909 Social Science students of whom 72% were females and 28%
were males with average age of M = 30.6, SD = 9.9
Measures
Tutor Marked Assessments (TMA)
Across 111 modules
11. How do STEM students compare to
Social Science students
Social Science Science
Variance in students’ initial
achievements
6% 33%
variance in students’ learning
gains
19% 26%
VPC module 4% 25%
VPC student 56% 51%
VPC TMAs 40% 22%
12. Effect of socio-demographic factors
Variable Social Science
Beta
Model1
Beta
Model2
Beta
Mode 3
Gender 0.64*
Unknown -2.4*
Other -6.68**
Mixed -3.27**
Asian -4.66**
Black -7.99**
HE qualification 0.29
Lower than A levels -3.11**
No formal qualification -6.93**
PG qualification 2.62**
Variance explained in learning gains 0.1% 3% 3.1%
Variable Science
Beta
Model 1
Beta
Model 2
Beta
Model 3
Gender 0.29
Unknown 1.76
Other -4.09
Mixed -0.59
Asian -7.31**
Black -13.07**
HE qualification 2.64**
Lower than A levels -4.59**
No formal qualification -9.48**
PG qualification 8.19**
Variance explained in learning
gains
0.3% 2.2% 3%
14. Social Science Science
A levels or equivalent
HE Qualification
Lower than A levels
No formal qualification
PG qualification
15. University 1
1,990 undergraduate students
University 2
1,547 undergraduate students
20 degree programmes within each university
DV – average grade yearly grade
University 1 University 2
Year M SD M SD
1 60.65 7.62 63.75 12.66
2 61.31 6.81 65.64 12.74
3 63.32 6.63 64.12 14.02
Can we use this approach to look
across different universities?
18. Variance partitioning
University 1 University 2
Variance at Department level 13.1% 22%
Variance between students 59.8% 22%
Variance within students (between years) 27.1% 56%
19. Summary of findings
Although both universities overall showed positive gains,
substantial differences were present in variance at departmental
level.
Aggregate learning gains estimates can result in misleading
estimates of students’ learning gains on a discipline or degree
level.
Multilevel modelling is a more accurate method in comparison
with simple linear models when estimating students’ learning
gains.
20. Possible implications
Support for subject level TEF
Guidance on where to focus interventions and resources
Visualisation could promote data informed learning
design decisions
Questions still to answer:
Does grade trajectory reflect students’ learning gains?
Can we make a meaningful comparison between
universities?
What impact grade trajectory has on students? Do
they need to know their own trajectory and how it
compares to others?
Assessment = already collected metrics as opposed to new measures
Level 1 – Grade: repeated measures on students and tell us about students learning trajectory
Level 2 – student: between students variations
Level 3 – Course: between course variation
10 - modules
10 - modules
Student-level intercept-slope correlation - Students with low initial achievements show high learning gains that students with initial high achievements.
Module level intercept-slope correlation Students in modules with the low initial achievements show higher learning gains than students in courses with the high initial achievements
VPC – is variance partition coefficient. VPC module shows variance that can be attributed to the differences between modules. VPC student shows the mount of variance that can be attributed to the differences between students, and VPC TMA shows variance within each student across assessments. In this last one we can see that it is almost double in Social Science hence further supports that assessments play important role.
Gender: male-female learning gains gap (male reference group)
Other-white gain gap
Mixed-White gain gap
Asian-White gain gap
Black-white gain gap
HE qualification - A levels gain gap
lower that A levels - A levels gain gap
No formal qualification – A levels gain gap
PG qualification - A levels gain gap
On average students showed improvement in standardised grades, although this was only significant for University 1
University 1 variance was mainly nested between students
University 2 had more variance at the departmental level and within students
The negative learning gains seen in some students in University 2 does not imply that these students are losing knowledge or ability per se. However it highlights the complexity of factors that have to be taken into account when using students’ academic performance as a proxy for learning gains, such as ‘assessment difficulty’ and ‘learning design’ (Rienties & Toetenel, 2016). We need to understand these if we are to use academic performance as a proxy for learning gains within Higher Education.
We set out to see if multilevel modelling of assessment data could be a valid and useful proxy for learning gain
Aggregate learning gains estimates can result in misleading estimates of students’ learning gains on a discipline or degree level. (or show no gain hiding a more complex picture – University 2)
Multilevel modelling is a more accurate method in comparison with simple linear models when estimating students’ learning gains. The simple models are not able to detect differences between modules when looking at the department and degree level performance, whereas multilevel modelling can.
The HEFCE funded programme of which this project is a part wants to devise a sector wide APPROACH (not method) and characterisation of learning gain, and explore possible proxies.
The contribution of the ABC project is that if we are to use LG research for optimising learning (LAK definition) then we need to use a methodology that allows observation of differences in patterns of gain between departments, courses or modules.
We have seen that there are subject level differences and so our research supports the introduction of a subject level TEF.
Ideally, we’d like to see LG not just used to judge institutions but integrated into curriculum design. Designing for learning is becoming more learner centred – moving from ’what am I going to teach?’ to ‘How are students going to learn?’. A LG methodology such as this could provide data to support course teams to make evidence based decisions (complementing student satisfaction data). For example, do similar modules with different types of learning activities produce different LG trajectories? We know from the field of Learning Design that visualisation is important for course team making decisions, the multi level modelling approach provide data visualisation.
Some questions still to answer….