Presentations morning session 22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence”
•Download as PPTX, PDF•
2 likes•410 views
With the Teaching Excellence Framework being implemented across England, a lot of higher education institutions have started to ask questions about what it means to be “excellent” in teaching. In particular, with the rich and complex data that all educational institutions gather that could potentially capture learning gains, what do we actually know about our students’ learning journeys? What kinds of data could be used to infer whether our students are actually making affective (e.g., motivation), behavioural (e.g., engagement), and/or cognitive learning gains? Please join us on 22 January 2018 in lovely Milton Keynes at a free OU- and HEFCE-supported event on Using data to increase learning gains and teaching excellence. 10.30-11.00 Welcome and Coffee 11.00-11.30 Lightning presentations by participants, outlining insights about learning gains 1130-1300 Insights from the ABC-Learning Gains project Dr Jekaterina Rogaten (OU): Reviewing affective, behavioural and cognitive learning gains in higher education of 54 learning gains studies Prof Bart Rienties & Dr Jekaterina Rogaten (OU): Are assessment scores good proxies of estimating learning gains: a large-scale study amongst humanities and science students Prof Rhona Sharpe (University of Surrey) & Dr Simon Cross (OU): Insights from 45 qualitative interviews with different learning gain paths of high and low achievers Dr Ian Scott (Oxford Brookes) & Dr Simon Lygo-Baker (OU): Making sense of learning trajectories: a qualitative perspective
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Presentations morning session 22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence”
Similar to Presentations morning session 22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence” (20)
More from Bart Rienties
More from Bart Rienties (20)
Recently uploaded
Recently uploaded (20)
Presentations morning session 22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence”
- 1. 22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence” https://twitter.com/LearningGains #learninggainsOU https://abclearninggains.com/
- 2. “The HEALTH AND SAFETY messages” No fire drill today Toilets Jennie Lee Building Live-recording, photos and streaming: so be mindful of what you are saying or doing Contribute to live-tweeting: @learninggains #learninggainsOU
- 3. Lightning presentations Carol Calvert Open University Heike Behle Warwick University Ruslan Ramanau Open University David Reeve JISC Liz Bennett University of Huddersfield Paul Hazel QAA David Boud University of Technology Sydney Selena Killick Open University Fiona Cobb University of London Ed Foster Nottingham Trent University
- 4. 4 Why not engage early with inexperienced learners- 4% increase in retention to be won and a lot of student satisfaction Ever Reg'd Regi'd at start Reg'd at 25% fee point TMA01 submitted ICMA41 submitted 2016 1026 943 885 795 778 2017 1042 982 934 841 835 0 200 400 600 800 1000 1200 Numbers on M140J- ( data from Institutional dashboard) 2016 2017 16 more students registered for October 2017 than 57 more submitted most recent assessment in 2017/18 - Immerse in just one module in discipline + - Network with peers + - Work with some tutors + - Brush up/ acquire academic skills + - Try out materials at own pace + - Build confidence about software used Students are able to
- 5. Assessment and learning gain A dilemma: >If assessment results aren’t an adequate indicator of student learning, what are they supposed to be? >If they do demonstrate learning, then why bother with other measures of learning gain? A way out: >Take seriously the idea that summative assessment should assure course learning outcomes, and work back from that David Boud Centre for Research in Assessment and Digital Learning, Deakin University University of Technology, Sydney HEFCE open event: Using data to increase learning gains and teaching excellence
- 6. Dr David Reeve Head of Information Strategy Jisc Here be GDPR dragons
- 8. Photo by Annie Spratt on Unsplash
- 10. Students’ response on learning analytic dashboards SRHE Scoping Award Aims: • To identify which elements of dashboards design were most valued by students; • To identify students’ learning responses to seeing data presented about themselves via a dashboard; • To identify the potential and limitations of using dashboards with undergraduate students; • To identify questions raised by their use for future research in the area. Key findings: • Students want information about their progress; • Even the weaker ones; • Personal component of getting this feedback is huge and varied; • Move from self-regulated learning to seeing feedback as having 3 dimensions (Sutton 2012): – understanding – being – acting.
- 12. Comparative perspectives on learning gains Learning gains is a appealing construct as it helps to move away from focus on desirable learning outcomes (e.g. higher satisfaction rates and better scores) vs. less desirable to create a richer, more nuanced perspective of learning and its benefits; Comparative perspective would be useful to explore the generalisability of the construct by comparing views of students and teachers across: Different academic disciplines; Different types of online and blended courses (e.g. fully online, most using synchronous or asynchronous tools etc.) ; Different institutions and systems of higher education.
- 13. What if the answer was in the Library all along? Selena Killick @SelenaKillick
- 14. Presenters this morning Dr Jekaterina Rogaten Open University Dr Simon Cross Open University Dr Ian Scott Oxford Brookes Prof Rhona Sharpe University of Surrey Dr Simon Lygo-Baker University of Surrey
- 15. Reviewing affective, behavioural and cognitive learning gains in higher education. Jekaterina Rogaten Bart Rienties Simon Cross Denise Whitelock Rhona Sharpe Simon Lygo-Baker Allison Littlejohn https://twitter.com/LearningGains #learninggainsOU https://abclearninggains.com/
- 16. Research Team Dr Bart Rienties Dr Jekaterina Rogaten Dr Simon Cross Dr Ian Scott Prof Ian Kinchin Prof Denise Whitelock Prof Allison Littlejohn Prof Rhona Sharpe Dr Simon Lygo-Baker Dr George Roberts
- 17. Learning gains We define Learning Gains as: Growth or change in knowledge, skills, and abilities over time that can be linked to the desired learning outcomes or learning goals of the course Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (Submitted: 22-3-2017). Reviewing affective, behavioural, and cognitive learning gains in higher education.
- 18. How are learning gains measured: a meta-analysis The concept of learning gain is primarily used to examine the effect of any particular educational ‘intervention’ There is a gradual increase in studies examining learning gains all across the world All learning gains can be classified into ABC 53% 16% 21% 10% Behaviour-Cognitive Learning Gains Affective-Behaviour-Cognitive Learning Gains Cognitive Learning Gains Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (Submitted: 22-3-2017). Reviewing affective, behavioural, and cognitive learning gains in higher education. Affective-Cognitive Learning Gains Year Numberofstudies
- 19. What type of learning gains are there Affective learning gains: • Attitude • Confidence • Enjoyment • Enthusiasm for a topic • Feeling comfortable with complex ideas • Interest in a topic • Motivation • Satisfaction • Self-efficacy Cognitive learning gains: • Students’ ability to evaluate and create knowledge • Analytical ability • Autonomous cognition • Critical thinking • Ethical thinking • Creative and higher order thinking Discipline specific skills • Knowledge and understanding of the topic, • Oral and written communication • Problem solving • Scientific reasoning • Statistical and research kills/knowledge Behavioural learning gains: •Ability to work independently •Applied conceptual understanding •Effort and engagement •Leadership skills •Team/group working skills •Practical competence •Resource management •Responsibility •Preparation skills •Time management skills
- 20. Methodologies Affective LG Behaviour LG Cognitive LG Pre-post objective - - 31 (21,836) <g> ranged 0.26 to 0.39 comparison between control and treatment (e.g., Andrews et al., 2011; Emke et al., 2016; Georgiou & Sharma, 2015) Pre-post subjective 9 (1,561) <g> = 0.39 (e.g., Beck & Blumer, 2012; Cheng, Liang, & Tsai, 2015; Mortensen & Nicholson, 2015) 1 (114) (Stolk & Martello, 2015) 6 (12,942) <g> = 0.34 (e.g., Hatch et al., 2014; Lim, Hosack, & Vogt, 2012; Stolk & Martello, 2015) Cross-sectional subjective 10 (1,772) M=3.89 (e.g., Gok, 2012; Liu et al., 2014; Moorer, 2009) 12 (4,154) M=3.85 (e.g., Casem, 2006; Gill & Mullarkey, 2015; Gok, 2012) 16 (5,082) M=3.7 (Casem, 2006; Douglass et al., 2012; 2012) Note: 47% off all studies assessed more than one type of learning gains and as such, one sample can fall into more than one category and number of samples in the table not strictly add up to the total number of samples examined in this review.
- 21. Computation of learning gains Raw gain True gain The normalised gain Normalised change ANCOVA on pre-test and post-test scores ANOVA on raw gain scores. ANOVA on residuals Repeated measures ANOVA
- 22. Conclusion There are variety of approaches have been undertaken to date. Whilst a range of learning gains have been found there is a lack of consistency in approach. In general pre-post testing is considered the most appropriate strategy for capturing learning gains. There are variety of methods of computing learning gains and each computational approach has its disadvantages. There seems to be limited difference between self-reported gain and more objective measures of learning gains.
- 23. Are assessment scores good proxies of estimating learning gains: a large-scale study amongst humanities and science students Dr. Jekaterina Rogaten Prof Bart Rienties https://twitter.com/LearningGains https://abclearninggains.com/
- 24. ABC project to measure learning gains? Affect Behaviour Cognition Academic performance VLE Satisfaction
- 25. What students think they gain? I think I am more openly critical (in the positive sense) Day to day when I have my book I have very different approach from recording my notes for example [in my new job], there will reports and planning to be drawn and I think that this will be an aspect of my job where I can say yes the OU study and discipline I’ve received from the OU has actually contributed to that. I observe things better, work into deeper and work on the whole picture rather than narrow. I think more logically and more ‘why did that happen, why did that happen’, there is more questioning, instead of just to accept things. I am much better at time management, I am much more organised now and planning things in advance. now I say, ‘you know what, I can do that in future’. I feel more confident and I am happier because I am doing something I have always wanted to be doing and something that interests me I think I can go confidently to speak what I learned. But even to a job that isn’t directly related to this subject area. I could talk about my experiences, my time management, team working, computer skills as I feel much more confident, I can say, ‘actually I have done this’. Which was one of the reasons I wanted to a degree.
- 26. Do grades matter? How well do your grades represent your progress? probably in the same way that many other people when they look at their own assignment results and exam results …. I feel that I am doing fairly well but I’d always like to improve myself to my results. I get quite upset when I get around 70s … because I am putting so much effort I want my grades to reflect it. They usually go up. But it is Marginal. 5 marks across all the TMAs that’s the variance, it just varies very slightly Even if it is 1-2 marks I say what did I do differently and I go back to tutor to see what did I do differently. What happened, what caused it? Well there are questions with the text books, exercises. So if I get correct answer, I know I am doing fine. When I say correct answer that’s not the end product that’s the whole answer check through it “I suppose you could say… the skills you learn, like group work, presenting and being able to talk to people… I would say the main way that you think about [achievement], it’s just the grade because… that’s what is going on your CV… and affect what job you get. … I’d say the skills you learn as well as becoming an all-rounded person are quite important as well”.
- 27. Do students make learning gains, and what is the power of LA to predict learning gains? Using assessment results for estimating learning gains has number of advantages: Assessment data readily available Widely recognized as appropriate measure of learning Relatively free from self-reported biases Allows a direct comparison of research finding with the results of other studies Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, Estonia
- 28. Estimating learning trajectories Level 1 Level 2 Level 3 Grade1 Student1 Grade3 Grade1Grade2Grade3Grade1Grade2Grade3Grade2 Student2 Student3 Course1 Course2 Grade1Grade2Grade3 Student4 Grade1Grade2Grade3 Student5 Course3
- 29. 1st year STEM students VPC course 51.6% VPC student 9.1% VPC TMA 40.3% Regression S.E. 2-level S.E. 3-level S.E. Intercept B0 70.81 0.21 69.97 0.31 74.52 4.22 Slope B1 1.06 0.08 1.76 0.11 0.58 1.39 Deviance 211024.410 202908.57 194660.91 X2 change 8115.84** 8247.67** ** p<0.01
- 30. 1st year STEM students Model 1 S.E. Model 2 S.E. Fixed Part Intercept 74.52 4.22 75.14 4.21 B1 Slope 0.58 1.39 0.58 1.39 B2 Black -6.86** 0.97 B3 Asian -4.67** 0.79 B4 Mixed/Other ethnicity -2.64** 0.86 B5 Unknown ethnicity -1.05 1.02 B6 HE/PG qualification 1.81** 0.36 B7 Lower than A level / No formal qualification -2.08** 0.36 B8 Unknown qualification -1.80* 0.79 B9 Low SES -1.58** 0.42 B10 Unknown/Not Applicable SES 0.94 0.58
- 31. How do STEM students compare to Social Science students Participants 5,791 Science students of whom 58.2% were females and 41.8% were males with average age of M = 29.8, SD = 9.6. 11,909 Social Science students of whom 72% were females and 28% were males with average age of M = 30.6, SD = 9.9 Measures Tutor Marked Assessments (TMA) Across 111 modules
- 32. Descriptive statistics: Social Science Social Science Science
- 33. How do STEM students compare to Social Science students Social Science Science Variance in students’ initial achievements 6% 33% variance in students’ learning gains 19% 26% VPC module 4% 25% VPC student 56% 51% VPC TMAs 40% 22%
- 34. Effect of socio-demographic factors Variable Social Science Beta Model1 Beta Model2 Beta Mode 3 Gender 0.64* Unknown -2.4* Other -6.68** Mixed -3.27** Asian -4.66** Black -7.99** HE qualification 0.29 Lower than A levels -3.11** No formal qualification -6.93** PG qualification 2.62** Variance explained in learning gains 0.1% 3% 3.1% Variable Science Beta Model 1 Beta Model 2 Beta Model 3 Gender 0.29 Unknown 1.76 Other -4.09 Mixed -0.59 Asian -7.31** Black -13.07** HE qualification 2.64** Lower than A levels -4.59** No formal qualification -9.48** PG qualification 8.19** Variance explained in learning gains 0.3% 2.2% 3%
- 36. Social Science Science A levels or equivalent HE Qualification Lower than A levels No formal qualification PG qualification
- 37. University 1 1,990 undergraduate students University 2 1,547 undergraduate students 20 degree programmes within each university DV – average grade yearly grade University 1 University 2 Year M SD M SD 1 60.65 7.62 63.75 12.66 2 61.31 6.81 65.64 12.74 3 63.32 6.63 64.12 14.02 Can we use this approach to look across different universities?
- 38. Data Analysis: Descriptive Plots University 1 University 2
- 39. Data Analysis: Model comparison University 1 Regression S.E. 2-levels S.E. 3-levels S.E. Intercept B0 60.252 0.114 60.2 0.16 60.337 0.687 Slope B1 0.429** 0.027 0.422** 0.024 0.365** 0.111 Deviance 74016.17 68227.62 67983.61 X2 change 5788.548** 244.012** University 2 Intercept B0 64.322 0.325 64.225 0.323 63.626 1.419 Slope B1 0.184 0.251 0.22 0.211 -0.131 0.723 Deviance 33145.79 32600.89 32255.87 X2 change 544.898** 345.028** **p<0.001
- 40. Variance partitioning University 1 University 2 Variance at Department level 13.1% 22% Variance between students 59.8% 22% Variance within students (between years) 27.1% 56%
- 41. Summary of findings Although both universities overall showed positive gains, substantial differences were present in variance at departmental level. Aggregate learning gains estimates can result in misleading estimates of students’ learning gains on a discipline or degree level. Multilevel modelling is a more accurate method in comparison with simple linear models when estimating students’ learning gains.
- 42. Possible implications Support for subject level TEF Guidance on where to focus interventions and resources Visualisation could promote data informed learning design decisions Questions still to answer: Does grade trajectory reflect students’ learning gains? Can we make a meaningful comparison between universities? What impact grade trajectory has on students? Do they need to know their own trajectory and how it compares to others?
- 43. Dr. Jekaterina Rogaten Prof Bart Rienties https://twitter.com/LearningGains https://abclearninggains.com/
- 44. Rhona Sharpe, University of Surrey Simon Cross, The Open University OU Learning Gain Conference / 22 January 2018 Insights from 45 qualitative interviews with different learning gain paths of high and low achievers
- 45. PRESENTATION OVERVIEW Using data from interviews undertaken at the Open University UK and Oxford Brookes University, this presentation will probe the relationship between how students understand and interpret the learning gains they experience and the meaning and significance they give to metrics in use for measuring learning gain. ABC Learning Gains Project
- 46. 22/01/2018 Learning Gains As TEF develops we will incorporate new common metrics on engagement … and learning gain, once they are sufficiently robust. Government Green Paper Policymakers are now called on to provide evidence to inform questions such as: How do students’ knowledge, skills and work-readiness change and improve through their experience of higher education?... Traditional measures for evaluating student performance remain essential …but they do not provide all of the evidence needed to address these questions. And this evidence is more important than ever… HEFCE Average gain score needs to be interpreted with caution and there remain many outstanding issues that need investigating. Pascarella et al. 2011.
- 47. 22/01/2018 1. How do students and alumni understand and interpret the learning gains they experience? 2. How do students understand and interpret measures of learning gain currently in use? 3. To what extent can these two be reconciled? 4. What is the contribution of learning gains to employability/work readiness? Research questions How do students understand and interpret learning gains?
- 48. 22/01/2018 Semi-structured interviews with 19 part-time distance learners resident in the UK lasting 30-60 minutes and 12 alumni from full time courses at campus based university. • Student perception of gain and progress • Cognitive, behavioural and affective change • Relationship between grades and progress • Study expectations • Graduate attributes and employability • Work relevancy and readiness Methodology and participants How do students understand and interpret learning gains?
- 49. 22/01/2018 First stage Interim analysis based on review of interviewer notes, second listen to interview and coding of interview transcripts. Part-time distance students How do students understand and interpret learning gains? Student 8 Female High Lowest STEM Student 19 Female Low Medium Arts/Soc.Sci Student 5 Female Low Highest Edn/Lang Student 10 Male High Medium Arts/Soc.Sci Student 12 Female Low Lowest STEM Student 18 Female High Highest STEM Attainment grouping as measured by assessment marks Progress grouping as measured by assessment marks Gender Faculty Sources of quotations used in presentation
- 50. 22/01/2018 First stage Interim analysis based on review of interviewer notes, second listen to interview and coding of interview transcripts. Full-time campus alumni How do students understand and interpret learning gains? Alumnus 8 Female 1st Upward Business Alumnus 5 Female 2:1 Upward Health & Life Sciences Alumnus 12 Female 2:1 Upward Business Alumnus 7 Female 2:1 Upward Health & Life Sciences Final degree classification Progress trajectoryGender Faculty Sources of quotations used in presentation
- 51. 22/01/2018 The ‘Turning point’ How do students understand and interpret learning gains? “ A wake up call The biggest game changer Where it all kind of came into place Self-realisation Finding myself Did not enjoy first year but ‘when I was part of the social enterprise, I think that was like the biggest game changer, with me feeling at home at university.. when I was part of the social enterprise, I found my niche, I found really good friends, a really good social network.’ (A5)
- 52. 22/01/2018 Turning points / Pivot moments How do students understand and interpret learning gains? Formative feedback Critical self- awareness Independence • LEVEL OF STUDY Goal setting Ownership of Learning Learning design and sequence Work-Life Level of study What Where
- 53. 22/01/2018 Critical self-awareness How do students understand and interpret learning gains? “And you do look back and see what you did, you analyse it and you try and implement it in the future and that’s exactly what I did in my second semester and sort of saw the same results.” (A12) Reviewing feedback and successful study strategies (A12) Recognising my entrepreneurial spirit, and being able to choose modules in the final year (A8) Finding out where you belong and what you are good at (A5) Discovering what motivated me (A7) “Realising within myself that I didn’t want to follow the same path as everyone else.” (A8) .
- 54. 22/01/2018 Independence How do students understand and interpret learning gains? “I was really able to kind of understand how much I was able to gain and achieve by just working independently. I really cherished my alone time, for me to explore things on my own and discover things.” (A12) Self-realisation of ability to study independently (A12) Experience of working independently on placement applied to final year tasks (A8) Personal development, emotional maturity “I fitted in my boots a lot better” (A5) Choosing a topic that I was interested in and motivated to ‘eat up’ the whole journal article (A7) “When I came back in my final year that really helped me to understand how I can work more independently and deliver results at the end.” (A8) .
- 55. 22/01/2018 Level of study ‘Student 8’ How do students understand and interpret learning gains? [Initially] it was very much like study for study purposes [but] about a year and half into it, my mind-set changed, it was like I enjoy what I’m doing and it’s giving me something tangible. This year was the first year when … it reflected back on my day job. The outcome was a decision to increase study intensity… I [feel] that I have the knowledge, it’s the time that I was missing… I don’t think for me it’s grade itself is… it’s actually the knowledge. [Getting the grade] is what’s needed this year for me not to give up so that I had an opportunity to do the exam and face the next step. Looking back Current module I have decided that I will care about my grades next year and try to get the highest scores I physically can just to see [how well I can do] if I don’t have that additional 60 credits. Looking ahead
- 56. 22/01/2018How do students understand and interpret learning gains? There were things in the first two modules that I already knew [but] when I come to this year [then] I would say probably 95% of what I’ve learned … I’ve never heard of. So, I think this has been a real turning point for me. [This year] was my turning point… Your TMA (continuous tutor marked assessment) is not everything. It’s not, it’s supposed to be what you actually physically know yourself inside… I think that’s really important. Looking back Current module Level of study
- 57. 22/01/2018 Work-Life Reference (Points) How do students understand and interpret learning gains? The way I think, the way that I possibly act at times, my life feels different now … I can talk confidently to people. It’s just when you learn something you [then] become aware of, for instance, either news, work itself, everyday life; which again kind of changes your perspective and then it allows you to properly build your own confidence because you understand things Confidence Awareness
- 58. 22/01/2018 1. Participants often described their learning gains as turning points/pivot moments. These were seen at different times e.g. at Level 2 (OU), returning from placement year (OBU) or during postgraduate study (OBU). 2. Sometimes turning points resulted in ‘improvement’ according to the metrics we use to measure of learning gain. However, sometimes not. e.g. changes in study strategies, grades, choice of modules/topics. 3. Both OU students and OBU alumni frequently perceive and measure gain with reference to outside work and life context. 4. If the impact of a turning point is not always seen in measures (e.g. grades,) how could they be made visible? 5. Should change/gain be conceived as incremental or paradigm changing? a. Can the same scale measure before and after a turning point? b. Is the presence of a turning point itself a measure of gain? Findings and more questions
- 59. Making sense of learning trajectories: a qualitative perspective Dr Ian Scott Dr Simon Lygo-Baker
- 61. Methods Semi-structured interviews, with students that had completed diaries, 4 students came forward, 1st and 2nd year students, from a variety of subjects. Students responded to adverts.
- 62. Alpha Alpha reported change with regard to use of technology, for example, using collaborative documents, recording lectures also becoming more independent as learning
- 64. Beta
- 65. Beta “Because I’m doing architecture, I think more architecturally about things now… When I look at a building or something, I try and see how it’s connected or what’s underneath it, instead of just seeing what other people see”. “Because I’m doing architecture, I think more architecturally about things now… When I look at a building or something, I try and see how it’s connected or what’s underneath it, instead of just seeing what other people see”.
- 67. Gamma However “I think I probably do a lot more outside of Uni but still relating to my topic… reading around things that maybe won’t help in an essay but I just find interesting anyway, which I didn’t do before.” “The difference from school in that you’d be taught something and you would accept it as right and wouldn’t really think to question it. “ She noted, “I think I work at a different pace to everyone else, it’s quite hard because you’re never doing work at the same time. I think that’s why I never really work with people on the course, but I would like to be able to because I think it would be useful.”
- 68. Gamma “It’s so different from school and I’d never experienced anything like it before, and how much more independent you are with your work, how much more it is reliant on you. You have to… be a lot more organised and know what you’re doing.” I’ve done a lot more sort of critical thinking, sort like criticising… the way some studies are done and why they might not be as reliable as others. … I don’t just accept everything and then I might… do more research myself if there is something that I don’t quite agree with or I think I’ve read something different before.”
- 70. Delta “I’m definitely become more mature.” “I just come up with my arguments and then find the books that I need to, and I’ve learned how to get straight to the point that I need to and then summarise all that information and then just write in an essay as quickly as possible”. “I spend a long time on forums every day.” A Non Gain: “it’s a combination of the fact that, firstly, I’m not working with anyone [else] and, secondly, I don’t speak to that many people in the university and the people I do [talk to] are from this country.” to always kind of question everything”. He used this skill more broadly when reading news media and was more sceptical “unless there’s [sic] sources to back it up”. He related this to the proliferation of “fake news”: “it’s definitely something you’ve got to question… and make sure what you’re reading is actually true”.
- 72. A quick summary ABC model is a useful anvil to elucidate student’s perceptions on their own learning and what they have gained. For these students no evidence of a link between how students articulate what they have gained and their grade trajectories. All of our case studies could describe gains, be able to think critically and study independently were common themes Some of the informants had clearly shown identifiable personal development e.g confidence, ability work in group, deeper understanding of discipline and people, capacity seek and take in more views. Such gains were not articulated by all participants, reinforces how these students engage in unique ways with their learning and the University.
- 73. 22 January 2018 HEFCE open event “Using data to increase learning gains and teaching excellence”. https://twitter.com/LearningGains #learninggainsOU https://abclearninggains.com/
Editor's Notes
- Stanislav Petrov, in 2017 1983 Soviet nuclear false alarm incident On 26 September 1983, just three weeks after the Soviet military had shot down Korean Air Lines Flight 007, Petrov was the duty officer at the command center for the Oko nuclear early-warning system when the system reported that a missile had been launched from the United States, followed by up to five more. Petrov judged the reports to be a false alarm,[1] and his decision to disobey orders, against Soviet military protocol,[2] is credited with having prevented an erroneous retaliatory nuclear attack on the United States and its NATO allies that could have resulted in large-scale nuclear war. Investigation later confirmed that the Soviet satellite warning system had indeed malfunctioned.[3]
- Add reference
- Web of Science core collection Science Citation Index Expanded (SCI-EXPANDED) Social Sciences Citation Index (SSCI) Arts & Humanities Citation Index (A&HCI) Conference Proceedings Citation Index- Science (CPCI-S) Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH) Book Citation Index– Science (BKCI-S) Book Citation Index– Social Sciences & Humanities (BKCI-SSH) Emerging Sources Citation Index (ESCI)
- The raw gain as a value of gain is inaccurate due to the difference between scores being less reliable than scores themselves i.e., raw gain represents compound error of pre-test and post-test. The raw change in scores between pre-test and post-test assumes linear relationship between test scores and ability. However, learning gain on a simple test will be higher for a low-ability student and lower for a high-ability student, whereas the raw learning gain on a difficult test will be higher for a high-ability student and lower for a low-ability student. The true gain is based on a linear regression procedure where the true gain for each individual is the difference between group mean pre-test score and group mean post-test score assuming the reliability of estimates is satisfactory i.e., the pre-test post-test variances and reliability are equal (Lord, 1956, 1958). However, when these assumptions are violated the reliability of raw gain scores is actually high and therefore use of raw gain scores produces would be a more accurate representation of a gain that true gain (Overall & Woodward, 1975; Zimmerman & Williams, 1982). Building on linear regression procedure for measuring true gain, the residual gain was proposed as a measure of change (Cronbach & Furby, 1970). The computation of residual gain is compatible with the raw gain. The advantage of this computation is that it removes the change from the posttest scores that are predicted from the pretest (Linn & Slinde, 1977). Although, residual gain allows to identify individuals that showed more or less than expected gain (i.e., are superior of inferior learners) residuals do not represent change as such, they only represent what was not predicted linearly (Baird, 1988). As they are residuals (essentially deviations) half of students will by default be above the mean and half bellow which makes judgement on the effectiveness of learning inappropriate (Pike, 1992). Similar, computation of learning gain is basefree measure of change (Tucker, Damarin, & Messick, 1966), but as they suggested in their work, basefree measure of change was directed at correlational research into understanding why certain individual demonstrate above average change. The basefree measure of gain was also criticised for incorrect attribution of the change scores in their computations (Bond, 1979; Cronbach & Furby, 1970) and as such was not used in the research on gains (for the exception of few studies in 70s and early 80s). Further attempt to compute gain was to compute normalised gain (Hake, 1998). The normalised gain is the most widely used measure of learning gain. The main advantages of using normalised gain is that it solves the problem of ceiling effect and bias towards strong students (high pre-testers) by using the difference between the maximum test score and pre-test score as denominator. Thus, normalised gain demonstrates realised gain to the maximum of possible gain. In addition average normalised gain for a group can be computed using either individual scores or group means for pre-test and post-test (Bao, 2006). Although in both cases the same principle is used, the two methods may yell different results for the same sample due to asymmetrical distribution of differences between low and high scoring pre-testers on their post-tests or scoring lower on the post-test than in the pre-test (for a review see Bao, 2006). In addition the normalised gain cannot be computed for individuals who scored absolute maximum on the pre-test scores or the average normalised gain cannot be computed using individual scores if any one person scored maximum on the pre-test (Marx & Cummings, 2007). Hoping that as a result of intervention/course/teaching it is more likely that students will demonstrate positive gain and score higher on the post-test than on pre-test. However that is not always the case and in the instances of negative gain or when maximum possible score was obtained at the pre-test the normalised learning gain cannot represent a meaningful learning gain or lack of learning. In cases where post-test scores are lower than pre-test scores computation of normalised change is more meaningful (Marx & Cummings, 2007). The normalised change has advantage over normalised gain in cases of negative gain by using analogous computation where observed loss is the ratio of possible maximum loss. However this method does not apply to the students who scored possible maximum or possible minimum on both pre-test and post-test (Marx & Cummings, 2007). In addition, normalised change ranges from -1 to +1 and removes low pre-test scores bias. Thus, students who score 90% on a pre-test can obtain a change ranging from -1 to +1 and the same is the case for students who scored 50% or 20%. As such, it is much easier and more intuitive to interpret normalised change rather than normalised gain. Marx and Cummings (2007) further argued that normalised change is suitable for computation if the pre-test and post-test when they are not the same. However it is important to understand that when averaging gains or losses to a group, both gains and losses are relative only to the maximum possible gain i.e., the result will show more gain than loss. All previous methods for computing change were aiming to address the issues posed by raw gain e.g., measurement errors, regression to the mean, bias towards high scores in the pre-tests. To further improve accuracy in computing gain in educational settings, Rasch model was used. The Rasch model is based on probability and the assumption is that scores obtained at the pre-test and post-test for each student are a combination of that student’s ability and difficulty of the test. In cases when the research design involves comparison between two or three groups, the most commonly used methods for computing learning gains from pre-test and post-test scores are: analysis of variance (ANOVA) and analysis of covariance (ANCOVA). The advantage of ANOVA and ANCOVA is in that they both reduce error variance and increase the power of the test (Sörbom, 1976). There are four ways of data analysis commonly used in mixed research. The preferred method is to use ANCOVA on pretest and posttest scores or ANOVA on raw gain scores. The two least favourable analyses are ANOVA on residuals and repeated measures ANOVA (Dimitrov & Rumrill Jr, 2003). In ANCOVA pretest scores are used as covariates of postest scores because that reduces error variance by adjusting posttest means to the pretest. However, ANCOVA will produce reliable results when the assumptions of linear relationship, randomization and homogeneity of variances are met. Same assumption should be met for ANOVA on raw gains analysis. However ANCOVA is more powerful than ANOVA and more flexible on the assumption of linear relationship and as such ANCOVA will also produce more accurate computation of gain if the relationship between pretest and posttest has quadratic or cubic component. In all, all four ways of data analysis have limitations, but results obtained from ANCOVA seems to be most accurate for the comparison of learning gains between two or more groups.
- Assessment = already collected metrics as opposed to new measures
- Level 1 – Grade: repeated measures on students and tell us about students learning trajectory Level 2 – student: between students variations Level 3 – Course: between course variation
- 10 - modules
- 10 - modules
- Student-level intercept-slope correlation - Students with low initial achievements show high learning gains that students with initial high achievements. Module level intercept-slope correlation Students in modules with the low initial achievements show higher learning gains than students in courses with the high initial achievements VPC – is variance partition coefficient. VPC module shows variance that can be attributed to the differences between modules. VPC student shows the mount of variance that can be attributed to the differences between students, and VPC TMA shows variance within each student across assessments. In this last one we can see that it is almost double in Social Science hence further supports that assessments play important role.
- Gender: male-female learning gains gap (male reference group) Other-white gain gap Mixed-White gain gap Asian-White gain gap Black-white gain gap HE qualification - A levels gain gap lower that A levels - A levels gain gap No formal qualification – A levels gain gap PG qualification - A levels gain gap
- On average students showed improvement in standardised grades, although this was only significant for University 1
- University 1 variance was mainly nested between students University 2 had more variance at the departmental level and within students The negative learning gains seen in some students in University 2 does not imply that these students are losing knowledge or ability per se. However it highlights the complexity of factors that have to be taken into account when using students’ academic performance as a proxy for learning gains, such as ‘assessment difficulty’ and ‘learning design’ (Rienties & Toetenel, 2016). We need to understand these if we are to use academic performance as a proxy for learning gains within Higher Education.
- We set out to see if multilevel modelling of assessment data could be a valid and useful proxy for learning gain Aggregate learning gains estimates can result in misleading estimates of students’ learning gains on a discipline or degree level. (or show no gain hiding a more complex picture – University 2) Multilevel modelling is a more accurate method in comparison with simple linear models when estimating students’ learning gains. The simple models are not able to detect differences between modules when looking at the department and degree level performance, whereas multilevel modelling can.
- The HEFCE funded programme of which this project is a part wants to devise a sector wide APPROACH (not method) and characterisation of learning gain, and explore possible proxies. The contribution of the ABC project is that if we are to use LG research for optimising learning (LAK definition) then we need to use a methodology that allows observation of differences in patterns of gain between departments, courses or modules. We have seen that there are subject level differences and so our research supports the introduction of a subject level TEF. Ideally, we’d like to see LG not just used to judge institutions but integrated into curriculum design. Designing for learning is becoming more learner centred – moving from ’what am I going to teach?’ to ‘How are students going to learn?’. A LG methodology such as this could provide data to support course teams to make evidence based decisions (complementing student satisfaction data). For example, do similar modules with different types of learning activities produce different LG trajectories? We know from the field of Learning Design that visualisation is important for course team making decisions, the multi level modelling approach provide data visualisation. Some questions still to answer….
- Needing to be robust,
- The original purpose of the alumni interviews was to assess the contribution of the GAs to employability. The interview schedule asked questions about personal development, employability and work readiness. We also asked them some general questions about what they though they gained from their experience at Brookes and prompted specifically for ABC types of learning gains.
- we had become interested through the project in trajectories and so for this presentation I’ve looked and listened again at the interviews of 4 students who were on an upward trajectory. We didn’t select alumni interviews on the basis of their academic progress or trajectory and unsurprisingly, only 1 downward trajectory person volunteered for interview, and that was because he had beef with Brookes. The interviewees were mostly high achievers. Out of the 12, only one got a 2:2, rest for good degrees. With a greater proportion than you would expect having chosen a combined honours degree and/or a year’s placement.
- Emotional language used to describe these points. Interviewees able to identify them. Express the scale of the impact they made on them in their choice of words. For example, for A5, Psychology, 3 years, 2:1, but did internship with a social enterprise while at Brookes, and that turned into employment in 3rd year, initially one day a week and then promotion to 2 days a week in a leadership role.. “I’m one of those people who can’t sit still, I literally feel I must keep learning.” No
- 2 examples from each interview set
- So what changed? The ability to critically reflect on own study strategies, A self awareness of own strengths An ability to see a plan for own career or life goals. This awareness came from Course and study (A12) Placement experience (A8) Or extra curricular work experience (A5) For 12, this turning point was about an awareness of own study strategies, partly a result of friends having left uni while she was on placement. Having always been driven and determined, she now set her sights on a goal (good grades) and discovered how to achieve it. For A8 this turning point was about an awareness of where own interests lie within a broad discipline and the ability to follow this new self knowledge by choosing more relevant modules, which she enjoyed and did better in, and so saw an increase in grades. For A5 this turning point was about finding a place where you belong and you are doing something you are good at. Being in a position to recognize own team leadership skills and put them into practice (here through social enterprise work experience) “Finding where you belong and what you’re good at and what you like. “I found where I belong if that makes sense and that’s where my drive towards thinking well actually I quite like HR, because I did a bit of HR and I did a bit of corporate relations, those were my two like job titles in the enterprise. And I also discovered that I, although I never take on a front facing leadership role, I often find myself being a leader within a group.”
- Another common feature of what changed for these 3 upward trajectory students was a confidence in working and learning independently For A12, The ability to study successfully independently. More than to do this successfully, but actually to revel in it. For A12, returning from placement and finding friends had all gone, started to study independently and enjoy that challenge and see it having a beneficial effect on marks. Coming back from placement, all my friends left, ‘self realization of helping myself…. I was really able to kind of understand how much I was able to gain and achieve by just working independently. Yeah and I really cherished my alone time, for me to explore things on my own and discover things.” (12) For A8, returning from a placement where had been supervised by a ‘hands-off’ manager who had largely allowed her to pursue her own interests. Now applying this new independence to study and assignment tasks. For A5 it was about personal maturity (independence?) “I definitely, when I started Brookes I was a completely different person to the girl who left Brookes. I think, I mean I think I’ve always been fairly mature for my age and I never really felt right in my shoes from that. But when I left Brookes I felt like I fitted in my boots a lot better, like I still wasn’t completely there, but I found my niche, if that makes sense. “ For A7 it was independence of choice I think being given the freedom to choose your own topic within a broader topic, because then you’re really getting down to something you’re interested in and from the psychology perspective you had like a role in the choice making that it seems more like your decision, so you’re more motivated to complete it.