Understanding learning gain and why this might matter to you
•Télécharger en tant que PPTX, PDF•
1 j'aime•1,340 vues
Understanding learning gain and why this might matter to you, Corony Edwards 23 June 2016
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Understanding learning gain and why this might matter to you
Similaire à Understanding learning gain and why this might matter to you (20)
Plus de Jisc
Plus de Jisc (20)
Dernier
Dernier (20)
Understanding learning gain and why this might matter to you
- 1. CONSULTING www.coronyedwards.co.uk Corony Edwards Understanding Learning Gain and why this might matter to you Jisc Student Experience Experts’ Group Birmingham 23 June 2016
- 2. CONSULTING www.coronyedwards.co.uk Corony Edwards Session outline and objectives 1. To clarify what learning gain means in the context of UK HE 2. To consider some drivers for the current interest in LG, including TEF 3. To introduce the types of measure used to gauge LG, and the challenges associated with developing these measures 4. To consider how learning technologies could be deployed to provide robust data for learning gain measures.
- 3. CONSULTING www.coronyedwards.co.uk Corony Edwards Definitions of learning gain “an attempt to measure the improvement in knowledge, skills, work-readiness and personal development made by students during their time spent in higher education” HEFCE Learning Gain website “the ‘distance travelled’, or the difference between the skills, competencies, content knowledge and personal development demonstrated by students at two points in time” RAND report: McGrath et al, 2015, p. ix)
- 4. CONSULTING www.coronyedwards.co.uk Corony Edwards Attempts to measure learning gain are not new But, as noted in the RAND report (2015, p. ix) “Debate on the measurement of the learning students gain in higher education has been prominent internationally, but the concept has not been studied extensively in the English higher education context.”
- 5. CONSULTING www.coronyedwards.co.uk Corony Edwards Drivers for measuring learning gain “There is growing interest around the world in measuring how much students in higher education learn … Amongst other factors, such measures can be seen as important to the debates about the quality and impact of higher education, how we evidence the value of investment in it, and how we evidence students’ skills acquisition for employers.” RAND report: McGrath et al 2015, p. xi
- 6. CONSULTING www.coronyedwards.co.uk Corony Edwards The need to measure impact Rich, 2015. pp 3 - 4 “… if universities are to command public investment, then a public good has to be served and observed.” “… what has been lacking is any measure of the actual value added by universities to students and to the economy.” “Without equations to demonstrate impact, it is hard to measure the public good and, in an austere world driven by econometrics, what is hard to measure is hard to fund.”
- 7. CONSULTING www.coronyedwards.co.uk Corony Edwards Transparency for students The desire to improve career prospects is the primary reason students cite for going to university (Kandiko Howson, 2012) ‘Many higher education applicants … imagine that the relationship between a course of study and a career is mechanistic, that higher education is about direct preparation or training for a job’ Rich, 2015 p. 18 “Higher education cannot reasonably offer [employment] guarantees to students. We need to shift the way we frame the student experience, to be more transparent about what they will really gain from it.” Rich, 2015 p. 18
- 8. CONSULTING www.coronyedwards.co.uk Corony Edwards Learning Gain and the Teaching Excellence Framework • Good teaching seen as a means to increasing graduates’ earning potential, and also their overall contribution to the economy and to society (the public good) (BIS 2015) • Proposed TEF2 will provide ‘better understanding of the range of skills and knowledge [students] bring from their course’ (BIS 2015 p. 21) • BIS envisage incorporating evidence of learning gain as part of TEF2 (BIS 2016 pp. 11, 12, 16 and 30) Department for Business, Innovation and Skills, May 2016
- 9. CONSULTING www.coronyedwards.co.uk Corony Edwards RAND report 2015: 5 main groups of measure • Standardised tests: measure the skills students acquire; generic or specialised • Grades: measure the progress in students’ achievement by comparing the difference between grades at selected points in time • Self-reporting surveys: ask students to report the extent to which they think they have gained knowledge and developed skills; run at a number of points throughout a degree programme • Other qualitative methods: e.g. PDPs; encourage students to reflect on their learning, acquired skills and skills gaps; stimulate productive discussions between students and their tutors • Mixed methods: draw on a range of tools and indicators to track improvement in performance, (e.g. through a combination of grades, student learning data and student surveys); can be used to predict students’ performance. See Wabash study for a comprehensive example https://www.youtube.com/watch?v=N4OKKsW-YdQ
- 10. CONSULTING www.coronyedwards.co.uk Corony Edwards RAND report 2015: types of LG measure • Proxies, e.g. engagement, experience ratings, employment (including HMRC data) ‘could prove efficient since they exploit existing metrics’ but validity is questionable • Direct measures include ‘before’ and ‘after’ grades; standardised tests (generic & discipline specific); student self-report surveys (again, validity concerns) • e.g. UKES pilot included 12 items on students’ perceived skills development; NSS contains one question on personal development
- 11. CONSULTING www.coronyedwards.co.uk Corony Edwards Unintended consequences and caveats • Gaming by institutions • Discouraging learning itself • Quality is relative and must take into account the diversity of missions and goals in HE • Quality of learning is more than just learning gain, and quality of HE is more than just quality learning • Skills and competencies may not be comparable across disciplines • Contextual and demographic characteristics are a factor, but difficult to determine which • Learning gain should not be the sole determinant of the quality of HE (cf TEF proposals). RAND report: McGrath et al 2015 pp. 11, 12
- 12. CONSULTING www.coronyedwards.co.uk Corony Edwards Challenges of developing LG measures which… • Have a clearly stated purpose • Identify the dimensions of analysis (personal development, generic skills etc) • Consider complexity of LG and diversity of students, HEIs, state mission etc • Adopt methods which have comparable inputs and outputs • Are practical, can be implemented effectively and are cost-effective • Remain aware of unintended consequences, for example, need to prevent ‘gaming’ • Achieve buy-in from HEIs and other stakeholders • Acknowledge the limits of the measures • Validate methods, for example by controlling for factors which could impact on validity, such as sociodemographic or neighbourhood effects
- 13. CONSULTING www.coronyedwards.co.uk Corony Edwards UEL/Roehampton/Brunel Focus on academic skills and engagement with learning Same measures being trialled across all 3 HEIs Academic Behavioural Confidence (ABC) scale to test students; also to predict grades and where students grades will lie in relation to cohort “A psychometric means of assessing the confidence that undergraduate university students have in their own anticipated study behaviours in relation to their degree programme” (Sander and Sanders 2009:19) Where are digital capabilities?
- 14. CONSULTING www.coronyedwards.co.uk Corony Edwards UEL/Roehampton/Brunel • A ‘Need for Cognition’ scale (drive to learn) may also be used • May also use NSSE or UKES in future • Also looking at developing a new tool to measure employability confidence
- 15. CONSULTING www.coronyedwards.co.uk Corony Edwards Lincoln https://www.adc.uk.com/assessment/products/dilemmas-situational-judgement-test/graduate/ Where are digital capabilities?
- 16. CONSULTING www.coronyedwards.co.uk Corony Edwards Russell Group consortium example 1 Career planning / readiness • Using existing methodology developed by Bob Gilworth (Leeds) • Formative tool embedded in annual registration process • Students pick one of 10 statements which they feel most closely reflects their current career thinking (3 x ‘Decide’, 3 x ‘Plan’, 3 x ‘Compete’ and 1 ‘sorted’) http://careerweb.leeds.ac.uk/info/13/career_planning
- 17. CONSULTING www.coronyedwards.co.uk Corony Edwards Russell Group consortium example 2 Graduate attributes • ‘Realise2’ questionnaire • Blue chip organisations (eg Ernst & Young) use this as part of their graduate recruitment process to test for a set range of graduate attributes (strengths) “Through 180 questions, 60 strengths are accurately assessed according to the three dimensions of Energy, Performance and Use – with each user receiving their unique, personalised Profile, revealing their realised strengths, learned behaviours, weaknesses and unrealised strengths.” http://www.cappeu.com/R2StrengthsProfiler
- 18. CONSULTING www.coronyedwards.co.uk Corony Edwards Discussion 1) Is your HEI using or developing measures of learning gain or learner analytics? If so, briefly tell others in your group what is being measured and how 2) How are, or could, learning (or other) technologies be used to support the robust measurement of learning gain? Consider: a) Student engagement data sources: already recorded / available but not recorded b) Administration of tests: existing and new 3) Which other professional and academic departments will you need to collaborate with in order to do this? 4) Are you ready for it?
- 19. CONSULTING www.coronyedwards.co.uk Corony Edwards Thank you for your participation! If you would like a longer version of this presentation, please contact me via my website
- 20. CONSULTING www.coronyedwards.co.uk Corony Edwards References • Britton, J. et al (2016) How English domiciled graduate earnings vary with gender, institution attended, subject and socio-economic background, IFS Working Paper W16/06. Nuffield Foundation. • Cacioppo, J. T., & Petty, R. E. (1984). The need for cognition: Relationships to attitudinal processes. In R. P. McGlynn, J. E. Maddux, C. Stoltenberg, & J. H. Harvey (Eds.), Social perception in clinical and counseling psychology. Lubbock, Texas Tech University Press. [see also http://www.liberalarts.wabash.edu/ncs/ for overview article] • Collini, S. (2012) What are universities for? Penguin, London • Fulfilling our Potential: Teaching Excellence, Social Mobility and Student Choice. Department for Business, Innovation & Skills, 2015 https://www.gov.uk/government/consultations/higher-education-teaching-excellence-social-mobility-and-student-choice • Greatbatch, D & Holland, J. (2016) Teaching Quality in Higher Education: Literature Review and Qualitative Research May 2016 Department for Business, Innovation & Skills https://www.gov.uk/government/publications/teaching-quality-in-higher-education- literature-review-and-qualitative-research • Teaching Excellence Framework Technical Consultation for Year Two. Department for Business, Innovation & Skills, 2015 https://www.gov.uk/government/consultations/teaching-excellence-framework-year-2-technical-consultation • Kandiko Howson, C. (2012) Student expectations and perceptions of higher education. Kings College London and Quality Assurance Agency
- 21. CONSULTING www.coronyedwards.co.uk Corony Edwards References contd. • Learning Gain: HEFCE Policy Guide http://www.hefce.ac.uk/lt/lg/ • McGrath, C. H. et al. (2015) Learning gain in higher education. RAND Corporation, Santa Monica, CA http://www.rand.org/pubs/research_reports/RR996.html (The HEFCE ‘RAND report’) • Meltzer, D. (2002) ‘The relationship between mathematics preparation and conceptual learning gains in physics: A possible “hidden variable” in diagnostic pretest scores’. American Journal of Physics. 70:12, 1259-68 • Morrison, B, (2005) Evaluating learning gain in a self-access language learning centre Language Teaching Research. 9:3 267-293 • Rankin, E. & Dale, L. (1969) ‘Cloze Residual Gain - A Technique for Measuring Learning Through Reading’ in Schick, G. & May, M. Eds. The Psychology of Reading Behavior; Eighteenth Yearbook of the National Reading Conference. National Reading Conference, Milwaukee • Rich, J. (2015) Employability: degrees of value. HEPI Occasional Paper 12. Higher Education Policy Institute, Oxford http://www.hepi.ac.uk/2015/12/10/employability-degrees-value/ • Sander, P & Sanders, L. (2009) Measuring academic behavioural confidence: the ABC scale revisited. Studies in Higher Education, 34:1, 19-35 • Thorndike, E. L.; Bregman, E. O.; Tilton, J.; Woodyard, E. (1928) Adult Learning. Macmillan, Oxford (Abstract at http://psycnet.apa.org/index.cfm?fa=search.displayRecord&uid=1929-00108-000)
Notes de l'éditeur
- NB employability/ employment features as a theme from the start in HEFCE’s discourse. Skills are generally defined as the abilities or proficiencies to apply cognitive or practical knowledge and use know-how to complete tasks and solve problems in certain areas. A competence, in addition to including a skill, covers abilities, behaviours and knowledge that are fundamental to the use of such skill. A competence is a proven ability to use knowledge, skills and personal, social and/or methodological abilities, in work or study situations and in professional and personal development. See RAND report top of p7 for some of the skills and competencies that Ss might develop during their HE. learning gain enumerates actual change in performance between two points in time learning outcome describes the output level. Cf value added, which is typically measured by the difference between a student’s actual test score at a second point in time and his or her predicted test score at a first point in time [NB that value added models are used in performance tables and systems for school improvement in the UK, which is an international leader in the development and use of such models
- Earliest example I found was Thorndike et al 1928
- Recent International debates on how to measure the quality of education have heightened interest in this originally US-based concept in England and elsewhere. Interest has grown because learning gain has the potential to say more about the extent to which higher education fulfils its key mission of contributing to students’ knowledge, skills and understanding than traditional output indicators, such as the number of graduates. (RAND p1) Note that employability / employment is a key theme from the outset. The change in funding arrangements in UK HE (fees etc) over the last decade = pressure to justify to students, employers and governments the investment in higher education. Commission on the Future of Higher Education in the United States (the ‘Spellings Commission’) 2006 - questioned the learning outcomes from US higher education, and recommended that post-secondary institutions should measure and report ‘meaningful student learning outcomes’ (ie learning gain). (OECD) feasibility study to compare generic and discipline-specific skills in economics and engineering across HEIs: Assessment of Higher Education Learning Outcomes (AHELO). The Bologna Process, a European intergovernmental series of agreements, also covered the debate during discussions on learning outcomes.
- Interestingly ambiguous choice of wording, where ‘value’ often seems to equate to ‘employment prospects’ or ‘earnings’. Is this fundamentally about money, and is this now all that really matters as an outcome of a university education, or am I just being cynical? RAND says ‘Value is measured not only in terms of the qualification acquired, but also in terms of what is learned during higher education - including the growth in what students know/understand/can do throughout their degree programme’ – but to what end? Note that the HEAR does to some extent address the need to specify what has been learned rather than simply on the results of students’ formal assessment. It does not, however, demonstrate progress made during the course of studies.
- What students really gain is (or should be) the transferrable skills that employers say they want. Note findings of very recent experimental report into graduate earnings (Institute for Fiscal Studies) using HMRC PAYE and Student Loans Company data for graduates, with real data on earnings ten years’ post-graduation by subject.
- Green paper and TEF consultation is primarily concerned with teaching excellence (among other themes), but explicitly identifies learning gain as an aspect of excellent teaching that should be measured. So it seems that measurement of learning gain is coming, whatever our views.
- Wabash presentation on HEFCE LG website. 49 colleges, 17,000 students, very wide range of measures to identify not just LG, but also the types of intervention, activity or good practice that will help students to improve in a liberal arts context
- UK is behind the US and Europe in the use of LG measures • There are two main types of standardised test: o Discipline-specific test, such as the Progress Testing administered in some schools of medicine in England. o Generic skills test, which are not discipline specific, such as the Collegiate Learning Assessment (CLA), which relies on open-ended questions and the Measure for Academic Proficiency and Progress (MAPP), which is a more traditional test relying on multiple choice questions. Both these examples originated and are
- ‘It is important to note that unintended consequences observed in the context of value added measurements in secondary education, such as ‘gaming’, could also apply to the context of higher education if the objective is to use the measure for accountability purposes’ [such as TEF] ‘Learning gain should not be considered as a sole determinant of the quality of higher education. Quality is relative to the many missions and goals that higher education has - there is both a broader conception of what is meant by quality of learning than solely learning gain, and there are more aspects to what is understood as quality of higher education in institutions, than just quality learning’
- These are the principles or criteria which acceptable measures of LG must meet See also p. 17 of RAND report for a framework of questions to use when evaluating methods to measure learning gain. These look at: Validity and comparability (subjective or objective tests, and how validated) Longitudinal or cross-sectional data, LG or value added measure Representativeness (eg sample size)
- Survey 1 at entry, surveys 2 & 3 in lectures to maximise response rates; have got ethics clearance, ready to start (Brunel already started data collection)
- Brunel – will investigate impact of 2 interventions on student performance in ABC – peer assisted learning and ‘Headstart’ (a week long transition programme for WP students covering study skills, support sources etc) Roehampton is also using the ABC questionnaire in a handful of volunteer faculties, and may also follow up with focus groups to get feedback on the ABC tool and questions and to verify answers UEL will administer ABC on a larger scale, currently across their Faculties of Education and Psychology
- NB Also Durham skills audit – pre entry self-report Qs, linked to Durham graduate attributes. See vignette in RAND report P.28
- RG consortium includes 17 RG HEIs. Led by Warwick. workstream leads are Birmingham, Nottingham, Cambridge, Warwick. Some RG HEIs (eg Manchester, Uni of London) also leading separate projects. University College London, University of Birmingham, University of Bristol, University of Cambridge, University of Durham, University of Exeter, University of Leeds, University of Liverpool, University of Manchester, University of Nottingham, University of Southampton, University of Sheffield, University of York, Imperial College London, King’s College London, Newcastle University, Queen Mary University of London
- Embedding transferrable or employability skills into the curriculum is not a new idea, but are you doing enough of this – and assessing it – to make sure that Sunderland will perform well in any new measure of LG?