Keynote Designing Assessment, Assessing Instructional Design
•Télécharger en tant que PPTX, PDF•
1 j'aime•1,132 vues
Assessment plays a vital role in delivering, evaluating, monitoring, improving and shaping learning experiences on the Web, at the desk and in the classroom. In the process of orchestrating educational technologies instructional designers are often confronted with the challenge of designing or deploying creative and authentic assessment techniques. The talk provides examples of the conceptual development and implementation of assessment approaches in three different areas: Needs Assessment: At the outset of an instructional design project, we work with stakeholders to gather data that helps us to reach the audience effectively and design user-friendly interfaces. Typical techniques are focus groups, surveys, qualitative interviews, personas and scenarios. Impact Assessment: Once the program or project is launched, we seek to understand how learners access online material or move through the curriculum, which helps us improve their experience. Data sources comprise Web analytics, social media metrics, learning analytics, surveys and interviews. Classroom Assessment: In the classroom, we aim to implement assessment techniques that support students’ critical thinking abilities and transfer learning skills. This includes peer-to-peer assessment, rubrics, portfolios and problem-based learning.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à Keynote Designing Assessment, Assessing Instructional Design
Similaire à Keynote Designing Assessment, Assessing Instructional Design (20)
Plus de Stefanie Panke
Plus de Stefanie Panke (20)
Dernier
Dernier (20)
Keynote Designing Assessment, Assessing Instructional Design
- 1. Designing Assessment, Assessing Instructional Design TCC Hawaii 2014 Stefanie Panke
- 2. Aloha Ya’ll!
- 3. From Pedagogical Concepts to Practical Applications
- 5. AACE Special Interest Group ‘Assessing, Designing, and Developing E- Learning’ http://panke.web.unc.edu/tcc2014
- 6. “Practically everybody in the academic community gets assessed these days, and practically everybody assesses somebody else”(Astin, 2012).
- 7. “We need to get past the emulation society, where instead of learning you spend all your energy on convincing others what you know” “Assessment is the fabric of education, if you change assessment, you change everything” Ambjorn Naeve, Tel-Map Group Expert Meeting, Lisbon 2011
- 8. Learning and assessment are two sides of the same coin, and they strongly influence each other (Gulikers, Bastiaens & Kirschner, 2004).
- 9. Assessment = Student Feedback “Effective learning occurs when students receive feedback, i.e. when they receive information on what they have (and have not) already learned. The process of generating such information is assessment” (Entwistle, 2000).
- 10. Assessment = Learning Organization Catalyst “We view assessment as the gathering of information concerning the functioning of students, staff and institutions of higher education. The basic motive is to improve the functioning of the institution and its people”(Astin, 2012).
- 12. From ‘assessment of learning’ towards ‘assessment for learning’ Knowledge imparted by the instructor (input) Competencies students can apply (output).
- 13. “To change student learning in the direction of competency development, authentic competency-based instruction aligned to authentic competency-based assessment is needed” (Gulikers, Bastiaens & Kirschner, 2004).
- 14. “For learning objectives not easily measured by multiple choice or true/false questions, assessment and evaluation can be time- consuming and difficult” (Anders, 2012).
- 15. The growth of virtual conferences has high potential as a tool for a more authentic assessment in higher education graduate programs (Menchaca, Ho & Hoffmann, 2013).
- 16. Five dimensions of authentic assessment: (a) the task, (b) the physical context, (c) the social context, (d) the results (e) the criteria. Gulikers, Bastiaens & Kirschner (2004)
- 17. “The perception of what authenticity is may vary among individuals as a result of educational level, personal interest, or amount of professional experience” (Gulikers, Bastiaens & Kirschner, 2004).
- 18. Threshold Concepts (Meyer & Land 2003, 2005): • Transformative • Integrative • Oftentimes bounded • Probably irreversible • Potentially troublesome
- 19. Designing creative assessment helps students unfold emerging understanding of key concepts.
- 20. Moving on…
- 21. Examples &Tools (1) Needs Assessment (2) Impact Assessment (3) Classroom Assessment
- 22. Needs Assessment • Stakeholder input • Example: Website Redesign Process • Workshop Series: – Audience – Content – Categories – Navigation
- 23. Audience • Goal: Identify potential audiences for the online resource • Exercise: Personas • Structure: Participants create personas in groups (approx. 20 minutes), present and vote on most important / fitting representations of their audience.
- 24. Content • Goal: Identify organizational output, recurring website elements Exercise: Content Toolkit • Visual building blocks • Create representation of different assets (books, news, user profiles, articles….)
- 25. Results
- 26. Categories 1. Review existing categories 2. Label sample content 3. Taxonomy on the fly
- 27. Examples &Tools (1) Needs Assessment (2) Impact Assessment (3) Classroom Assessment
- 28. Impact Assessment • Formative evaluation of material, learning environment, curriculum • Example: Learning Analytics – MPA competence assessment process
- 29. Data is the new oil – are learners the new sardines?
- 31. “Deciphering trends and patterns from educational big data” (Horizon Report 2013). “The measurement, collection, analysis and reporting of data about learners and their contexts” (SOLAR 2014).
- 34. • Analytical rubrics facilitate the assessment of student performance in central learning outcomes. • Collaborative effort of instructional designer, program leaders and faculty. Rubrics
- 35. Rubrics
- 36. Examples &Tools (1) Needs Assessment (2) Impact Assessment (3) Classroom Assessment
- 37. Classroom Assessment • Support students’ critical thinking abilities and transfer learning • Example: Portfolio – Infrastructure – Scaffolds – Peer Learning – Assessment – Evaluation
- 38. Portfolio Components • WordPress Multisite • Sample Portfolio • Portfolio Guide • WordPress Training • Competence Memo Template • Peer Feedback and Milestones in PUBA 746 • Portfolio Assessment Rubric • Portfolio Evaluation (Questionnaires, Focus Groups, Interview Guides)
- 39. Benefits of e-portfolios • Fostering digital literacy /multimedia storytelling (Wakimoto & Lewis, 2014). • Supporting career development (Reese & Levy, 2009). • Strengthening organizational ties (cf. Lorenzo & Ittelson, 2005; Reese & Levy, 2009). • Encouraging reflection (Roberts, Maor & Herrington, 2013). • Promoting self-regulation (Abrami et al, 2008, Meyer at al, 2010). • Improving knowledge management (Chang, Tseng, Liang & Chen, 2013). • Acknowledging diversity and transfer learning (O’Toole, 2013).
- 40. Types of Portfolios (Hewett, 2004) • Documentation portfolios: growth toward learning goals • Process portfolios: phases of the learning process • Showcase portfolios: accomplishments and competences
- 41. What nuggets can you use in your work or personal environment?
- 42. Thanks you – have a good conference!
Notes de l'éditeur
- Aloha Ya’ll! My name is Stefanie Panke, I work as an instructional analyst at the University of North Carolina at Chapel Hill for the School of Government. I am native German, and have worked for universities and research organizations in both Germany and the United States. My research interests comprise personal learning environments, online communities, open educational practices, digital publishingand instructional design in higher education. This is my third virtual visit to the TCC online conference and I am excited and honored to be part of this event. If you have not already done so, I would like you to use the chat window to introduce yourself so I have a chance to get to know who you are.
- In the next hour, I invite you to think about the role of assessment in instructional design. Why assessment? When Curtis Ho asked me about a keynote contribution, we discussed several possible topics – open educational practices, e-books and digital publishing, for all of which I would have had a presentation ready to pull out of the drawer. But somehow assessment emerged at the top of the list – which has made my life rather stressful for the past few weeks… but I am also excited to share something new and unique at this conference. Let me give you a bit of an outlook. I roughly divided the talk into two parts. I will spend approximately the first 25 minutes on conceptual background and the second halfon sharing concrete tools and examples. I have reserved 10 minutes in the end to take questions, and there will be polling and discussion prompts throughout the talk.I am looking forward to learning from you and use your feedback and input to refine my own thinking about assessment.
- When you build something from scratch, the nice thing is that the process itself is a learning experience and I thought it would be a neat idea to share this. I have created two web resources that reflect the thought processes and resources that went into this presentation. One is an annotated bibliography on assessment, the other is an assessment tool box, with creative assessment instruments and ideas. Both are creative commons so you can download, edit and add to it.
- Assessment was an obvious topic of choice for this talk Curtis and I currently co-chairing the AACE Special Interest Group on ‘Assessing, Designing, and Developing E-Learning’ (ADD-E-Learn). AACE launched this initiative at last year’s E-Learn conference, and the group currently has around 50 members, who are loosely organized on LinkedIn and Academic Experts. Why do I mention this? Membership is free and open, so I want to take the opportunity to invite all of you join the Special Interest Group.
- Another reason why assessment emerged as a central theme for this talk, is its ubiquitous nature. A few years ago, I assessment was merely on my peripheral vision – I did not really think about it as an interesting topic. I was far more intrigued by informal learning. open educational practices and personal learning environments. However, in the past couple of years I had project after project after project dealing with assessment. At the School of Government, we implemented a unified approach to course evaluation, the Carolina Master of Public Administration program started to conduct competence assessment and implemented student e-portfolios as a graduation requirement and we are currently conducting a series of needs assessment workshops. These projects have led me to rethink the connection between instructional design and assessment. Poll: But maybe that’s just me. I would like to know about you and your practice, so here is the first poll: “Practically everybody in the academic community gets assessed these days, and practically everybody assesses somebody else”(Astin, 2012). Let’s put this statement to the test.
- My first intense debate about assessment was at an expert meeting about ‘Futures for Technology Enhanced Learning (TEL)’ that tool place June 2011 in Lissabon. It had great participants, among others David Kennedy, James Morrison, George Siemens. The goal of the meeting was to talk about our vision for technology enhanced learning within the next ten years. As one can imagine, this prompt lead to a diverse collection of topics. After the initial discussion, the moderator asked us to focus on assessment as “the fabric of education”. The group focused on ways in which assessment can become a part of the learning process, not an end in itself, where we can focus on mastery, not performance. The shared vision was to leverage educational technologies for exactly this purpose.“Assessment is the fabric of education, if you change assessment, you change everything”. Agree? Disagree?
- Maybe not the fabric of education, but I think it is safe to say that assessment does play a vital role in delivering, evaluating, monitoring, improving and shaping learning experiences on the Web, at the desk and in the classroom. Learning and assessment are two sides of the same coin, and they strongly influence each other (Gulikers, Bastiaens & Kirschner, 2004). Consequentially, instructional designers are frequently confronted with the challenge of designing or deploying assessment. From my experience, oftentimes, the next question is ‘what coin are we looking at?’ In the field of instructional design, we juggle several, different currencies.Let’s have a look at assessment from two different, distinctively singular angles.
- The first perspective simply equates assessment with giving feedback to the learner. The idea is that to learn effectively, we need to know how well we are doing and the goal of assessment is to provide information on what we have and haven’t learned so far. Poll: Do you agree with this definition? YES /NO
- The second perspective sees assessment as a holistic endeavor that goes beyond giving feedback to an individual learner. Instead, the focus of assessment is on the learning organization - the functioning of the institution and its people. By gathering information on students, staff and organizational units, assessment seeks to improve the learning organization. Poll: Do you agree with this definition? YES /NO
- In instructional design, we deal with both perspectives on assessment – feedback for individual learning and data for organizational improvement. In both areas we feel tensions between measure and treasure. The assessment criteria we express are not or not always the learning outcomes we have in mind. For individual learning, there is a growing dissatisfaction with standardized test scores. On the organizational level, we hear criticism about program rankings and the prevalence of the social science citation index. What we measure through standardized tests and metrics is not necessarily what we treasure. In the following, I want to focus on authentic assessment as a goal for instructional design. More and more, the task of designing assessment does not equate to creating really slick multiple-choice quizzes, but instead calls for creative and authentic assessment techniques. Critical thinking, communication, problem solving, creative innovation - all these are learning objectives that cannot be easily measured through multiple choice tests.
- Fostered by the rise of constructivist learning theory, authentic assessment, a.k.a. performance assessment as well as connected approaches and tools like rubrics, portfolios and competency-based learning outcomes have been discussed in educational research since the mid-nineties. This paradigm shift from ‘assessment of learning’ towards ‘assessment for learning’ plays an important role for changing from input to output orientation of teaching and learning and support students’ critical thinking abilities. Instead of assessing how well students can reproduce knowledge imparted by the instructor (input), the focus shifts to the competencies students can apply (output).
- Learning and assessment are two sides of the same coin. That means that learning goals and assessment techniques need to be aligned. If we want to change student learning in the direction of competency development, we need to align problem-based, competency-oriented instruction with authentic assessment.
- For instructional designers this means to shift from creating quizzes towards designing activities. This is more time-consuming and difficult and it is less clear what will and will not work. However, technology is a great catalyst for authentic assessment approaches.
- This conference itself is certainly a best practice for how to conduct authentic assessment. Graduate students from Department of Educational Technology at the University of Hawai'i at Manoa take a two-semester course sequence in their last year in which they design, implement and analyze their capstone project and ultimately they present the results at the TCC online conference. That is a great example of having a legitimate performance experience. The community of practice that students get exposed to is usually primarily limited to the immediate department where they are enrolled. In this case, technology offers the potential for international interactions that can link to a far wider community and encourage future professional connections. The growth of virtual conferences is not only positive for faculty impacted by budget cuts, but has high potential as a tool for authentic assessment in higher education graduate programs.
- Well, if I lack the Aloha spirit to partner with a successful online conference, what are alternatives? How can instructional designers create activities that are meaningful, contextualized and connected to real-world problems? Though there is no alchemistic formula, it is important to understand that authenticity is a continuum. Gulikers, Bastiaens & Kirschner (2004) distinguish five dimensions of authentic assessment: (a) the task, (b) the physical context, (c) the social context, (d) the results, and (e) the criteria. Each dimension forms a continuum, which means that authenticity is not an all or nothing trait.
- Second, keep in mind that authenticity is a subjective measure.What counts as authentic is likely to vary from student to student and depends on the individual learner’s educational trajectory, personalinterests, and life experiences.
- A great way to frame assessment in the disciplines are threshold concepts. The idea of threshold concepts emerged from a national research project in the UK, where researchers looked into the possible characteristics of strong teaching and learning environments for undergraduate education.Disciplines have ‘conceptual gateways’ or ‘portals’ that lead to a previously inaccessible way of thinking in a process of liminal transition - these are ‘threshold concepts’. Meyer and Land (2003, 2005) characterize threshold concepts with the following qualities: transformative (significant shift in the perception of a subject), integrative (exposing the previously hidden interrelatedness of something), oftentimes bounded (demarcating academic territories), probably irreversible (unlikely to be forgotten, or unlearned only through considerable effort) and potentially troublesome (often problematic for learners, because the concept appears counter-intuitive, alien, or incoherent). An example from the social sciences is that ‘you cannot make causal inferences from correlational data’. Mastering a threshold concept puts learners in a liminal state where they oscillate between old and emergent understandings - just like an ethnographic researcher who not outside, but also not quite inside the group. So one way to think about assessment is to identify the threshold concepts in the domain you are working on and coming up with creative ways to help learners traverse these portals.
- Often times, when we talk about ‘authentic assessment’ in the instructional design process, we really mean creative assessment. We are looking for techniques that are engaging, surprising, puzzling, challenging, unexpected or different. What we want to support is assessment for learning, that encourages students to question their preconceptions and can help them to evaluate their grasp of crucial concepts in their discipline. This can happen in many ways – creating a mindmap, producing a comic strip, developing an information graphic, creating a game. It does not necessarily mean to be as close as possible to the ‘real world’. In fact, the dichotomy higher education as ‘unreal’ and everything outside as ‘real’ is itself a construct that is not necessarily productive.
- Finally, I don’t want to come across as overly critical of standardized, multiple choice testing – in fact, this is really not my area of expertise. I completed my education up to my PhD in Germany, back ‘in the good old days’, prior to Bologna Reform. Thinking back on my personal educational experience, I have to say that the only standardized multiple choice quiz I ever took was the theoretical part of my driver’s license test. And it might be worth mentioning that I passed that one with flying colors. However, I did fail the practical part the first time around. And, while on second try I eventually did get my permit, I still remember the tester saying, ‘Young lady, I would never let you drive my car’.
- Okay, let’s move on to something I am actually good at, or at least better than parallel parking. We have arrived at the second half of the show and I now want to move on to three practical examples that illustrate creative and authentic assessment approaches. I will provides examples from three different areas: (1) Needs Assessment, (2) Impact Assessment and (3) Assessment for Learning.
- 1. Needs Assessment: At the outset of an instructional design project, we work with stakeholders to gather data that helps us to reach the audience effectively, design user-friendly interfaces or formulate an organizational strategy. Typical techniques in needs assessment are for instance focus groups, surveys, qualitative interviews, personas and scenarios. As an example for creative techniques in needs assessment, I want to share something very recent that is literally sitting on my desk right now. The School of Government is currently in the process of redesigning the School’s website and we are organizing a series of workshops to understand the needs of faculty, staff and our external website users.
- The first website workshop exploredthe School of Government website from the perspective of the external visitors. We used the ‘personas technique’ to create and share narratives.POLL: Who has used personas before?
- During the second workshop, we look at patterns and reoccurring elements on the website. Once content is structured in a logical and functional form, it is easy to filter the information, provide customized views and offer flexible building blocks for aggregating and updating information. During the workshop, we brainstormed different types of resources relevant to the organization (e.g., articles, books, reports, faculty profiles, news, events, courses…). We then used visual building blocks to represent each content type.
- The goal was to leverage content types to display the content in more than one place and more than one way. The participants came up with 15 different content types we could then translate into mockups of the entry form and web view.
- During the third workshop, we discussed metadata – information about information. Categories and tags allow us create dynamic connections between different content items. I learned that the School has several, overlapping ways in which we organize information, for instance in our course and book catalog, on the website, in the faculty expertise list or in the advising reporting tool. Hence the goal of the workshop was to come up with one taxonomy that works across different contexts.
- Okay, let’s move on to something I am actually good at, or at least better than parallel parking. We have arrived at the second half of the show and I now want to move on to three practical examples that illustrate creative and authentic assessment approaches. I will provides examples from three different areas: (1) Needs Assessment, (2) Impact Assessment and (3) Assessment for Learning.
- 2. Impact Assessment: In instructional design our involvement usually carries on after the program or project is launched. At this point, we seek to understand its impact and conduct formative evaluation. We want to know how people use the material, move through the curriculum or interact with the learning environment. Ideally, we not only collect this data, but turn it into activities and adjustments that help improve the learners’ experience. Data sources for assessing impact are manifold and may comprise Web analytics, social media metrics, retention data, registration rates, course evaluation results, sales reports, but also informal feedback or questionnaires. In the light of omnipresent budget cuts, higher education institutions feel an ever greater need to demonstrate impact. The School of Government has launched several initiatives in the past couple of years that deal with impact assessment. We introduced a standardized course evaluation, we monitor the advising our faculty do through an online reporting instrument, we track participation in courses and webinars, we monitor the website traffic and we have created an online form which asks people to ‘tell a story about how the School made a difference’. The example I chose to look at more closely comes from the Carolina MPA program and falls in the domain of learning analytics.
- Data is the new oil – this saying has gained prominence not only in business and sales, but also in education. Learning Analytics has evolved as a central educational technology trend over the past few years.
- Usually when you hear learning analytics, you think about dashboards embedded in a learning management system that gives feedback to both learners and teachers. Here is an example from a Blackboard analytics module. However, I think learning analytics does not necessarily have to look like this.
- In fact, there are two different ways to conceptualize learning analytics, a broad and a narrow definition. Narrow: Smarter version of intelligent tutoring systems, where the user model is dynamic and stems from ALL the different traces that previous users have left in the system. Broad definition: Any way in which organizations collect and use comprehensive data on students’ trajectories in a systematic way to improve learning experiences. SoLAR Society for Learning Analytics Research
- My example is a recent project on competency assessment at a program level. Prompted by a shift in accreditation standards that call for a competency-focused curriculum, the Carolina MPA has undergone a complete curriculum redesign. The faculty committee developed a set of eight broad competencies with 24 intermediary competencies that together define the learning outcomes the program seeks to impart. Starting in fall 2012, all program requirements and basic course content are built around these competencies.
- Let’s zoom in to one specific learning outcome.The intermediary competencies break down the major learning outcome into concrete areas of knowledge and applicableskillsets. In this example, the overall learning outcome ‘Analyze information for decision making’ is broken down into 5 distinct competencies that are relevant for public service leaders.
- How can we collect data from different instructors and ensure that they have a shared understanding of the competence? In the context of authentic assessment, rubrics are a popular tool to measure student achievement. To specify what it means to be or become competent, we use rubrics as an assessment tool. In close collaboration of instructional designers, program leaders and program faculty, eight analytical rubrics were developed to capture how well students are performing in the central learning outcomes.
- The rubrics specify different levels of attainment for each intermediary competency. Each student receives a score between 1 and 5. At an entry level, the student understands what it takes to become competent. At an evolving level, the student is learning relevant skills. At an accomplished level, the student is able to perform tasks that demonstrate the ability.
- Okay, let’s move on to something I am actually good at, or at least better than parallel parking. We have arrived at the second half of the show and I now want to move on to three practical examples that illustrate creative and authentic assessment approaches. I will provides examples from three different areas: (1) Needs Assessment, (2) Impact Assessment and (3) Assessment for Learning.
- There are many techniques to implement assessments that support students’ critical thinking abilities and transfer learning - drawing a mindmap, creating an online comic strip, producing a podcast, developing an information graphic, editing a wikipedia entry, creating a game or engaging in an online community. But since I only have time to talk about one thing, I want to focus on portfolios since they are probably the #1tool of choice to monitor and demonstrate competence development.
- We use the UNC Wordpress Multisite web.unc.edu as an infrastructure for our student portfolios. We scaffold the portfolio process by giving clear guidelines for the design, navigation and portfolio components and providing a template for the reflective memos. For the portfolio assessment, we designed a rubric that the faculty committees were able to use. We are currently in the process of evaluating our portfolio process.
- To this end, I did a lit review of portfolio outcomes so we can see of we achieve this.Benefits of e-portfolios documented in the literature include:Fostering digital literacy and multimedia storytelling: By creating digital exhibitions spaces of their work, students gain technology, writing and multimedia communication skills en passant (Wakimoto & Lewis, 2014).Supporting career development: The digital collection of work samples and skill demonstrations can be easily shared with potential employers (Reese & Levy, 2009).Strengthening organizational ties: E-Portfolios may link students to their alma mater even after graduation and can be used to connect alumni and prospective students (cf. Lorenzo & Ittelson, 2005; Reese & Levy, 2009). Encouraging reflection: E-portfolios enable learners to review their learning processes and outcomes by self-reflection and comments from peers (Roberts, Maor & Herrington, 2013).Promoting self-regulation: Portfolios that stress the reflective component have the potential to raise students’ metacognitive awareness (Abrami et al, 2008, Meyer at al, 2010).Improving knowledge management. The process of conceptualizing, implementing and developing an e-portfolio involves collecting, arranging, re-organizing and presenting information. E-portfolios can thereby facilitate knowledge management performance (Chang, Tseng, Liang & Chen, 2013). Acknowledging diversity and transfer learning: Portfolios are a natural fit for assessing networked learners in their personal learning environments Panke, 2013). The e-portfolio brings diverse student outputs from a range of different learning and working contexts into one common format and thus facilitates assessment (O’Toole, 2013).This makes portfolios sound like the proverbial jackalope. However, not all of the benefits will manifest in every instance of a portfolio centered assessment. It is important to keep in mind that there are different portfolios for different purposes.
- Hewett (2004) distinguishes three basic types of portfolios that support different assessment purposes:Documentation portfolios show the growth toward achieving specific, pre-defined learning goals. They support diagnostic assessment and allow students and instructors to both plan and check how the learner is progressing. Process portfolios document the phases of the learning process and reflect upon the students’ journey towards mastery. They make students cognizant of how they learn best and support self-assessment of learning strategies. Showcase portfolios focus on the students' accomplishments and competences. They include the students' best works and reflections on how and why the work products were selected. They support summative assessment of students’ competences and learning outcomes.
- Summary:I know that I have covered a lot of ground during the past 45 minutes, so let me give you a quick recap: I have talked about facets of assessment from an instructional design perspective, I have argued for authentic assessment and I have given three specific examples for assessment projects. I am ready to take questions now, but I also want to encourage you to engage further in the online discussion forum.Teaching without learning is just talking (Angelo & Cross, 1993) I would like you to take a minute, think about what you have heard and write down Ideas or concepts that that you plan on using for your work.Ideas or concepts that are maybe useful and worth exploring furtherIdea or concept that you find not useful.