How to Improve Library Instruction: Assessment in Five Minutes
1. How to Improve Your Library
Instruction: Assessment in Five
Minutes
Sarah Steiner
January 10, 2013
2. During Today’s Class, We Will…
• Analyze the purpose and benefits of quick
assessment in one-shot instruction sessions.
• Draft learning outcomes for a session using Bloom’s
Taxonomy.
• Compose assessment questions based on our
learning outcomes.
• Select appropriate assessment tools based on your
instruction goals and population.
http://www.flickr.com/photos/topgold/4858921901/
3. Our Focus
• Classroom assessment
(versus programmatic or institutional)
• Fast and free assessment tools
– Polls
– Quizzes/Questionnaires
– Minute papers
http://www.flickr.com/photos/rogersg/3814863064/
4. What Can Quick Classroom
Assessment Do?
• It can identify instructional gaps or disconnects.
• It can help you determine how you spend class
time.
• It can build your confidence.
• It can provide evidence of efficacy and impact.
• It should be a basis for change.
http://www.flickr.com/photos/sercasey/324341982/
5. What Should I Assess?
• To assess:
– Comprehension
– Retention
– Achievement of learning outcomes
• Not to assess:
– Your personal shortcomings
– Your speaking mistakes
– Your hairstyle
– Random things
http://www.flickr.com/photos/toniblay/52445415/
6. Have you ever written learning
objectives or outcomes?
• Nope, not yet.
• Yes, a few times.
• Yes, regularly.
• I’m not sure.
http://www.flickr.com/photos/dadaistic_fantastic/5089525186/
7. Learning Outcomes
“A Learning Outcome is a statement of what
the student should understand and be able to
do as a result of what she has learned ... ‘the
essential and enduring
knowledge, abilities, and attitudes or
dispositions’ that enable a learner to practice
and apply her learning in the real world.”
-Valencia Community College
http://www.flickr.com/photos/gblakeley/5583354276/
8. A Good Learning Outcome Will…
• Identify the audience.
• Set a time frame and a context.
• Be jargon-free.
• Be measurable.
• Be action-oriented.
• Be brief.
• Be linked to learner needs.
http://www.flickr.com/photos/auntiep/4310267/
9. Determine the Class Priorities
• Choose three to five
– What must the students accomplish?
– What must they know in order to
accomplish it?
– What do they already know/find
obvious?
http://www.flickr.com/photos/paloetic/4795592340/
17. Choose Verbs: Synthesis
Students will…
categorize, combine, compile, compose, create, devis
e, design, explain, generate, modify, organize, plan, r
earrange, reconstruct, relate, reorganize, revise, rewr
ite, summarize, tell, write
http://www.flickr.com/photos/littleredelf/3081832532/
18. Choose Verbs: Evaluation
Students will…
appraise, compare, conclude, contrast, criticize, criti
que, defend, describe, discriminate, evaluate, explai
n, interpret, justify, relate, summarize, support
http://www.flickr.com/photos/christinamatheson/108818379/
19. Add Product or Outcome
What do they need?
• A book on their topic
• A thesis statement
• Keyword search strategies (basic or advanced)
What is the context?
• A class assignment
• A real-world scenario
http://www.flickr.com/photos/fdecomite/2907794565/
20. All Together Now!
Stem + Verb + Product/Outcome
By the end of this class, you will be able to
identify a scholarly source.
By the end of this session, you will be able
to locate books on your paper topic in the
library using the online catalog, GILFind.
http://www.flickr.com/photos/revlimit/2186765034/
21. What’s Wrong Here?
Today I’ll talk about…
1. The library website.
2. Database searching for peer-reviewed
journals using boolean logic and the
thesaurus.
3. How you can search the catalog, search
the databases, and use ILL.
http://www.flickr.com/photos/mythoto/2604818537/
22. Question Time
http://www.flickr.com/photos/sugarhiccuphiccup/5485691671/
23. Have you ever assessed
students’ achievement of pre-
set learning outcomes?
• Nope, not yet.
• Yes, a few times.
• Yes, regularly.
• I’m not sure.
http://www.flickr.com/photos/dadaistic_fantastic/5089525186/
24. Today’s Assessment Tools
• Polls
• Quizzes (pre and post or post only)
• Minute papers
http://www.flickr.com/photos/azarius/225340667/
26. Poll Example Questions
• How many of you have had a library
instruction class here before?
• Is this an article or a journal?
• Is this source peer-reviewed? / Is this
source scholarly or popular?
• Who is the author of this book?
• Are you confused about topic x?
http://www.flickr.com/photos/rakka/3451170932/
32. Best Practices for Question
Drafting
• Convert your learning outcomes to
questions.
• Include three to five “meaty”
questions, two or three general
questions.
• Avoid overly easy or “all of the above”
answers.
• Avoid nebulous or complicated
questions.
• Work in terminology to ensure it’s
understood.
http://www.flickr.com/photos/djmccrady/6118626233/
33. Best Practices for Question
Drafting
• Get a reviewer.
• If at first you don’t succeed….
More help with constructing questions:
http://ets.tlt.psu.edu/learningdesign/effective_
questions/why
http://www.flickr.com/photos/djmccrady/6118626233/
34. Review Responses
• On the spot? Yes!
• Look for…
• Percentage of correct answers.
• Trends in missed answers.
• What to change next time.
• Keep your chin up.
http://www.flickr.com/photos/xingty/3421423768/
41. Review
• Write learning outcomes.
• Base your questions on the
outcomes and on your needs.
• Review and reflect on the answers
you get.
http://www.flickr.com/photos/demonbaby/4228327167/
43. Other Resources
• “Bloom’s Taxonomy of Cognitive Domain.”
http://classweb.gmu.edu/ndabbagh/Resources/Resources2/bloomstax.htm
• Educational Oasis, “Framework of Learning Outcomes.”
http://www.educationoasis.com/instruction/bt/learning_objectives.htm
• London Deanery, “Setting Learning Objectives.”
http://www.faculty.londondeanery.ac.uk/e-learning/setting-learning-
objectives
• Penn State, “Writing Effective Questions to Promote Learning.”
http://ets.tlt.psu.edu/learningdesign/effective_questions/
• Utah State University Library, “Assessment.”
http://library.usu.edu/instruct/assessment/index.php
• “Virginia Tech Instruction Clearinghouse.”
http://www.lib.vt.edu/RIS/clearinghouse/assessments.html
Notes de l'éditeur
All images are creative commons, cited in the bottom right corner.
Choose your questions and model based on the outcomes you need to meet.
This session will focus on classroom. You have 50 or 75 minutes only—how can you fit in assessment?When we think of assessment, we often think it needs to be really involved in order to be useful. Of course, you can do hugely involved assessments, but that’s not the focus today. This is about how important and beneficial quick stuff can be. Our assessment doesn’t need to have pre and post tests, doesn’t need to span an entire semester, doesn’t need to have IRB approval and pre-testers and a million steps. Formative assessment is utilized to immediately determine whether students have learned what the instructor intended. This type of assessment is intended to help instructors identify material which needs to be clarified or re-taught and should not be used to evaluate or grade students. Results of formative assessment can assist instructors to ascertain whether curriculum or learning activities need to be modified during a class session or before the next class meets.Summative assessment is cumulative in nature and is used to determine whether students have met the course goals or student learning outcomes at the end of a course or program.
Example: terminology—article, journal, databaseAssessment is not an end in itelf—it supports change and best practice in teaching.
Did they take away what you thought they were going to take away? If not, why?My favorite lesson: don’t know what a database is. Don’t know the difference between an article and a journal. They tend to be highly constructive
Two part question—have you done both of these things together? There are other kinds of assessments—observation, initial questions—but today we’re going to knit these two things together.
Our first step in good, quick assessment is to write learning outcomes. The assessment questions which you choose will, in general relate specifically to those learning outcomes. They will also provide a roadmap for you, as the instructor, in terms of what you must cover. Whatever assessment method you choose later will tie back to these. A good learning outcome will state what the student will be able to do, and WHY. It will use Bloom’s taxonomy verbs of various cognitive levels.http://valenciacc.edu/facultydevelopment/tla/Candidate/documents/ExamplesandDefinitionsofLOs.pdfDifferent from an objective. Objective states what we aim to do. Outcome describes what the student will be able to do at the end. Learning outcomes are relevant to every part of your tiny assessment plan. Help you to focusHelp to reduce students being overwhelmed
Review the outcomes for this sessionWhich one might be hard to measure?
First step: consider the top three or four class priorities. Write them down. Find a bookIdentify a scholarly articleLocate a scholarly article“Learning outcomes should flow from a needs assessment. The needs assessment should determine the gap between an existing condition and a desired condition. Learning outcomes are statements which described a desired condition – that is, the knowledge, skills, or attitudes needed to fulfill the need. They represent the solution to the identified need or issue. Learning outcomes provide direction in the planning of a learning activity.”http://www.aallnet.org/Archived/Education-and-Events/cpe/outcomes.html
Future tense—timeframeIdentifies learner (some people say “students”)
Benjamin Bloom, an American educational psychologist, helped to write these in 1956. Domains of learning. Cognitive domain. Also Affective and Psychomotor domains.I want to go through these in depth a little, because they will directly inform your assessment questions. So, as you choose verbs, consider them not just as objectives, but as assessment questions for your post test.
“Knowledge is the act of remembering previously learned material. It’s the lowest level of learning outcomes in the cognitive domain. “
“Comprehension is the ability to grasp the meaning of material. Students can demonstration comprehension through interpretation or summarization of material.These learning outcomes are a stepbeyond just remembering. They represent the lowest level of understanding.”
“Application refers to the ability to use learned material in new and concrete situations.”We do this a lot in one shots.
“Analysis is the ability to break down material so that its organizational structure can be understood. Differentiation, comparing and contrasting, identification.At this point you may notice that some of these verbs are on more than one list.
“Synthesis is the ability to put parts together to form a new whole.”It can be hard to get to this in an hour. Our students often do this at the course level—they have to write a paper, a speech, a bibliography, a literature review. Synthesis is the ultimate goal, in many cases.
“Evaluation is the ability to judge the value of material in a particular scenario. The judgments are to be based on definite criteria. Learning outcomes in this area are highest in the cognitive hierarchy because they contain elements of all the other categories, plus value judgments.
This should include context as often as possible without being too wordy or jargon-filled. So, for example, in this last I’ve said “advanced techniques” rather than saying boolean logic, subject header searches.We’ll get to those things, but not in our outcomes. You don’t have to have the context, but I like it. If you don’t include it in your written outcomes, still think about it and be able to articulate the context to the students.
Stem and outcomeDemonstrable/assessableBloom’s cognitive domain—high levelNo jargonOften INCREMENTAL—step-based. They fit together to make a whole.
Learner not identified—teacher focusNot assessablePoor verbNo verbJargon, too specificHUGE in scopeTAKE A BREAK
Two part question—have you done both of these things together? There are other kinds of assessments—observation, initial questions—but today we’re going to knit these two things together.
Just read through, no elaborationWe’re going to talk about each and which might be best for you, given your setting.
Polls are good for feedback throughout the session. They’re great if you have information that builds and is cumulative. So, you can force them to answer a few questions before you move on to the next topic. They’re also a fun active learning exercise and they give you immediate feedback.The questions have to be simpler, with no short answer or essay, but you can have more of them. You can set them up to be embedded on a site/LibGuide (if your students have computers), or use free tools to set them up as text messages. If you have a clicker system, you can use them as well. Not too expensive, but they do cost $$$. (1000 to 2000... Other?) You can take those polls and embed them into PowerPoints, and then the answers will show up there (PollEverywhere). They can also just raise their hands, of course!You can have students work together to submit a group answer, or you can give each individual a clicker. Drawback—can be distracting, can have technical problems, can take up a lot of class time.
Regardless of what method you choose to deliver the content (phone, embedded poll, clicker) you can have different question types. Multiple choice, yes/no.Substantive or friendly—can also help you to set the stage for what you’ll cover in the beginning. These can be great, because they are anonymous in the moment. So, if someone feels shy about raising a hand to say s/he is confused, this can help them to do it without drawing attention to themselves.
So, we’ve all had quizzes—they are my weapon on choice. You can set them to happen at the end, or at the beginning and end to gauge what they knew versus what they know. If you want to measure long-term retention, you can give them the same quiz at a later date. With both quizzes and polls, you can easily convert your learning outcomes to question, if they’re carefully drafted. So, “You will know how to find a book on your topic” becomes “find a book on your topic.”Not more than five “meat” questions, maybe a couple of close out questions that are a bit more free-form.
Pre and post, or just post. Pre and post can gauge learning, post can gauge what they’re walking out with. If you can get teacher cooperation, you can also do them again as a follow up, to gauge retention on various time scales.Some of these are open ended, some multiple choice. It’s a good idea to have a variety. Not too many easy ones, or you may not learn anything useful. You can also format these as quick assignments. Great for classes with one set task—evaluate and cite websites, for example. You can also give them an article citation and ask them to find it. Then describe briefly why they think it’s reputable.
You don’t need to be perfect your first time. If your questions aren’t perfect the first time or your instruction isnt perfect, that’s okay You will realize even as you get more accomplished, that you sometimes don’t say things perfectly! You can correct that on the spot. You can use answers as an opportunity to congratulate them for their awesomeness.
What is a minute paper? You have them write or type for just one minute—only two real questions (and maybe some basic information).On paper or online is your choice. It’s easier to tabulate things online though. Drawbacks:It’s hard to evaluate these in the moment, but you can. It’s also very hard to tie these in with your objectives—however, they are useful. I like to do a combination of hand-raising polling, with a post quiz that has a minute paper attached.
Gives them space to clarify and explain, if they want to. You can also invite them to leave contact info. You’ll see these final questions again in the minute paper, when we talk about that.
With polls and quizzes, if they’re brief, it’s easy to tabulate and review responses immediately. I like to do this. Skim over the answers and address problems right there. “It seems like I was unclear on Point X. Can someone describe the answer? HAVE YOUR ELEVATOR SPEECHES READY.This is the most important step. If you don’t integrate the information, it’s not useful to anyone. We’re not perfect—this kind of thing can be jarring at first. Change can relate to the way the questions are worded, ordered, or what is included, and also in the ways you teach.
Warn the students that it’s coming—but you don’t have to treat it like a test. Just say—”I’m trying to make sure that I’m explaining everything you need, so if you felt something was unclear, please do say so--please help out people who come after you!”Do it ten minutes prior to the end, if at all possible. That way they have five minutes to take it, and you have five minutes to do a final recap and address anything that looks problematic. Have your elevator speeches ready on each of the objectives—be able to talk about each one very quickly and clearly. It’s important that they know WHY this is important. You will use the information to make sure they know everything they need to know. You will use it to make your instruction more effective for future students.
Most of these can be used for both.
Basic: free10 questions per survey100 responses per surveySelect: $17 per month ($204 per year)Unlimited questionsUnlimited responsesCustom urlsSkip logic etc.
Free account: 100 responses per month LIMITED!!!Ten questions per survey or quizPRO account:$200 per year1000 survey responses per monthUnlimited questions per survey or quiz
Free version40 responses per pollPersonal$15 per month ($180 per year)50 responses per pollPresenter model $65 per month ($780 per year)250 responses per poll
Free, but somewhat clunky to use.
List of verbs:http://www.acu.edu/academics/adamscenter/course_design/syllabus/verbs.html