Abstract:
The ability to authentically capture and assess student opinions and growth has always been a challenge. Polls, surveys, and focus groups are some of the most common ways for administrators to gather assessment data at an institutional level; however, these indirect sources only serve as proxy indicators of student voices and experiences. And while ePortfolios may provide better and more authentic direct assessment, they are usually not faster or easier. How can we merge the efficiency of the forms, polls, and surveys with the power of ePortfolios to have a faster, better, and a deeper look into our students’ expectations? We will provide an overview of our current text mining assessment methods for over 2000 students ePortfolio reflections given specific prompts. These methods will provide a counter-narrative to indirect surveys that liberate all of our students’ unique voices and allow us to better align their aspirations in the arena of higher education.
Presenters:
G. Alex Ambrose, Program Director of ePortfolio Assessment, University of Notre Dame
Trunojoyo (Atun) Anggara, Academic Advisor and Educational Data Assessment Postdoctoral Fellow, University of Notre Dame
Liberate Learning through Next Generation Assessment -AACU 2018 Closing Plenary
1. Liberate Learning Through
Next Generation Assessment
G. Alex Ambrose, Ph.D, Professor of the Practice and ePortfolio Program Director
Trunojoyo (Atun) Anggara, Ph.D, Academic Advisor and Educational Data Assessment Postdoctoral Fellow
Kaneb Center for Teaching & Learning, University of Notre Dame
Closing Plenary at the 9th Annual Forum on Digital Learning and ePortfolios
January 27, 2018
4. Many meanings in this image...
What do you notice?
How do you react and with which emotions?
Can you relate and what does it mean to you?
What title or caption would you give it?
5. What does assessment in higher education
actually do to the American dream?
Free
Thrive
Liberate
Restrain
Harm
Limit
ASSESSMENT
LEARNINGTEACHING
HIGHER
EDUCATION
AMERICAN
DREAM
6. Assessment Design Questions
What role does technology play?
What problem does it address?
If technology is the answer, what was the question?
12. ● During academic year of 2016-2017:
First year advising at Notre Dame is likely to be different from the
advising you received in high school. Your advisor will assist you in making the
transition to college academics and accessing the resources ND has in place to smooth the way. He
or she will also help you understand and begin to make informed choices about the basics of the
University’s requirement and elective course system. You will talk with your advisor about your
academic hopes, concerns, and plans, and he or she will suggest paths and opportunities to fit your
goals. During your first year, with the assistance of your First Year advisor and others, you will begin
to build an education suited to your goals and interests. List the top three expectations you have for
your academic advising experience.
Advising Initial Thoughts prompt evolution:
2015-2016 to 2016-2017
● During academic year of 2015-2016:
List the top 3 of your expectations walking into an advising session.
13. What is the effect of prompt-priming with respect to student
response in Initial Thoughts?
● By the total number of words written:
○ 2015-2016: 78,479 total words
○ 2016-2017: 111,619 total words
○ Word increase of 42% between the two years, even though there is 17 less ePortfolios being
written in 2016-2017
Out of 1795 ePortfolios
Out of 1778 ePortfolios
14. What is the effect of prompt-priming with respect to student
response in Initial Thoughts? | Unigrams
2015-2016
Higher mentions since higher
number of words generated by
the students
2016-2017
Hope is less prevalent here
Major and academic related
words more prominent here
15. What is the effect of prompt-priming with respect to student
response in Initial Thoughts? | Bigrams
2016-20172015-2016
More mentions in:
● study abroad
● medical school
● career path
High school is mentioned less here
16. 1. Better prompt-priming resulted in more engagement (around 40% increase of
writing) from the First Year Students in 2016-2017 compared to 2015-2016
2. More substantive words (major, study abroad, medical school, etc. being
mentioned) in 2016-2017 compared to 2015-2016
First case Takeaways
24. Challenge
National Center for Educational Statistics https://nces.ed.gov/collegenavigator/
Can we identify those 2% (40 students) & help everyone thrive?
How can we ensure everyone has the opportunity to thrive?
25. Context: Course Bottom Up Student Assessment
Moreau First Year of Experience Course Scale & Stats:
- 2,000+ students
- 114 classes (class size of <19)
- 56,000 assignments turned in and graded with common rubrics per year
- 2.3 million click tracks and 15 million words text mined per year
26. Next Generation Digital Learning Environment (NGDLE)
Ambrose, G. Alex, Abbott, Kevin, Lanski, Alison (2017) “Under the Hood of a
Next Generation Digital Learning Environment in Progress” Educause Review.
29. Identify --> Notify --> Boost --> Evaluate
Subject:
Request for Immediate Moreau Action Plan
Message:
Dear Moreau Student,
As Assistant Dean and Co-director of the Moreau First Year Experience, I recently reviewed a report on the progress of all Moreau students over the first five weeks
of the semester.
I am sending this message to you as a way of offering assistance and support since the report indicates that you might be showing signs of struggling. We want
to ensure that you have access to any and all resources you might find useful.
Out of the 2000+ students in the course, you are one of only 39 (2%) who have not submitted or scored a 0 on at least two weekly prompts. In the past
two years of this course, this type of early performance is often associated with low midterm and final Moreau grades as well as a low overall semester
GPA.
In order for me to understand your situation better and for you to learn about helpful resources, please take two minutes to fill out this personal action plan. You
can also reply to this message to get in touch with me directly. If you haven't provided an action plan response within 48 hrs of this message, I'll plan to share the
information above with your FYS advisor to ensure that you are succeeding at the highest possible level.
30. Identify --> Notify --> Boost --> Evaluate
(Early)
Early Semester Learner Action Plan
31. ReasonsforMissingAssignment
Technical/Grading
Forgot to submit
Stressed about things on
campus
Struggling to keep up w/
classes
Stressed about other things
at home
Talk to instructor about
submission
Be more careful in the future
Talk to instructor about
submission
Contact Rector
Contact care consultant
Contact peer advisor
Contact counselor
Contact FYS advisor
Talk to instructor
Contact Program in
Academic Excellence
33. Thrive after Boost Impact
2,053 Total FYS
75 ID & Given a Boost
Identify --> Notify --> Boost --> Evaluate
1% Did not thrive with ≤B- grade
99% Thrived with ≥B Grade at final
13/75 Did not thrive after at least 1
boost
6 Students Crashed after Midterms
62/75 Thrived after at least 1 Boost
35. What role does technology play?
What problem does it address?
If technology is the answer, what was the question?
How can we move beyond efficient assessment to
more effective assessment thereby liberating the learning and the learners?
37. References
Related Presentations:
Ambrose, G. Alex, Martin, Holly, Page, Hugh (2014) “Linking Advising and ePortfolios for Engagement: Design, Evolution,
Assessment, & University Wide Implementation” AAC&U Peer Review Winter 2014, Vol. 16
Ambrose, G. Alex, Abbott, Kevin, Lanski, Alison (2017) “Under the Hood of a Next Generation Digital Learning Environment in
Progress” Educause Review.
Image Sources:
https://www.tolerance.org/magazine/spring-2013/the-schooltoprison-pipeline
https://www-03.ibm.com/ibm/history/exhibits/specialprod1/specialprod1_9.html
https://thumbs.dreamstime.com/x/find-needle-haystack-22905126.jpg
http://images.techtimes.com/data/images/full/25830/sven-sachsalber.jpg
http://media.gettyimages.com/illustrations/looking-for-a-needle-in-a-haystack-illustration-idiwdayala0001c?s=170667a
http://mitchellsfrontpage.com/wp-content/uploads/2017/02/800x400-elite-daily-robot-lawyer.jpg
38. How to Cite this Presentation:
Ambrose, G. Alex, Anggara, Trunojoyo (2018) “Liberate
Learning Through Next Generation Assessment”
Association of American Colleges & Universities, 9th
Annual Forum on Digital Learning and ePortfolios.
Washington, D.C.