Student Affairs Assessment Committee Training

Assessment Fellows
Training
Brian Clark, Stan Dura
Assessment Fellows Meeting
4/17/24
Brian Clark, Stan Dura 11/15/13

Training Agenda
• Research, Assessment, & Program Evaluation
• Foundational Concepts
– Measurement is imprecise
– “Things and the stuff about them”
– Variables, variance, and modeling
• Q & A

11.85
11.9
11.95
12
12.05
Ruler 1 Ruler 2 Ruler 3 Ruler 4 Ruler 5 Ruler 6 Ruler 7 Ruler 8 Ruler 9 Ruler 10 Laser
Ruler
Ruler 1
Ruler 2
Ruler 3
Ruler 4
Ruler 5
Ruler 6
Ruler 7
Ruler 8
Ruler 9
Ruler 10
Laser Ruler
Measurement is imprecise

Measurement is imprecise
11.85
11.9
11.95
12
12.05
Ruler 1 Ruler 2 Ruler 3 Ruler 4 Ruler 5 Ruler 6 Ruler 7 Ruler 8 Ruler 9 Ruler 10 Laser
Ruler
Ruler 1
Ruler 2
Ruler 3
Ruler 4
Ruler 5
Ruler 6
Ruler 7
Ruler 8
Ruler 9
Ruler 10
Laser Ruler

Surveys are especially vulnerable
What
“construct” are
you measuring?
Survey and
question design
and content
Responses

What
“construct” are
you measuring?
Survey and
question design
and content
Responses
Is it really measuring it?
Does it measure it the same
each time?

What
“construct” are
you measuring?
Survey and
question design
and content
Responses
each time?
Any errors in the questions or
scales?
Any bias in the questions
asked or not asked?

What
“construct” are
you measuring?
Survey and
question design
and content
Responses
each time?
scales?
asked or not asked?
Response set–
-Are they representative?
-How complete are they?
-Is there non-response bias?

What
“construct” are
you measuring?
Survey and
question design
and content
Responses
each time?
scales?
asked or not asked?
Response set–
-Are they representative?
-How complete are they?
-Is there non-response bias?
Over 80
cognitive and
memory
biases

Respondents prone to error
• Response is a cognitive act. 4 Stages:
• Comprehension
• Retrieval
• Judgment
• Response
All kinds of errors can happen in these stages

Cognitive Stages Definition Vulnerabilities
Stage 1 Comprehension
Understanding the question
as author intended
Unknown or misused words,
ambiguity, complexity, length
Stage 2 Retrieval
Search of memory for
relevant information
Memory bias, recall error,
fatigue
Stage 3 Judgment
Considers information
retrieved, makes
“guestimations” and decides
Social or political bias, “fuzzy
logic,” error in “guesstimation”,
personal sensitivity
Stage 4 Response Provides the information
Human error, incomplete
response, wrong format

• Change bias – Remembers as more difficult
• Context effect – out of context more difficult
• Consistency bias – thinking past attitudes same
as current
• Anchoring – relying too heavily on one piece of
info in order to make an estimation
• Availability heuristic – overestimation due to
recency or emotional strength of memory

• Confirmation bias – tendency to search for and
remember info confirming one’s preconceptions
• Framing effect – drawing different conclusions
or interpretations base on how info is presented
• Optimism bias – tendency to overestimate
likelihood of positive outcomes
• Social response bias – tendency to underreport
socially undesirable behaviors and vice/versa.
Lots more!!!!

Proof is in the Polls
• Ever seen this?
• Down in the polls by 6 points
• Down in the polls by 3 points
• Ahead in the polls by 1 point
• Landslide victory with 60% of the votes!
Keep these limitations in mind when drawing
inferences from survey data.

Research, Assessment, & Program
Evaluation
Research
• Systematic collection of data
• Generalize to larger populations
• Based on Research question
• Rigorous methodology
• Used to test hypotheses
• Contributes to the development
of theory and models
Assessment
• Systematic collection of data
• Generalize to larger population
• Based on Assessment question
• Borrows rigor when practical
• Used to evaluate effectiveness
• Contributes to judgments of the
quality of programs or activities

Program Evaluation:
• Act of using data (assessment, operational,
etc.) related to outcomes, inputs, and
processes of a program
• To assign judgment on its effectiveness
Research, Assessment, & Program
Evaluation

Assessment (and statistics) is about describing:
• Things
• Stuff pertaining to those things
• The relationship between things & stuff
Foundational Concepts
“Things and Stuff”

What is a “thing”?
• An entity of some kind
• An idea
• A quality or characteristic

What is “stuff”?
• Material out of which something is made
• Essential substance or elements
• Essence

Things are the objects we assess,
and
Stuff characterizes those things

Let’s use more concrete terms:
• Entity – something that exists separate from its
parts
• Property – a characteristic, trait or attribute and
other stuff that describes something

Types of entities:
• Organisms (including humans, trees, etc.)
• Physical objects (rocks, buildings, etc.)
• Actions and events (running, stroke, etc.)
• Cognitive phenomena (ideas, emotions, etc.)
• Organizations (governments, boards, etc.)
• Scientific entities (waves, motivation, etc.)
• Math entities (numbers, functions, vectors, etc.)

Different Properties:
– Height
– Age
– Perceptions
– Weight
– Density
– GPA
– Level of engagement
– Magnitude
– Levels and expressions of anger, fear, anxiety

Notice that some properties could be examined
as entities themselves
– Relaxation
– Motivation
– Intelligence
– Satisfaction
– Health

What entities are we concerned with?

What properties of those entities concern us?

Entities and properties
– Humans
• Height, satisfaction, health, perceptions, emotions, beliefs,
etc.
– Students (whether human or not  )
• Age, GPA, knowledge, skills, engagement, class standing, etc.
– Physical Objects
• Weight, chemical composition, density, etc.
– Forces
• Magnitude, direction, etc.

Most critical thing about properties -
they vary
– Height varies
– Age varies
– Perceptions vary
– Weight varies
– Density varies
– GPA varies
– Level of engagement varies
– Levels and expressions of anger, fear, anxiety vary

These are the first two foundational concepts:
• Entity
• Property
The others are:
• Variable
• Prediction and Control
• Relationship
• Statistical techniques

Properties are represented by
variables -
– Height is represented by inches, meters, or relational terms
(bigger)
– Age is represented by months, days, years or relational terms
(older)
– Weight is represented by kilograms, pounds or relational terms
– Density is represented by pound per square inch, etc.
– Grades can be represented by symbols (A+), numbers (92), etc.
– Engagement can be represented by number of activities
attended, etc.

Some variables are straightforward-
–Height is often measured in Inches
–Weight is often measured in Kilograms
–Age is often measured in years

Some variables are not-
–Satisfaction is often measured in… ???
–Anger is often measured in… ???
–Engagement is often measured in… ???

Thus variables are just
representations
of the properties

That describe an entity
to some degree

That describe an entity to
some degree
That may be precise or not

Broad goal of assessment/research:
To understand the population
Two underlying fundamental goals:
– Prediction and control of variables
– Understanding of entities & their properties
Which is more important?
Prediction and Control

Let’s pretend we are looking at student
performance of a complex task:
Let’s look at these statements:
– Haste makes waste
– Practice makes perfect
How would we assess these?

What are our variables?
– What variables might represent haste?
– What variables might represent waste?
– What variables might represent practice?
– What variables might represent perfection?
What is our model?

What is our model?
–More haste = more waste?
• Some of the waste is the result of haste?
• Haste predicts waste?
– Simply looking at means, which is what we often do:
• Those that were hastier answered fewer questions
correctly than those who took more time.

Waste
Haste

Waste
Haste
• The red area represents the degree to which Waste is due to Haste.
• The green area is the degree to which Waste is influenced by other factors

What is our model?
–More practice = better performance?
• Some of the performance is the result of practice?
• Practice predicts performance?
– Simply looking at means, which is what we often do:
• Those who practiced more scored higher than those who
practiced less.

Performan
ce
• What is the red area?
• What is the green area?
Practice

Is it really that simple?
–What if hasty practice results in worse
performance?
–How do we know the true impact of practice,
if we don’t remove the impact of haste?

So in order to accurately understand
entities and properties, we must be
able to predict and control.
–Prediction and control are more important –
accurate understanding depends on it
–We must understand relationships to do so

Back to those statements
– Haste makes waste
– Practice makes perfect
Both are engrained in our culture and both are
about relationships
Establishing relationships between objects is how
we naturally understand the world

Let’s operationalize this?
–Haste = time on task (in minutes)
–Practice = # of repetitions
–Performance = Evaluation score
• (waste = lost points or distance from a perfect score)

Our Model
• Performance = Practice + Haste
Or more specifically…
• Evaluation Score = # Repetitions + Minutes on Task

Each variable contributes to the
variance in evaluation scores
scores
#
repetitions
minutes
on task

scores extent to which
observed scores vary
across all individuals
We need to control for the impact of Haste on Performance
in order to understand the impact of Practice.

scores
#
repetitions
extent to which # of
repetitions varies
across all individuals

extent to
which minutes
on task varies
across all
individuals
scores
#
repetitions
minutes
on task

scores
#
repetitions
minutes
on task
variance in scores that we
can’t explain (individual
differences, fatigue, human
error, etc.)

scores
#
repetitions
minutes
on task
Variance attributed to
the # of repetitions

scores
#
repetitions
minutes
on task
This is the degree to
which “Practice
Makes Perfect”

scores
#
repetitions
minutes
on task
This is the variance
explained by
minutes on task

scores
#
repetitions
minutes
on task
In other words, the
degree to which
“Haste makes
waste”

scores
#
repetitions
minutes
on task
SHARED variance
between minutes on task
and # repetitions

scores
#
repetitions
minutes
on task
variance in Scores
attributed to the
combination of
minutes on task AND #
repetitions

scores
#
repetitions
minutes
on task
variance in scores
uniquely attributed
to # repetitions

scores
#
repetitions
minutes
on task
Put another way:
variance in scores
attributed to #
repetitions AFTER
variance
attributed to
minutes on task is
controlled

Our Model
–Performance = Practice + Haste

Performance
Practice
Haste

If we did not Control
for Haste, we would
think all of this was
the result of Practice
Performance
Practice

When really, this was
the impact.
Performance
Practice

So…
• Recognizing the relationships between
variables, and
• controlling for them
• is critical to understanding the impact
of any one of them.

Why is this important in assessment?
• Consider you are studying the impact of stress on student
persistence?
• You ask students to describe their stress level at one point
in time, and you see that:
– 80% of those who reported high stress dropped out, while only
– 25% of those who reported low stress did.
• You devote 70% of a staff person’s time to doing stress
relief programming, totaling appx. $60,000

What variables are there in stress and
persistence?
Stress
• Family support and relationships
• Peer support and relationships
• Coping skills
• Academic work load
• Financial Support
• Employment work load
• Work environment
Persistence
• Academic advising
• Social connectedness
• Engagement
• Faculty / Staff connections
• Institutional Bureaucracy
• Meaningful learning experiences
• Support Services

Stress
• Family support and relationships
• Peer support and relationships
• Coping skills
• Academic work load
• Financial Support
• Employment work load
• Work environment
Persistence
• Academic advising
• Social connectedness
• Engagement
• Faculty / Staff connections
• Institutional Bureaucracy
• Meaningful learning experiences
• Support Services
• What should the staff member focus their efforts on?
• Is it possible that general Stress Relief may not be a
very influential variable?
• Would the $60k be well spent?

When we conduct quantitative or qualitative
analyses, we are constructing models
• That model can be very, very simple
• Mean – this is how the average person did…
• Mode – The most frequent score was X
• Standard Deviation – roughly 68% scored between these two
points
• Range – All students scored between X and Y.
• Descriptive stats = the weakest and least informative models

When we conduct quantitative or qualitative
analyses, we are constructing models
• Other models can be complex
– For every 1 repetition of Practice, when Haste is controlled, Evaluation
Scores increased by 2.4 points.
– For every hour engaged in co-curricular activities, when controlling for HS
GPA, Class standing, and Major, retention likelihood increases by .45%
– More complex stats = more informative and more
accurate models

So when we assess, we need to
Think of our Model
• What are the entities and properties we’re examining?
• What are the variables involved? Operationalize them.
• What is their relationship; how do they interact?
• How simple or complex/accurate a model do I need?
• How can we predict and control for these variables?

Questions?
Concerns?
Snide Comments?

Homework
Office-work
• Use your notes and slidedeck to
create a study guide
• Email to me by next Friday

Student Affairs Assessment Committee Training

Recommandé

Recommandé

Contenu connexe

Similaire à Student Affairs Assessment Committee Training

Similaire à Student Affairs Assessment Committee Training (20)

Dernier

Dernier (20)

Student Affairs Assessment Committee Training

Notes de l'éditeur