This document contains brief descriptions and examples of direct and indirect assessment methods and techniques.

1. An Inventory of Conventional and Technology-Enabled Direct and Indirect Assessment
Methods
Direct Methods
Conventional or Technology-Enabled Standardized Instruments: Tests and Inventories
Description: Historically, standardized instruments, such as objective tests, have served
as the primary direct method to assess student learning. Content or disciplinary experts
identify the standard content, knowledge and tasks that students should know and be able
to perform. In addition, they determine what constitutes levels of achievement based on
the construction, design, and sequencing of questions or prompts. Students’ achievement
is referenced against other student groups’ achievement on the same instrument, referred
to as norm-referencing. Primarily designed to make decisions about students,
standardized instruments perform a gate keeping role. They certify competence in a
profession or field, such as in the case of licensure examinations for nursing students, or
program-level knowledge or skills, such as in the case of general education tests. Results
are also used to place student in appropriate courses or to identify level of achievement at
points in students’ studies, such as in the case of rising junior examinations.
What They Provide:
- Content and tasks developed by external experts within fields or
1

2. programs of study
-Psychometric approach to assessment that values quantitative methods of
interpreting student achievement
-Evidence of what students know or can do within the universe and framework of
questions, prompts, and tasks of an instrument
- Evidence to make gate-keeping decisions such as professional certification or
end-of-study achievement or to meet state mandates that value
measurement as proof of learning
-Evidence to track student learning if instrument can be used formatively and if
results have utility for programs and the institution
-Quick and easy adoption and efficient objective scoring
-History of validity and reliability studies
-One possible source of evidence within an institutional commitment to assessing
student learning through multiple lenses
What Standardized Instruments Do Not Provide:
2

3. -Evidence of the strategies, processes, and ways of knowing, understanding, and
behaving that students draw upon or apply to represent learning
-Evidence of the complex and diverse ways in which humans construct and
generate meaning
-Alignment with institution- and program-level learning outcome statements and
students’ learning histories
-Realistic timeframes or contexts that reflect how humans solve problems, seek
additional information or resources, correct mistakes, or reposition their
thinking. Students respond within a timeframe that might well
affect their decisions or actions, such as deciding to make a last-minute guess
among
options in a question.
-Highly useful results that directly relate to pedagogy and educational practices.
Results relate to the construct of the instrument itself and what it is designed to
measure. Patterns of student performance reported in scales or scores
identify discrete areas of performance, such as a skill level, that identify
strengths and weaknesses in curricular or co-curricular attention.
However, these patterns do not assist in learning about why students responded in
3

4. the ways they did. Or did they learn successful strategies for selecting or
making “good guesses?”
Some Examples:
Instruments that test general education knowledge and abilities include the following:
* Academic Profile: http://www.ets.org/hea/acpro
* Collegiate Assessment of Academic Proficiency (CAAP):
http://www.act.org/caap/index.html
* Collegiate Learning Assessment (CLA)
* MAAP
Instruments dedicated to measuring specific skills include:
* ACCUPLACER : (http://www.collegeboard.com/highered/apr/accu/accu.html
* Watson-Glazer Critical Thinking Appraisal
* California Critical Thinking Skills Test
* Tasks in Critical Thinking
* Reflective Judgment Inventory
* Measure of Intellectual Development
* e-Write, a component of ACT’s COMPASS/ESL system
Examples of achievement tests in a particular field of study or profession include:
* Graduate Record Examinations' Subject Tests:
4

5. http://www.gre.org/pbstest.html#testreg
* The PRAXIS Series: Professional Assessment for Beginning Teachers:
http://www.ets.org/praxis/index.html
* Area Concentration Achievement Tests: www.collegeoutcomes.com
* Graduate Management Admission Test: http://www.mba.com/mba
Examples of inventories that assess students’ knowledge include:
include:
* Force Concept Inventory (FCI)--designed to assess the initial knowledge state
of students before they begin undergraduate physics courses. Students
respond
to a series of force-related problems or statements that reveal their knowledge
(Halloun and Hestenes,1985).
Knowledge surveys
Locally Designed Tests and Inventories
In a collective and shared commitment to assessing student learning, core working groups
may well determine that no appropriate standardized instruments exists that aligns with
institution- and program-level outcomes. That decision generates the design of local tests
or inventories or use of existing instruments that have an institutional history of providing
useful and reliable results. Technological advancements, such as WebCT
5

6. (http://www.webct.com/transform) and Blackboard
(http://www.blackboard.com/products/ls/index.htm) offer an online environment for
constructing some locally developed instruments.
What Local Tests or Inventories Provide:
-Strong alignment of content and format with learning outcome statements and
course-based assessment methods students’ have experienced along their
learning histories
-Useful results that can be interpreted within the local contexts of teaching and
learning and then used to improve student learning
-Opportunity to establish local instrument criteria that reflect what an institution
and its programs value in educational practices.
-Opportunity for faculty, staff, administrators, students, teaching assistants,
tutors, intern advisors, advisory board members, and institutional
researchers, for example, to contribute their perspectives on what should be
assessed and how it should be assessed.
What Local Tests or Inventories Do Not Provide
-Immediate reliability and validity results that verify content, construct, format
and consistency in scoring, unless instruments have been pilot tested and
evaluated over several semesters. Thus, time to pilot test an
instrument for an institution’s representative student populations is a
necessary component of an institutional commitment to assessing for learning.
6

7. Conventional or Technology-enabled Authentic Performance-based Methods
Authentic performance-based methods prompt students to represent their learning in
response to assignments and projects that are embedded into their educational
experiences. These methods value divergent thinking and responding, as opposed to
convergent thinking, most typically represented in standardized tests. Focusing on how
students think, problem solve, react, interpret, or express themselves becomes the focus
of these kinds of direct methods. Further, these methods can easily be embedded into
students’ continuum of learning, providing evidence of their growth over time often
demonstrated in self-reflective writing and responses to feedback from those who
contribute to their education, such as peers, internship advisors, external reviewers or
evaluators, faculty and staff.
What They Do Provide:
-representation of integrated learning
-direct alignment with students’ learning experiences
-opportunities for students to reflect on and receive formative feedback about
their learning and development
-student-generated opportunities to demonstrate learning, as opposed to test-
generated occasions
_opportunities for students to create as in multi-media contexts
What They Do Not Provide:
-Easily quantifiable evidence given the complexity they capture
-Efficient scoring opportunities
7

8. Range of Authentic Assessments:
• Portfolio or e-portfolio—collection of multiple kinds of student-generated
texts stored electronically or in paper form. Developments in technology now
make it possible for students to create digital portfolios. This method of
storage and collection provides a longitudinal representation of learning,
demonstratinghow students make meaning within their contexts for learning
through assignments, projects, narrative self-analyses and self-reflection.
They provide an opportunity for students to demonstrate how they integrate
learning over time from multiple learning experiences and opportunities
within the curriculum,co-curriculum, and learning experiences that extend
beyond their formal educational experiences. They provide a valuable source
of evidence for institution- and program-level learning by providing a range of
texts that represent student learning. Webfolios, digital learning records, or
electronic learning records are other current terms for this method. The
development of 2.0 electronic portfolios offers students a context to
demonstrate their learning processes as well as learning products. Two major
national resources that focus on technological developments in electronic
portfolios are the following: 1. the newly founded global academic
organization, Association for Authentic, Experiential and Evidence-based
Learning (AAEEBL: http://www.aaeebl.org/) and 2. the community of
practice, Electronic Portfolio Action and Communication (EPAC:
http://epac.pbworks.com). Cambridge, Cambridge, and Yancey’s 2009
8

9. publication, 2.0 Electronic Portfolios: Emergent Reseearchon Implementation
and Impact (Stylus Publishing,LLC), provides a taxonomy of electronic
portfolios and describes the potential of 2.0 electronic portfolios For
information about Alverno College's Diagnostic Digital Portfolio, go to:
http://www.ddp.alverno.edu/.
• Learning Record Online
A running record of students’ learning is the Learning Record Online (LRO).
Originally developed for K-12 and now being developed for higher education,
the LRO provides formative and summative evidence of students’ learning
within their curricula and against agreed upon criteria and standards of
judgment. Like a portfolio, the LRO provides ongoing evidence of students’
emerging learning within contexts for learning, including evidence of their
reflective process, their metacognition. Facilitated by online technology,
educators and others interested in learning about students’ development are
able to aggregate or disaggregate groups to draw inferences and make decisions
about students’ progress and eventual levels of achievement: .
http://www.cwrl.utexas.edu/~syverson/olr/
* Capstone project —a culminating independent research, collaborative, or professional
project at the end of students’ careers that provide evidence of how students solve
representative higher order disciplinary, professional or interdisciplinary problems often
represented in more than one kind of text, that is in writing as well as in speaking or in
visual texts, such as poster presentations. These projects provide evidence of how well
9

10. students integrate, synthesize, and transfer learning; in addition they also can provide
evidence of how well students integrate institution-level outcomes. This method can also
be integrated as a formative means of assessment. The beginning of a second year of
graduate study or the beginning of a third year of students’ undergraduate study might
index a time for students to demonstrate accumulated learning. Senior theses or senior
research projects are also examples of capstone projects that provide opportunity for
observers to assess students’ masterly level within a field of study, discipline, or
profession.
* Performances, products, creations—required over time as well as at the end of
students’ studies, their work represents how they interpret, express, and construct
meaning. Traditionally, the arts’ faculty have formatively assessed students’
performances to provide them with immediate feedback that shapes their future
performances. This approach provides not only students with immediate feedback but
also faculty and others who contribute to students’ learning.
* Visual Representation--Representing learning visually through charting, graphing,
mapping, for example, provides students with alternative ways to represent their
learning, often a practice within disciplines such as mathematics and the sciences.
Mathematicians often represent developments in their thinking in mind maps:
http://jonathan.mueller.faculty.noctrl.edu/toolbox/examples/seaver/mindmappingtask.htm
10

11. Visual representation offers a way to assess how well students make connections or
understand a concept. In addition, visual representation extends students’ repertoire of
making meaning, developing a versatility in forms of representation that respond to the
different needs of audiences, contexts, and the purposes of communication. The Biology
Teaching Homepage provides examples of different kinds of conceptual maps, well as
procedures for incorporating them into students' learning.
http://www.fed.cuhk.edu.hk/~johnson/misconceptions/concept_map/cmapguid.html . The
Innovative Learning Group presents different forms of thinking maps.
file://A:ILG%20%20Thinking%20Maps%20Information.htm
* Case studies-- used over time, as well as at the end of students’ studies, case studies,
often used in business programs, provide opportunity to assess students’ problem-
solving abilities within a major program of study or along students’ continuum to
determine how well they are integrating the learning expressed in institutional learning
outcomes—knowledge, perspectives, abilities, values, attitudes. Parallel case studies
used over time provide evidence of students’ abilities to solve representative
disciplinary, professional, or more generalized problems. In addition, they provide
evidence of student’s writing. Adding to the dimensionality of traditional paper- or
document-based case studies are online case studies that incorporate multi-media into
the case study and hyperlinks for users to read or see related resources. Harvard’s
* Professional or disciplinary practices—engaging students in practices that prepare
them for the kinds of problems, activities, or situations individuals address not only in
11

12. their fields of study but also as contributors to society and local communities. The Harrell
Professional Development Center in the University of Florida’s College of Medicine has
created an environment that permits observation of medical students interacting with
patients. Audio visual equipment captures, records, and displays these interactions.
Replaying these interactions provides opportunities for faculty and students to assess
students’ knowledge, understanding, behaviors and dispositions. file://A:Harrell
%20Professional%20Development%20and%20Assessment%20Center.ht
*Team-based or Collaborative Projects--With rare exception, humans work with other
humans during most of their lives in a range of workplace, social, and community
environments. Team-based or collaborative projects are direct methods that enable
assessment of individuals’ knowledge, understanding, behaviors and attitudes, as well as
their ability to work with others to achieve a final product or solve a problem. Often
videotaped, groups of faculty, staff, and students themselves have access to immediate
results that inform students as well as educators. Alverno College’s institutional example
on page x illustrates assessment that focuses on individual students as well as their
collective achievement. Community service projects provide another source of evidence
of how well students apply or transfer or integrate learning to solve a problem as well as
an opportunity for students to self-reflect on how they contributed and what and how they
learned from the experience.
Wikis and podcasts represent two technologically-based assessment opportunities:
student teams create new work and contributions are date and time stamped to document
12

13. who contributed material to a project. For example, at the end of a program of study
teams of students might collaborate to create a culminating assignment on a wiki site,
designed to assess how well majors integrate knowledge, abilites, habits of mind, ways
of thinking and knowing.
Educational online gaming-
Internships or Practica—How students actually apply or transfer their cumulative
learning can be assessed in internship or practica. That is, authentic experiences become
a direct method of assessment, providing opportunity for students to demonstrate the
dimensions of their learning within the context of a real environment. A step beyond
simulations, these direct methods assesses how well students translate their cumulative
learning into actual practice.
Oral examinations—often used at the end of a professional program or a graduate
program, as in an oral doctoral defense (University of Arizona, Graduate College:
http://grad.admin.arizona.edu/degreecert/ppfoedc.htm), oral examinations provide
opportunities for students to represent how well they integrate learning and apply it to
solving a case study or problem, responding to guided questions, or presenting a product.
Measurement Research Associates provides some guidelines for conducting effective oral
examinations: http://www.measurementresearch.com/media/standardizedoral.pdf.
13

14. Simulations—are now possible through virtual reality programs such as Second Life
perhaps the most used virtual reality Virtual labs, virtual patients Data mining
merlot or Ariadne offer studnets that opportunity
 Conversion--converts one form of representation into another form of
representation, such as into numerical or statistical, scientific, humanistic,
sociological, psychological, or artistic forms of representation
Those that provide opportunity for students to construct their own interpretaions, or
identity new or unepected problems
 Data mining--collects, records, analyzes, and interprets data (Merlot)
 Problem identification/solution—identifies a new or unique problem or poses a
new or unique way to view or solve a problem (Here’s a problem and the
solution; are there other solutions?)
 “Conceptual collisions” (Federation of American Scientists)—brings “previously
disconnected knowledge sources into contact for the first time.”
 Projection: identifies consequences, results, resulting conditions, challenges,
actions, responses
SecondLife sites such as Virtual Ability Island, the Theorist Project, and Sci-Lands
Conventional and Technologically-enabled Indirect Methods
Helpful in deepening interpretations of student learning are indirect methods, methods
that focus on perceptions of student learning by asking students or others to respond to a
set or series of questions. Indirect methods function to complement direct methods rather
than to substitute for them.
14

15. What Indirect Methods Provide:
-Evidence of students' attitudes, perceptions, experiences
-Evidence that may help to explain student performance levels
What Indirect Methods Cannot Provide:
-Work that represents evidence of student learning unless an instrument asks
students to produce a text as evidence
Some Representative Examples:
-Focus groups with representative students to probe a specific issue
that may have been identified in a survey or identified in patterns of
student performance as a result of formative or summative assessments.
Van Aken and collaborators describe the successful design of a focus group
to obtain perceptions of historically under-represented students enrolled in
an engineering program (1999).
-Interviews with groups of students representing the institutional population.
Tracking a cohort of students is one way of assessing learning for
formative and summative purposes.
-Student, alumni, employer and faculty-staff surveys and questionnaires
that provide information about students’ or others’ perceptions of students’
educational experiences and the institutions’ impact on their learning.
15

16. Alumni questionnaires and surveys provide a retrospective view of
graduates’ educational experience and create an opportunity for
them to recommend improvements in education based on what is relevant to
their current employment, profession, or graduate education. Faculty-
staff surveys or questionnaires provide perceptions of student learning—
that is what students are able and not able to demonstrate based on classroom-
based assessments or observations of student behavior.
-ACT's surveys for adult learners, alumni, entering students, withdrawing
students, and for institutional services: www.act.org
-College Student Experiences Questionnaire (CSEQ):
Community College Survey of Student Engagement (CCSSE)
http://www.ccsse.org/aboutccsse/aboutccsse.html
-Community College Student Experiences Questionnaire (CCSEQ):
http://www.people.memphis.edu/~coe_cshe/CCSEQ_main.htm
- The National Center for Higher Education Management Systems'
Comprehensive Alumni Assessment Survey: www.nchems.org
-National Survey of Student Engagement (NSSE):
http://www.iub.edu/~nsse/
16

17. -Noel-Levitz Student Satisfaction Inventories:
http://www.noellevitz.com/library/research/satisfaction.asp#ssi
--Incoming Freshman Assessments (such as CIRP):
--First-year expericne surveys such as YFCY:
-Small Group Instructional Diagnosis (SGID)—a facilitated small group
interview process conducted by a trained interviewer who asks students
to identify what and how they have learned and what kinds of learning
obstacles they are facing or have faced. Used initially at mid-point in a
course to provide feedback to a faculty member about how students are
or are not learning, this method can also provide useful formative
assessment data at the program-, department-, or institution-levels ,
providing students’ first-hand explanations for dominant patterns of
weakness in their work:
http://www.miracosta.edu/home/gfloren/sgid.htm
--Student Assessment of Their Learning Gains (SALG)—a customizable
online end-of-course survey that asks students to rate the components of
a course, such as lab work or discussion groups, and the degree to which
these components contributed to their learning. This survey could well be
used for program-, department-, or institution-level assessment if a sample
of students were asked to rate how well and why the components of
17

18. learning they have been explosed to have contributed to their learning of
stated learning outcomes in their program of study . Given that the survey
is customizable, institutions can add additional kinds of questions that
probe students’ perceptions of their learning.
http://www.miracosta.edu/home/gfloren/sgid.htm
Inventories that track how well students have changed or shifted attitudes, values,
sensitivities, perspectices based on self-perception or responses to situations or case
studies.
* Some Representative Examples include:
--Global Perspective Inventory (GPI)—46-item on-line inventory designed
to assess the effect of cultural experiences on learners’ global
perspectives through three dimensions: cognitive, intrapersonal and
interpersonal (https://gpi.central.edu/)
--Intercultural Development Inventory (IDI)-- 44-item inventory based
paper and pencil instrument designed to assess the extent of an
individual’s intercultural sensitivity along a continuum that ranges
from ethnocentrism to ethnorelativism, identifying a person’s ability to
shift from denial of difference to integration of difference
(http://www.intercultural.org).
* A list of current inventories frequently used in student services is available at:
18

19. The Office of Institutional Research and Planning, The University of Texas at
Arlington. Student Affairs-Related Outcomes Instruments: Summary
Information.
Inventories that gauge students’ , such as the Intercultural Development Inventory
(IDI) and the GPI that guage the extent to which studnets have shifted or changed
attitudes, values or even perspectives representa yet another kind of indirect methodof
Assessment that may provide evidence of how much or to what degree students change
their views based on their learning either in the curriculum or co=curriculum. Often used
as pre- and post-tests in a program or service, these kinds of assessments provide
evidence of how much or to what degree students chage their views, attitudes or beliefs
Examples of Graphic Organizers
http://www.eed.state.ak.us/tls/Frameworks/mathsci/ms5_2as1.htm#graphicorganizers
19