American Public University System (APUS) has developed internal processes to ensure the academic quality and integrity of online courses and programs. This session will focus on the process for conducting a system wide interdepartmental program review of academic programs. Quantitative and qualitative metrics will be discussed.

1. An Interdisciplinary Approach to Institutional Level Evaluation of
Online Learning Environments
15th Annual Sloan-C International Conference on Online Learning | Orlando, FL |
October 2009

2. • Introductions
• APUS Mission and Demographics
g p
• Assessment Website
• Goals of the Program Review Process
• Program Review Process, Tools,
and Resources
• Use of Continuous Improvement
in Business Program
• Use of Data to Drive Decisions
• Conclusion and Dialogue

3. Frank McCluskey, Ph.D.
Executive Vice President and Provost
Phil Ice, Ph.D.
Director, Course Design, Research &
Development
Chad Patrizi, ABD
Dean of Business Administration
Jennifer Stephens, Ph.D.
Associate Vice-President, Dean of Assessment
Dave Becher
Director, Academic Information Analysis

5. To educate the nation's military
and public service communities
by providing respected, relevant,
affordable, and student-focused
online programs which prepare
them for service and leadership
in a diverse, global society.

6. • Founded in 1991 as American Military
University to provide affordable and
convenient distance learning to service
g
members
• In 2002, expanded reach to national security,
public safety and other professionals seeking
affordable education and online convenience
convenience,
establishing American Public University
System with member institutions:
American Military University and
American Public University
• We are a 100% online for-profit school that
serves 53,000+ students who study online
from 100 countries.

7. Data External
Collection Review
Follow‐Up Analysis
Program
Three Year
Review
Plan
Meeting

8. Electronic Repository of Information
Library and Curriculum External
Learning Resources Assessment Reviewer Feedback
• Course books • Student learning • Expert reviewer report
• Electronic resources outcomes • Industry Advisory
• Instructional strategies Council report
• Learning strategies
• Evaluation procedures
• Academic rigor

9. Electronic Repository of Information
Faculty Students Learning Outcomes
Assessment
• A l i of faculty
Analysis f f lt • Student demographic •C i l M
Curricular Mapping
i
credentials and information
• Assessment measures
expertise to ensure • Enrollment History
breadth and diversity • Fact book
• Growth trends

10. Electronic Repository of Information
Program Program Directory Program Review
Benchmarking Summary Findings
• Program benchmarking • Evaluation of findings • Dean’s observations
with similar programs • Program • Meeting minutes
and institutions recommendations
• Three year proposed
strategic plan

11. Dave – insert slides
D insert slides
a v e

16. • Introduction
• Bachelor of Business Administration
Program (BBA) vs. Institutional EOC
(end of course survey)
• Without Micro and Macro Economics
• Fact Book

17. BBA Program vs Institutional EOC Data 2006
4.60
4.40
4.20
4.00
3.80
3.60
3 60
BBA Program Average (n=1139) 4.42 4.42 4.50 4.31 4.27 4.45 4.35 4.29 4.40 4.01 4.53 4.49 3.87
Institutional Average (n=28344)
3.40 4.44 4.45 4.46 4.37 4.35 4.44 4.38 4.27 4.44 4.02 4.39 4.48 3.83
Clear course objectives assignments supported course
Course Course materials relevant to
Course assignments fairly evaluated
Instructor's teaching style appropriate
Knowledgeable instructor
Instructor provided feedback to provided grades and between the instructor and
Instructor Interactions Interaction among classmates created
Electronic classroom easy to Assignments easy to post support questions answered
navigate Technical
objectives objectives work inquiries within 72 hours
appropriate feedback within one week
the students were positive a sense of community within 2 days

18. BBA Program vs Institutional EOC Data 2006 w/o Micro and Macro Economics
4.50
4.00
3.50
3.00
2.50
2.00
1.50
1.00
0.50
0.00 Co urs e Co urs e materials Co urs e Ins tructo r's Ins tructo r Instructo r Interactio ns Interactio n Electro nic Technical
Clear co urs e Kno wled g eab le Ass ig nments
as s ig nments relevant to as s ig nments teaching s tyle p ro vid ed p ro vid ed g rad es b etween the amo ng clas s ro o m eas y s up p o rt
o b jectives ins tructo r easy to p o s t
s up p o rted o b jectives fairly evaluated ap p ro p riate feed b ack to and ap p ro p riate ins tructo r and clas s mates to navig ate q ues tio ns
BBA Pro g ram Averag e (n=113 9 ) 0 .0 0 0 .0 0 0 .0 0 4 .3 9 4 .3 8 0 .0 0 0 .0 0 0 .0 0 0 .0 0 0 .0 0 0 .0 0 0 .0 0 0 .0 0
Ins titutio nal Averag e (n=2 8 3 4 4 ) 4 .4 4 4 .4 5 4 .4 6 4 .3 7 4 .3 5 4 .4 4 4 .3 8 4 .2 7 4 .4 4 4 .0 2 4 .3 9 4 .4 8 3 .8 3

20. Community of
C i f
Inquiry Framework
a process model of learning in online and
blended educational environments
grounded in a collaborative constructivist
view of higher education
assumes effective online learning requires
the development of a community of
learners that supports meaningful inquiry
and deep learning

21. social presence cognitive presence
LEARNING
teaching presence

22. Social Presence
the ability of participants in a community
of inquiry to project themselves socially
and emotionally -- as ‘real’ people
the degree to which p
g participants in
p
computer mediated communication feel
socially and emotionally connected

23. Cognitive Presence
the extent to which learners are able
to construct and confirm meaning
through sustained reflection and
discourse in a critical community of
inquiry
i i

24. Teaching Presence
the design, facilitation and
direction of cognitive and social
processes for the purpose of
realizing personally meaningful and
educationally worthwhile l
d ti ll th hil learning
i
outcomes

25. Community of Inquiry
Surveyy
9 social presence items (3 affective expression,
3 open communication, 3 g p cohesion)
p , group )
12 cognitive presence items (3 triggering, 3
exploration, integration,
exploration 3 integration 3 resolution)
13 teaching presence items (4 design &
facilitation,
facilitation 6 facilitation of discourse 3 direct
discourse,
instruction)

26. CoI Survey Validation
tested in graduate courses at four
institutions in the US and Canada
principal component factor analysis
three factor model predicted by CoI
p y
framework confirmed
Arbaugh, Cleveland-Innes, Diaz, Garrison,
Cleveland Innes,
Ice, Richardson, Shea & Swan - 2008

27. APUS Utilization
American Public University System
100% online
monthly course starts
CoI is the end of course survey
eight semesters of data collection

28. Application
comparison of at the University, School,
Program, Course and Instructor levels
assessment of instructional design efficacy
assessment of l
t f learner diff
differences –
demographic factors
assessment of integration of new
technologies

29. Frank McCluskey, Ph.D.
fmccluskey@apus.edu
Executive Vice President and Provost
Phil Ice, Ph.D.
pice@apus.edu
Director, Course Design, Research & Development
Chad Patrizi, ABD
cpatrizi@apus.edu
Dean of Business Administration
Jennifer Stephens, Ph.D.
jstephens@apus.edu
Associate Vice-President, Dean of Assessment
Dave Becher
dbecher@apus.edu
Director, Academic Information Analysis