1. Quality Standards and
Organizational Self-Assessment for
Distance Learning and Continuing
Education Organizations
Kim A. Scalzo, Director
SUNY Center for Professional Development
Alexandra M. Pickett, Associate Director
SUNY Learning Network
2. SLOAN-C Quality Scorecard
• Quality of online education is always in question.
• How do we determine online program quality?
Kaye Shelton, Ph.D. former Dean of Online
Education for Dallas Baptist University and currently
Associate Professor, Educational Leadership,
Lamar University.
3. SLOAN-C Quality Scorecard
Institute for Higher Education Policy National Education
Association report, Quality On the Line: Benchmarks for
Success in Internet-Based Distance Education (2000)
http://www.ihep.org/Publications/publications-detail.cfm?id=69
• Identified 24 benchmarks that are essential to ensure
quality in Internet-based distance education.
• These 24 IHEP standards were used as the starting
point for the study that resulted in the SLOAN-C award-
winning and endorsed Quality Scorecard.
4. Quality Scorecard Development
Process
• Delphi Research method was used.
• Panel of experts was identified by Sloan-C and consisted
of online education administrators in higher education.
7. Quality Scorecard Development
Process
Results of process - 9 areas of quality/70 indicators
• Quality Institutional Support
• Technology Support
• Course Development and Instructional Design
• Course Structure
• Teaching and Learning
• Faculty Support
• Student Support
• Evaluation and Assessment
8. Implementing the SLOAN-C Quality
Scorecard
• Online interactive scorecard: http://sloanconsortium.org/
node/add/scorecard-online-program
• Your institution must be a member of Sloan-C.
• Available to SLN education members.
http://wiki.sln.suny.edu/display/SLNED/Sloan-
C+Institutional+Membership+Benefits
• Paper version:
http://sloanconsortium.org/quality_scorecard_online_pro
gram
• Free available to anyone.
9. Quality Scorecard: Guidelines for
Scoring
9 areas of quality/70 indicators
• 0 points = Not Observed. The administrator does not observe any
indications of the quality standard in place.
• 1 point = Insufficiently Observed. The administrator has found a
slight existence of the quality standard in place. Much improvement
is still needed in this area.
• 2 points = Moderate Use. The administrator has found there to be
moderate use of the quality standard. Some improvement is still
needed in this area.
• 3 points = Meets Criteria Completely. The administrator has found
that the quality standard is being fully implemented and there is no
need for improvement in this area.
10. Quality Scorecard: Guidelines for
Scoring
9 areas of quality/70 indicators
• A perfect score = 210 points.
• 90-99% = 189-209 - Exemplary (little improvement is needed)
• 80-89% = 168-188 - Acceptable (some improvement is
recommended)
• 70-79% = 147-167 - Marginal (significant improvement is needed in
multiple areas)
• 60-69% = 126-146 - Inadequate (many areas of improvement are
needed throughout the program)
• 59% and below = 125 pts and below - Unacceptable
12. Category 1: Institutional Support
1 Quality Scorecard Indicators of Institutional Support:
4. Governance structure enables decision making.
5. Student authentication policies.
6. IP policy - Copyright ownership policy for course
materials.
7. Communication of strategic value of online education.
13. Implementing the SLOAN-C Quality
Scorecard
Example Justification Comments in the Category of
Institutional Support:
14. Category 2: Technology Support
2 Quality Indicators of Technology Support:
2. There is a technology plan with security measures in
place.
3. Technology delivery is reliable and there is a process
system task benchmarking for the servers.
4. There is a central support system for infrastructure.
5. Online course delivery is considered mission critical –
such as a commitment to 24X7 hour uptime.
6. A backups system is in place.
7. The development and the use of new technologies is
supported.
15. Category 3: Course Dev & ID
3 Quality Indicators of Course Development & Instructional
Design:
2. Minimum standards for design and development are in place.
3. Technology used as a tool to support learning outcomes – not
just there for bells and whistles.
4. There is a systematic process for the review of course
materials.
5. Courses designed to meet learning outcomes.
6. The course learning outcomes are measurable.
7. Appropriate assessments are designed for the course and the
audience. For example, it would be difficult for an 18 year old to
complete an assignment based on work experience the
corporate world but appropriate for an adult learner.
16. Category 3: Course Dev & ID - continued
3 Quality Indicators of Course Development & Instructional
Design:
1. Courses are student-centered.
2. Courses across the program have a consistent design/navigation.
3. Courses are designed for faculty-student engagement.
4. The program has a process to evaluate and recommend emerging
technologies.
5. There is instructional design support for effective online pedagogy
provided to the instructor.
6. Faculty have responsibility for curriculum.
17. Category 4: Course Structure
4 Quality Indicators of Course Structure:
1. Students have access to an online course syllabus that details all
aspects of the course & requirements prior to registration.
2. Online students have access to learning resources such as library,
writing center, labs, etc.
3. Grading expectations and assignment instructions are provided.
4. Access to technical support for students is clearly provided in the
course.
5. Course materials are easy to access and to use.
6. The course adequately addresses special needs students.
7. Student to student collaboration is encouraged, supported, and
technically facilitated in course activities, assignments, etc.
8. Consistent document formats used to present course content/share
content.
18. Category 5: Teaching & Learning
5 Quality Indicators of Teaching and Learning:
1. Student-student and faculty-student interaction is
facilitated.
2. Faculty provide effective, engaging and timely feedback.
3. Effective research methods are taught.
4. Library support is provided.
5. Instructor presence is demonstrated in specific, deliberate,
and effective ways.
19. Category 6: Social & Student Engagement
6 Quality Indicators of Social and Student Engagement:
1. Students are provided a common space for student
interaction outside the course.
20. Category 7: Faculty Support
7 Quality Indicators of Faculty Support:
1. Technical assistance is provided for both online course
development and delivery.
2. Faculty development and support is provided.
3. Faculty understand Fair Use, ethical issues, and guidelines
for plagiarism.
4. Ongoing professional development is provided to
continuously improve online teaching and learning.
5. Standards are established for faculty regarding online
teaching expectations.
6. Workshops are provided for faculty in the use of emerging
technologies.
21. Category 8: Student Support
84.
Quality Indicators of Student Support:
Students can determine if they will be successful online.
5. Minimum technology and skills standards are provided.
6. Clear program information is provided.
7. Student receive training and information on how to access
and use library materials and resources.
8. Technical assistance is provided.
9. There is a process in place to collect and address student
questions, problems, bugs, feedback, etc.
10. Students are provided academic, personal, & career
counseling.
22. Category 8: Student Support - continued
8 Quality Indicators of Student Support:
4. Minimum requirements for tech skills and equipment are
established and made available to students.
5. Support services such as advising, financial aid help, payment
services, peer support are provided.
6. Services are provided for students with disabilities.
7. Course material information including (ISBN) numbers are
provided at the time of registration.
8. Support services are tailored and provided specifically for the
online student.
9. engagement w/ prog./institution is facilitated at a distance.
23. Category 8: Student Support - continued
8 Quality Indicators of Student Support:
1. Students are instructed in appropriate netiquette.
2. Technology changes are supported.
3. Tutoring is available.
4. Students are instructed on how to get help when needed.
24. Category 9: Student Support
9 Quality Indicators of Evaluation and Assessment:
5. There are specific standards for online program evaluation.
6. A variety of data is captured for continuous programmatic
improvement.
7. There is a clear regular and systematic process for the
evaluation of learning outcomes at both course and program
levels.
8. Faculty and student support services are systematically
assessed.
9. Course and program retention is assessed.
25. Category 9: Student Support - continued
9 Quality Indicators of Evaluation and Assessment:
5. Recruitment and retention are reviewed.
6. The program demonstrates ADA compliance.
7. Course evaluations are taken into account in faculty
performance evaluations.
8. Faculty performance is regularly assessed.
9. Learning outcomes align from course to course.
10. Student feedback is collected and reviewed.
26. Questions about the SLOAN-C
Quality Scorecard
1. How is this different from Quality Matters?
QM is a course design rubric that focuses only on the
instructional design of an online course, the scorecard examines
the entire online program.
2. How does it fit in with the Sloan-C 5 pillars?
The quality standards for the scorecard clearly fall into those
same 5 pillars in the Sloan-C Quality Framework of learning
effectiveness, access, faculty and student satisfaction, and scale.
27. Questions about the SLOAN-C
Quality Scorecard
3. Can I use the scorecard freely without constraints?
The scorecard is free to use.
To use the interactive scorecard on the website, you must be a Sloan-C
institutional member.
SLN education members have an affiliated membership that gives them
access to Sloan-C resources as a benefit of membership in SLN.
4. What if I already have a list of standards that I use?
The scorecard should mesh with other standards easily.
The Sloan-C pillars of quality aligned easily.
28. Questions about the SLOAN-C
Quality Scorecard
5. Can I give the scorecard to others?
Yes, share the link http://tinyurl.com/qualitysc
6. Will my information in the interactive scorecard public?
Private, the information you provide will not be shared.
7. Why is the scorecard input-based and not output?
The scorecard offers broad goals for outputs, but focuses on things
administrators can influence.
There are rubrics for online courses such as QM, but not a rubric or scorecard for an online education program that clearly defines standards. She saw a need to assist administrators identify elements of quality within an online program and measure: standards, process, guidance, tools. need tools and guidance for assessing their programs. So she set about developing a quality scorecard to evaluate the adminisstration of online education programs.
Institute for Higher Education Policy published a report commissioned by the National Education Association in 2000 According to IHEP these benchmarks may assist policymakers, faculty, students, and others in making reasonable and informed judgments with regard to the quality of Internet-based distance education.
To determine what quality standards are relevant to online education programs today and would be necessary for creating a quality scorecard. It is a structured flow of information using a series of systematic surveys and reciprocal feedback, used to gain consensus from a panel of experts on the selected topic with the results being fed back to the panel of experts and iterations occur until there is consensus. The study took 18 weeks with 6 survey rounds to gain consensus on what elements indicate quality Panelists were identified by Sloan-C – I was one. Broad representation from a variety of institutions with at least 5 years of experience. 76 experts were identified 43 completed the first round More than 83% of the panel members had 9 or more years of experience in the administration of online education programs 69% had 10 or more years of experience Level of experience of the panel strengthened the validity of the study.
This process resulted an instrument or rubric with 9 different areas of quality for a total of 70 indicators.
The scorecard is for measuring and quantifying elements of quality within online programs in higher education. It is an easy-to-use tool for online administrators for program evaluation. Hand out the paper version of the scorecard: By evaluating each of the respective quality indicators within the established categories, an online administrator can determine strengths and weaknesses of their program. The identification of the weaknesses can be used to support program improvement and strategic planning. The scorecard may also be used to demonstrate elements of quality within the program as well as an overall level of quality to accrediting bodies. Sloan C has developed an interactive version to use. Your institution must be a member of Sloan-C to access this version of the scorecard. The site allows you to select your score for each quality indicator and provide justification notes and upload artifacts for support such as org charts, faculty training materials, and survey results. Anything you have to substantiate your score. However, the paper version is just fine because more importantly, it is about the justifications and artifacts you can produce to justify your score. The justification supports your self-assessed score. For example, to support a score of 3 - meeting the criteria completely on Institutional Support one could use a flow chart for decision making for #1 and one ’ s org chart for #4 Go to the interactive scorecard to show: You type the title of the program and then choose the name of your institution from the drop down menu. You see the 9 categories to the left, when you click on each one, it brings up the corresponding quality indicators for that category. The first is Institutional Support.
The panel agreed on guidelines for scoring.
Here is an example of scorecard for one category showing the three points available for each indicator.
The institutional support category addresses 4 areas. Review recommendations on Pgs 8-12
The technology support category addresses 6 areas. Review recommendations on Pgs 13-20
The course development and Instructional design Category addresses 12 areas Review recommendations on (pp. 21-35)
The course development and Instructional design Category addresses 12 areas Review recommendations on (pp. 21-35)
The course structure Category addresses 8 areas Review recommendations on (pp. 36-44))
The teaching and learning category addresses 5 areas Review recommendations on (pp. 45-52;)
The social and student engagement Category addresses 1 area Review recommendations on (pp. 53-64)
The Faculty Support Category addresses 6 areas Review recommendations on (pp. 55-62)
The student Support Category addresses 17 areas Review recommendations on (pp.63-81)
The student Support Category addresses 17 areas Review recommendations on (pp.63-81)
The student Support Category addresses 17 areas Review recommendations on (pp.63-81)
The student Support Category addresses 11 areas Review recommendations on (pp.82-95)
The student Support Category addresses 11 areas Review recommendations on (pp.82-95)
The interactive tool lets you view your score card by the pillars.