1. Evaluation of Online Instruction
ET 755
Zeta: Shelby Simmons, Sarah Tolson, Lindsay Tucker
Evaluation 1
Site:
MIT Open Courseware
Course Title:
Combinatorics: The Fine Art of Counting
Evaluators: Shelby Simmons,
Sarah Tolson, & Lindsay Tucker
Course URL: http://ocw.mit.edu/high-
school/mathematics/combinatorics-the-
fine-art-of-counting/index.htm#
Introduction:
The first course we evaluated was entitled “Combinatorics: The Fine Art of Counting”
and was presented by Andrew Sutherland. Combinatorics is defined as “a fascinating
branch of mathematics that applies to problems ranging from card games to quantum
physics to the Internet.” This course was designed for the High School Studies
Program (HSSP) from MIT, which offers “non-credit, enrichment courses to 7th–12th
grade students on Sundays at MIT.” The course is designed to be engaging and
appeal to students’ interests, as well as preparing students for math contests and
competitions.
Sutherland, Andrew. Combinatorics: The Fine Art of Counting, Summer 2007. (MIT OpenCourseWare:
Massachusetts Institute of Technology), http://ocw.mit.edu/high-school/mathematics/combinatorics-the-fine-art-of-
counting (Accessed 10 Jun, 2014). License: Creative Commons BY-NC-SA
1. Course Overview and Introduction
Strengths:
The intentions of the course are made
clear through the syllabus and
introduction from the instructor. The
syllabus and instructor made the
structure of the course clear, outlining
each week’s lecture, providing notes,
and a relevant homework assignment.
The syllabus clearly states that students
must have prerequisite knowledge in the
area of algebra, and encourages a love
for mathematics. The instructor provides
a self-introduction that was appropriate
and available online, which emphasized
his love for Combinatorics and why/how
he teaches the course.
Weaknesses:
There are no explicit instructions on how
to follow the course online, since
lectures were offered in person.
However, things are clearly labeled and
accessible (including lecture notes). Not
much of this course seemed to focus on
the online instruction; rather it seemed to
use the internet as a method of
communication and sending documents
between the instructor and students.
There were no evident “netiquette”
expectations listed, nor were there clear
course and institutional policies. There
is no evidence that students were
introducing themselves to each other
online, but it may have happened during
the introductory session from Week 1.
The information that is accessible online
does not discuss minimum technical

2. skills needed by students, nor does it
discuss many syllabus elements: course
materials (implies that course notes may
be all), assessments (introduction video
says there are problems, syllabus states
no grading, but he still needs to assess
how well his students are grasping
material—that information is not clear),
nor does it provide clear information
about the feedback provided by the
instructor (are homework assignments
due to him? Will he provide feedback
individually? How? When?)
2. Learning Objectives
Strengths:
The instructor does cover what the
course is about briefly in his introduction
video, which could be seen as objectives
for the course. The syllabus also states
that the course will prepare students for
mathematical courses.
Weaknesses:
Learning objectives for the course are
not clearly stated, so we cannot
determine whether they would be
measurable or consistent with course-
level objectives. Even though this
course is catered toward younger
students and does not supply grades, it
would benefit from having clear,
measurable objectives for the course
and each meeting, rather than simply a
list of topics that will be covered. Since
there are no objectives stated, it is
impossible for there to be clear
instructions on how to meet the
requirements and expectations for the
course.
3. Assessment and Measurement
Strengths:
The homework assignments are the only
assessment that are accessible online.
The assignments seem to align with
what is covered in each lecture/class
and are consistent with the course
activities. The homework is offered as a
menu, allowing students to choose
which “selections” they would like (what
problems they would like to answer.)
These homework assignments get more
in-depth and reinforce concepts that
have been covered in the lecture/lecture
notes, as well as encouraging students
Weaknesses:
Since there are not clear learning
objectives, we cannot determine whether
the assignments align with and measure
the objectives. There is no grading
policy for this course, but a rubric, or
another tool would probably be
beneficial as feedback for students.
There is no criterion for assessing the
work of or participation of students, and
there is no evidence of opportunity for
self-reflection.

3. to use high order thinking skills. Many of
the problems offered in these
assignments are relevant to students,
such as determining odds for games
(dice, cards, etc.), game shows (Monty
Hall – The Price is Right), probability, or
simply challenging questions.
4. Instructional Materials
Strengths:
The instructional materials provided
online for this course are the lecture
notes, which are extensive. They cover
the topics as listed on the schedule. All
of the instructional materials are optional
as the course is not graded; however, it
is clear that many of them need to be
completed in order to understand the
following week’s lecture and additional
material to be covered. Instruction
materials available online (everything but
the live lecture itself) can be easily
accessed for students by opening and
printing (if desired) a PDF document.
Throughout many of the lectures
Sutherland refers to different
perspectives and tries to explain some of
the material in laymen’s terms.
Weaknesses:
Without clear learning objectives it is
impossible to determine if the materials
contribute to the achievement of the
objectives. Citations are not provided on
the lecture notes or the homework
problems. The problems could have
been created by Sutherland himself, but
more than likely he at minimum referred
to additional texts that were not
referenced to create his lectures and/or
problem sets. It is explained that as
students work on problem sets
(activities) they should refer to their
notes (instructional material), which is
clearly explained; however, there is no
alternative way to understanding the
content via a different avenue if a
student didn’t quite grasp the concept.
5. Learner Interaction and Engagement
Strengths:
One way that the instructor engaged the
students in the course content
(homework problems) was writing them
as a menu. Students were able for
instance for one assignment to choose
an appetizer, salad, entrée, and dessert
option. For another assignment that had
to choose a couple of different items
from the tapas “menu”. It is a creative
way to engage students! There is also a
competition on the final day of class
where students take part in answering
actual contest questions. If nothing else,
most students love competition!
Weaknesses:
The learning activities (homework sets)
provided to students cannot promote
achievement of unstated learning
objectives. They are still linked to what
was covered in class as well as the topic
(not objective) stated on the schedule. In
one of the introductory videos
Sutherland discusses in class activities
where students work in groups, but it is
never referenced in the material
provided online. It is not indicated for the
course whether individual feedback was
provided to students, the solutions to all
of the homework sets and one class
problem set were provided. No student-
to-student or faculty-to-student
interaction is required, facilitated, or

4. encouraged in any way. The recorded
videos allow for students to be
introduced to the instructor, but that is
not an actual interaction.
6. Course Navigation and Technology
Strengths:
The course was easy to navigate using
the tabs located on the left-hand side. All
the links were active and all PDF files
were able to be opened. The navigation
was logical, consistent, and efficient.
Weaknesses:
The technology necessary for this class
was to have access to the internet; no
other technology was used or
encouraged. Technology was not used
to engage students, only to allow access
to information.
7. Learner Support
Strengths: Course appears to be
designed to provide extensive and useful
supplement to an in-person or video
based course and is an excellent
resource under those conditions.
Weaknesses: The course is not for credit
and does not provide grades. Students
are able to read solutions and self-
assess, however there is no support
available from the instructor outside of
links to related materials (such as on
Wikipedia). The course does not specify
technical or academic support to
learners to promote success.
Actual lectures are not included online.
No direct communication with an
instructor or facilitator is available.
8. Accessibility
Strengths: The actual homework
assignments and solutions are in PDF
format and a quick check in Adobe
Acrobat indicates that the PDF is
accessible. The lecture notes include
intuitive naming for the files and are
listed in sequence. Because images are
not used there are no issues with alt text
for screen readers (excluding issues with
link text). Most website accessibility
issues are minor.
Weaknesses: Meet the author videos have
clear audio and answer useful common
questions, however they do not feature
captions, and no transcript is available.
Large file sizes for downloads, but are
appropriately labeled with file type and
size, although they do not contain
descriptive text for links.
Videos appear in pop-up boxes over main
page text and do not auto-start which can
cause difficulty for persons who do not
have the physical dexterity to easily start
the video due to involuntary movement or
poor fine motor skills and those who are
using a screen reader.
A WebAIM WAVE evaluation of the
lecture notes page found: 2 Errors, 12
Alerts, 9 Features, 43 Structural
Elements, 5 HTML5 and ARIA, and 11
Contrast Errors of concern. Most of

5. these are minor, but reduce accessibility
for some students. The actual homework
assignments are in PDF format and a
quick check in Adobe Acrobat indicates
that the PDF is accessible. However, the
PDF does not include any images that
could assist students with reading
disabilities. The lecture notes are the
same, however they do include intuitive
naming for the files but do not include
file size.
There is only text representation of
content, not multiple representations as
called for by UDL principles.
9. Course Administration
Strengths: For students who attended
the original or a similar course the
materials are useful and the organization
and pacing would be appropriate.
Weaknesses: Due to the design and
purpose of the course, there is no
gradebook or scoring system. The
recommended schedule likely reflects a
campus-based course and timeline.
Summary:
The instructor has provided a series of self-study experiences in combinatorics for the
mathematically inclined. No lecture is available and the course is designed to assist
students who are confident in mathematics and are in search of additional problems
to work in preparation for competition or to interest students who are more advanced
than the typical high school student (as noted in the instructor’s videos). There are
formative self-assessments as homework in the course, however there is no system
for assigning a score, grade, or credit for the course.
There is extensive evidence that the course content and environment is not designed
for students who have cognitive or physical disabilities. There are low levels of
possible engagement and little adherence to Universal Design for Learning principles
of multiple representation. The course has little to offer the casual mathematician, but
would be helpful in the stated context of competition or advanced practice.
Reflection:
This course on Combinatorics: The Fine Art of Counting is designed, as stated on the
syllabus, to be an enrichment course for high school students who are bored with their
math classes. The syllabus also states that only basic algebra skills are required as a
pre-requisite; however, based on the lectures and homework problems assigned,
students must also possess a desire to think critically and understand and apply more
challenging mathematical concepts. Sutherland has put a lot of work into the creation

6. of the materials for this course, but the materials provided online are a supplement to
the in class lectures. It is truly unfair to critique this course as if it were strictly online
instruction. Regardless, there were still many strengths and weakness that we were
able to discuss that should be common practice in both the classroom and online
environments. One imperative piece of information that was missing was the learning
objectives for the course as a whole. The topics that would be covered were listed,
but there were no objectives to determine either the purpose of the course or what the
students should be able to know or do by the end of the course. There was also not
much student engagement supported by the lesson activities and materials nor was
there any type of student-to-student or student-to-teacher interaction that was
required or encouraged.
The overall idea of this course was to give advanced mathematical students a class to
take for fun without having to worry about receiving a grade. At the same time the
course helps students prepare for talent search and mathematical competition
examinations. Sutherland managed the course well considering he was most likely
teaching students who love math and want to work challenging problems for the fun of
it. For those of us who live in the real world teaching students who are not all
advanced mathematicians, it was easy to pick apart the course and find its faults. We
do give Sutherland credit for his innovative and creative homework assignments
written in menu style where students are able to make selections of problems as you
would a menu item.

7. Evaluation 2
Site:
MIT Open Courseware
Course Title:
Game Design
Evaluators: Shelby Simmons,
Sarah Tolson, & Lindsay Tucker
Course URL:
http://ocw.mit.edu/courses/comparative-
media-studies/cms-608-game-design-
fall-2010/
Introduction:
The course we evaluated was Game Design by Massachusetts Institute of Technology’s
online open courseware website. The course was presented by Philip Tan and Jason
Begy, as course CMS.608 for undergraduate students and course CMS.864 for
graduate students. The course focused on the design and analysis of non-digital
games. Students learn the different genres and aspects of games and create their own
games. A major emphasis of this course is on the evolution of game rules.
Tan, Philip, and Jason Begy. CMS.608 Game Design, Fall 2010. (MIT OpenCourseWare: Massachusetts Institute of
Technology),http://ocw.mit.edu/courses/comparative-media-studies/cms-608-game-design-fall-2010 (Accessed 10
Jun, 2014). License: Creative Commons BY-NC-SA
1. Course Overview and Introduction
Strengths:
The purpose of the course is stated in
the syllabus: “to understand the
interaction and evolution of game rules.”
The syllabus briefly introduces students
to the structure of the course, including
two weekly lectures and a weekly lab. It
also discusses how student grades will
be computed. The syllabus is clear,
although concise, and could use more
elaboration. However, grades will be
determined through a combination of
student projects, class participation, and
forum discussion. As discovered
through the lectures, much
communication was done through email
and is no longer accessible—we cannot
access nor visualize any resources they
mention.
Weaknesses:
Instructions are vague; it is unclear on
how to begin the course. This may be
covered during the first session of the
course, but the audio recording is not
available. “Netiquette” is not clearly
stated, but the syllabus does mention
that students will work together in teams
and may be “docked for antisocial or
disruptive behavior.” There is no link to
current policies of the course, nor the
school. Prerequisite knowledge and
competencies are not stated, other than
students are expected to demonstrate a
proficiency in English. The course also
does not discuss what technological
skills and competencies students must
have. There is no introduction from the
facilitator, nor is there an introductory
discussion board post or other evidence
of students getting to know each other
online. There is evidence that students
could access a discussion board, but it is
no longer accessible. Many course

8. materials are not elaborated on, nor are
they available currently—some of the
weekly meeting sessions have been
recorded and some assignments are still
accessible. There is nothing mentioned
about feedback from the instructor.
2. Learning Objectives
Strengths:
Objectives are posted on the
assignments page to accompany each
assignment. The objectives are
measureable. Directions are fairly clear
and help students understand how to
meet requirements for each assignment,
but could be supported with some
examples.
Weaknesses:
Objectives for the course are not clearly
stated, but some are entwined in
directions to the assignments.
Objectives should be made more clear
and written from the students’
perspective. Cannot determine whether
or not objectives are appropriately
designed for the level of this course.
3. Assessment and Measurement
Strengths:
The assignments that are presented,
suggested from the course lectures,
align with what students learned through
their course readings and lecture
sessions. The assignments seem to get
progressively more in depth, requiring
more design knowledge and
responsiveness to feedback and issues
that arise as students move through the
course. The assignments can be work-
based and relevant, depending on the
route a team chose to take when
designing their game.
Weaknesses:
The only assessments that are
accessible through the online materials
provided include the three team-based
project assignments to create games. If
there were any quizzes, etc., they are
not accessible from this site. The
syllabus states that part of the students’
grade will come from their discussion
posts, but there is no apparent rubric for
these posts, nor are they currently
accessible. We cannot determine
whether or not the assignments truly
assess the course objectives, since the
course objectives are not explicitly
stated. The course grading policy is
mentioned, but is not explicit—it lacks a
rubric, or similar tool, to determine exact
grades. The only information available
on grading is the break down that each
assignment counts for 25% of a
students’ grade. In turn, it says that 5%
comes from team work, 10% from the
design process, and 10% from
responsiveness from feedback. There is
no evidence that students could self-
reflect about their learning.
4. Instructional Materials
Strengths: Weaknesses:

9. The instructional materials (audio
lectures, course text, supplemental texts,
and games played in lab) contribute to
the achievement of the learning
objectives for the three major
assignments given. All of the resources
and materials used are cited and
permission given for audio recordings
used for guest lectures. Much of the
instruction material is current, many
older games are used, but that is part of
the nature of the course. The
instructional materials provided allow for
multiple means of representation for
students. They are allowed to listen to
lectures, read text, as well as participate
in the playing and designing of games.
Although it can be inferred, it was not
clearly stated how the instructional
materials are to be used for learning. It
does not appear that any of the reading
material or designing assignments are
optional and that everything is required,
including attendance to class and lab. It
does say that students “should” and are
“encouraged” to participate in the forum
discussions, but it also says that they
are “expected” to do so. This provides
some confusion as to whether or not it
an actual requirement of the course. The
articles required for reading are linked
and easily accessible, but it appears that
many of the games must be purchased.
It is quite possible that they are available
for use in a lab setting, but as it is laid
out on the available course, it seems like
a lot of money would need to be spent
on games that the students might only
use once.
5. Learner Interaction and Engagement
Strengths:
The learning activities where students
work in groups to design their own card
and board games promote the
achievement of the objectives stated for
the activities. There is of course
interaction between students to support
active learning when they are playing the
games in the lab. It is also stated that
they should participate in a discussion
forum about the text and group projects
that they are working on. These tools are
not provided on the course website, but
they are required, so must be available
to the enrolled students. Student-to-
Student interaction is encouraged in a
variety of ways including the forum,
playing games with one another in lab,
and being part of a team to create new
games. Many different learning styles
are addressed including visual, textual,
auditory, and kinesthetic.
Weaknesses:
There is reference in the syllabus to
instructor feedback, but there is no plan
stated for how that feedback will happen
or the response time. There is a definite
indication that students must interact;
however, it is not clearly articulated
(other than a weight of a final grade) as
to how that will be assessed or the
specific requirements.
6. Course Navigation and Technology

10. Strengths:
Course navigation and content functions
as expected. Students have access to
the texts they need as well as audio
recordings of lectures. Navigation is also
consistent, logical, and efficient.
Students have access to previous
student work so that they can have prior
examples to follow while designing their
own games.
Weaknesses:
There are not many opportunities
available on the course website for
students to be an active online learner.
There is reference to a forum for
students to write on, but it is not
available for us to view.
7. Learner Support
Strengths: Site FAQ provides some
technical support to learners.
Weaknesses: The course instructions
and supporting documentation do not list
clear academic or technical support,
although some technical information is
listed in the sites’ FAQ information. The
instructor is not actively monitoring the
course, there is no provision for the
collaboration required for success in the
course. Students would be unable to
self-assess.
8. Accessibility
Strengths: Links open in current tab,
most issues in WebAIM Wave
accessibility evaluator are minor. Course
materials that do not have to be
purchased are in accessible formats with
intuitive links. Captions/transcripts are
provided for audio lectures.
Weaknesses: Course materials require
access to Amazon or some other
resource to purchase them. No explicit
information on accessibility within
course. Length of pages and detail may
contribute to excessive cognitive load for
some students. Proximity of some links
may be difficult for students with visual
or motor skill disabilities.
WebAIM Wave Results (2 pages)
Assignments: 3 Errors, 32 Alerts, 9
Features, 49 Structural Elements, 5
HTML5 and ARIA, 48 Contrast Errors.
Study Materials: 3 Errors, 54 Alerts, 59
Features, 48 Structural Elements, 5
HTML5 and ARIA, 48 Contrast Errors.
9. Course Administration
Strengths: Course is organized in
sections with individual sessions
identified. Some explanation and
philosophy of grading is included.
Grading is suitable for management
even with large class sizes.
Weaknesses: Students must spend
extensive time offline in collaborative
groups in order to be successful in the
graded course. No online gradebook is
available. Extensive
sessions/lectures/assignment info may
make it difficult for some students to

11. navigate course or instructor to keep
track of student progress during the
course.
Summary:
It is clear that the instructor/designer of this course has put extensive effort into the
design of the assignments and learning objects within the course. The recorded
lecture/collaboration indicates that this is an in-person course with online components
(also known as a blended format). The instructor addresses the goals for learning in
the course and provides extensive examples in a variety of appropriate formats. While
the course would benefit from explicit rubrics for discussion posts and assignments, it
does include details about the objectives of the assessments and available materials
include completed assignments.
Reflection:
At first this course sounds like it is going to be all fun and games, and there is
definitely some opportunity for that, but there is also a lot of really hard work that
students have to put into understanding the history, thought processes of game
designers, evolution of game rules, and in the creation of their own games. Again,
similar to the previous course, the information provided online supplements an in-
class lecture and lab. Students have plenty of opportunity for interaction with one
another when they are using the discussion forum, playing card and board games
with each other, and working in groups to design their own games. Overall this course
was very well planned with lectures (by the instructors and guests) accompanied by
readings, labs, and group designing activities. For each of the activities students are
given examples of previous student work to have an example to mimic and get their
ideas flowing. The rubrics were missing though and there was no reference to the fact
that one would be used in the grading process. By the end of the course the students
would have had the opportunity to meet the course learning objectives via many
different avenues and there seemed to be an appropriate blend of project based
learning, interactivity, text, lecture, and discussion opportunities. With as much work
and the critiquing of games that students endure throughout this course, it is our hope
that they still enjoy playing games when they are finished!