Reflections on Teaching presented at a workshop for faculty.

1. Reflections on Teaching

2. Course Improvement
• We can simply use the feedback to pat
ourselves on the back, documenting that
we are doing a great job
• Or we can use it to capitalize on strengths
and improve weak areas
• Either way, we can include some
information

3. Sources of Information
• Gut feelings
• Student performance on the content, in
the class or on external tests
• Informal feedback from students
• Peer visits and feedback
• Committee visits and feedback
• IDEA evaluations

4. Easy to interpret
• Recommendations from colleagues,
supervisors and students
• Strong student performance
• High evaluations in all categories
• The latter two don’t really suggest any
changes
• The first often comes easy to implement

5. Harder to Interpret
• Poor student performance
• Gut feelings
• IDEA forms
• The IDEA forms have a lot of information,
some of which can be useful

6. What is reliable anyway?
• This is simply a function of the number of
responses; lower numbers are considered
less reliable. 25 or more responses are
noted as “reliable”, 15?-24 are noted as
“fairly reliable”, 10-15 are noted as
“marginally reliable”, and 9 and below are
noted as “unreliable.”

7. And what is representative?
• This is the fraction of the class that filled
out the form; this is an attempt to ascertain
if the response is representative of the
class. Response rates of 65%-79% are
noted as “probably representative”;
response rates of 80% and higher are
noted as “representative.”

8. IDEA Raw vs. Adjusted
• The Raw score is the average of the
responses by the students.
• Why adjust this?
• There are factors that affect scores that
are independent of the instructor or the
class.
• An example of this is the students’ desire
to take the course, regardless of teacher

9. Factors to adjust for
• Class size (larger classes get lower ratings)
• Type of course (core requirements get lower
ratings)
• Student interest (interested students rate
courses higher, uninterested students rate them
lower)
• These are called “extraneous variables” and are
discussed in detail in IDEA Research Report #6

10. How are adjustments made?
• The raw scores are adjusted, based on
students’ answers to the questions
pertaining to interest and effort and based
on class size and level. In short, if most of
the students report a high desire to take
the course, the raw score will be adjusted
down a bit. For a large class, the ratings
adjust up a bit.

11. Why adjust scores at all?
• Once the scores are adjusted, it is possible to
make comparisons between classes, without
having to worry about class size, or the rest.
• You can compare your own ratings between
classes, or from one semester to another.
• We can compare the ratings (which are a
reflection of student perceptions, of course)
between faculty.

12. Key General Indicators
• Progress on Course Objectives
• Excellent Teacher
• Excellent Course
• These are always worth looking at as
summary statistics. They do not provide
much detail.
• Good for identifying serious issues, or for
patting ourselves on the back.

13. Progress on Course Objectives
• This is a summary evaluation, based on a
number of questions in the form.
• It does reflect student perception on their
progress.
• The student responses that are used for
this indicator are those objectives (items
21-32) that the professor picked as
“important” or “essential” (counting
double)

14. Excellent Teacher
• This is a summary evaluation, based on
Question #41 on the IDEA form.
• It does reflect student perception of their
teacher.

15. Excellent Course
• This is a summary evaluation, based on a
number of questions in the form.
• It does reflect student perception on their
progress.
• The question that is used for this indicator
simply asks the students if it was an
excellent course, question #42

16. Clear Indicators
• Good ratings in general, 4.0 or above in
the three summary numbers would mean
that the students are well satisfied with the
course.
• Poor ratings in general, 2.5 or below in the
three summary numbers would mean that
the students are very unhappy with the
course.

17. Mixed Results
• We saw this in FYEx 103. Several faculty
received very high ratings as teachers, but
the course ratings were very low. This
was echoed by the written comments.
• This reflects student satisfaction with the
professor, but not with the class. The rest
of the evaluation can help us discern what
about the class was a problem for them.

18. Information to modify a class
• The IDEA reports cannot substitute for
judgment on our part.
• They can give us information on what
students liked or didn’t like about a class.

19. Why do we care?
• Okay, students can’t tell if we are properly
qualified.
• They can tell if we seem competent.
• They do make decisions based on how
they respond to us.

20. What is this “converted” score?
• The converted scores are simply
normalized to an average of 50, with 63
being the upper 10%.
• This allows comparison between
questions, as well as between classes,
semesters, faculty, etc.

21. What information can be useful?
• Other than identifying student
dissatisfaction, some questions can
provide specific guidance.
• Specifically, the last page, statistical detail
can provide specific guidance

22. How Can We Use IDEA
Results to Improve
Instruction?
https://theideaonline.org/idea/cs/index.jsp

23. Teacher
Old Master

24. The “Dr. Fox”
Lecture
In a well-known 1970 study,
a professional actor (Michael Fox)
was hired to deliver a non-substantive
and contradictory lecture, but in an
enthusiastic and authoritative style.
Naftulin, Donald H., John E. Ware, and Frank A.
Donnelly, "The Doctor Fox Lecture: A Paradigm of
Educational Seduction," Journal of Medical Education
48 (1973): 630-5.
Rice, Lee. "Student Evaluation of Teaching: Problems
and Prospects," Teaching Philosophy 11 (1988): 329-
44.
X

25. The “Dr. Fox” Lecture
http://ecclesiastes911.net/doctor_fo
• The lecture was held at the University of Southern California School
of Medicine.
• The subject was "Mathematical Game Theory as Applied to
Physician Education."
• The speaker was introduced as Dr. Myron L. Fox from Albert
Einstein College of Medicine, a pupil of von Neumann and an
authority on the application of mathematics to human behavior. The
attendees were psychiatrists and psychologists (MDs and
PhDs) who were gathered for a training conference. They
listened to the lecturer with great interest, asked many questions,
and were satisfied with speaker's replies.

26. Let’s Watch A YouTube Excerpt
of Dr. Fox’s Lecture
• http://www.youtube.com/watch?v=Rc

27. Dr. Fox’s Lecture
• Dr. Fox gave his talk on a Scientific American
article on “Game Theory. “ He borrowed from the article some
phrases, but not any sense. He mixed them up with allusions to unrelated
subjects, which on purely verbal level may appear relevant. He conducted
himself with great confidence and showed such a mastery of the
aforementioned allusions that the audience was convinced that a luminary is
standing before them.
• 100% had answered in the questionnaire that Dr. Fox had
stimulated their thinking, 90% said that he presented material
in a well organized form, and 90% said that Dr. Fox put his material
across in an interesting and factual manner.
• Some audience members even commented that they
• had read Dr. Fox’s published papers!

28. Effects of the “Dr. Fox” Study
Can a brilliant delivery technique of a talk so
completely “bamboozle” a group of experts
that they overlook the fact that the content
was nonsense?
The published study in 1973 cast serious
doubts on results obtained from evaluation of
teachers and lectures.
More modern studies have affirmed results
from the “Dr. Fox” study.

29. What does “Ms. Mentor” in The
Chronicle Say?
The Torment of Teaching
Evaluations
•By Ms. Mentor
•Question: I work twice as hard at teaching as anyone I know, but I still can't get good student evaluations. I've tried outlines and
keywords on the board, handouts, individual meetings, midterm evaluations, peer observations, lecturing more, lecturing less.
Some student comments are so harsh and demoralizing that I put off reading my evaluations until school breaks, when I have time
to be depressed. Is it possible that my low evaluations stem from personality issues that I can't do anything about? Am I alone?
•Answer: Certainly you are not solo. Ms. Mentor's mailbag is full of complaints about student evaluations -- none of them from
students. Faculty members feel that they are cheated, mistreated, and misunderstood. Often they're right, for most evaluation
forms are so vague and perfunctory ("Concerned about students -- rate 1 to 9") that they do nothing to improve teaching. More
often, they become weapons to get rid of untenured profs who have made enemies. No one has ever really agreed on what makes
a "good teacher," and Ms. Mentor still grieves for Socrates, who taught his students to question everything, and wound up dead
instead of tenured.
•But your risk is smaller. You just need good evaluations, and Ms. Mentor can tell you how. (Naive and idealistic souls may wish to
stop reading at this point.)
•Simplest of all, you can give higher grades, which do correlate with student ratings. You can use more hand gestures, modulate
your voice more, and walk while you talk. Students give higher evaluations to teachers who are good-looking or very dramatic. This
is called "the Dr. Fox effect," named for a hired actor who purported to be "Dr. Fox" and who gave a nonsensical university lecture
in a wildly entertaining style, and got outstanding student evaluations for his brilliance.
•In one notorious study, those who saw just a 30-second soundless video of a teacher in action gave him virtually the same ratings
as the students taking his course -- who'd spent a semester reading, writing, thinking, and talking with him. Smile warmly for the
first 30 seconds of the first class in January, and you'll get good scores in May.

30. Give Higher Grades, Use More
Hand Gestures… What to Do?
There are some more effective ways to
improve student learning in the classroom.

31. Mine Your IDEA Results and
Other Data

32. Using IDEA Results to Improve Instruction &
Learning: EXAMPLES
1. Use student PERCEPTIONS of general and
specific SLO progress.
2. Compare student PERCEPTIONS with
some measure of REALITY.
3. Show PROGRESS in various areas over time in
the same course from semester-to-semester.
4. Sift through student COMMENTS to find
relevant and useful information.

33. Student Perceptions of
Achieving Progress on SLOs
• General (Progress on Course Objectives)
• This is a summary evaluation, based on a number of questions in the form.
• It does reflect student perception on their progress.
• The student responses that are used for this indicator are those objectives
(items 21-32) that the professor picked as “important” or “essential”
(counting double)
• Specific (Supplemental Questions
provided by the Instructor)

34. Indirect Measures-SLOs
• You can use information from your IDEA
results (supplemental questions) to
examine student perceptions of achieving
SLOs in your courses.
No.
students
SLO#1 SLO#2 SLO#3 SLO#4 SLO#5 SLO#6 SLO#7
SLO#8 SLO#9 SLO#10
Murphy
MWF 8
25 4.3 4.4 4.5 4.7 4.7 4.5 4.3 4.1 4.0 3.9
Murphy
TR 8
21 4.4 4.0 4.2 4.8 4.7 4.8 3.6 4.4 3.6 3.9
Mean
(Murphy)
23 4.3 4.2 4.3 4.7 4.7 4.6 4.0 4.3 3.8 3.9
INDIRECT MEASURES OF STUDENT LEARNING OUTCOMES
Chemistry 105: General Chemistry I
Fall 2010

35. Even More…
• Once you have a good idea of student
perceptions on whether or not they
achieved the SLOs, compare their
perceptions with REALITY (direct
measures).
• Pick questions which directly measure
student achievement on a single SLO on a
test or examination at the end of the
course. Mine the data.

36. Compare Student Perception with
Reality: GEN CHEM I
• SLO #6 “After taking this course, I feel I am
able to balance chemical equations.”
No.
students
SLO#1 SLO#2 SLO#3 SLO#4 SLO#5 SLO#6 SLO#7
SLO#8 SLO#9 SLO#10
Murphy
MWF 8
25 4.3 4.4 4.5 4.7 4.7 4.5 4.3 4.1 4.0 3.9
Murphy
TR 8
21 4.4 4.0 4.2 4.8 4.7 4.8 3.6 4.4 3.6 3.9
Mean
(Murphy)
23 4.3 4.2 4.3 4.7 4.7 4.6 4.0 4.3 3.8 3.9
From the ACS Gen. Chem. I Examination (questions #15 and #24) given right after
the IDEA Evaluations, 23/25 and 20/21 of the students could correctly balance a
chemical equation.
CHEM 105 Student Perception Direct Measure
MWF 8 4.5/5.0 23/25 (93%)
TR 8 4.8/5.0 20/21 (95%)

37. Compare Student Perception with
Reality: GEN. CHEM I
No.
students
SLO#1 SLO#2 SLO#3 SLO#4 SLO#5 SLO#6 SLO#7
SLO#8 SLO#9 SLO#10
Murphy
MWF 8
25 4.3 4.4 4.5 4.7 4.7 4.5 4.3 4.1 4.0 3.9
Murphy
TR 8
21 4.4 4.0 4.2 4.8 4.7 4.8 3.6 4.4 3.6 3.9
Mean
(Murphy)
23 4.3 4.2 4.3 4.7 4.7 4.6 4.0 4.3 3.8 3.9
From the ACS Gen. Chem. I Examination (questions #10 and #16) given right after
the IDEA Evaluations, 13/25 (52%) and 11/21 (53%) of the students could correctly
do this.
Student Perception Direct Measure
MWF 8 3.9/5.0 13/25 (52%)
TR 8 3.9/5.0 11/21 (53%)
• SLO #10 “After taking this course, I feel I am able to calculate the standard
enthalpy of reaction for a given series of reactions by using tabular data.”

38. SHOW TEMPORAL
PROGRESS
CHEMISTRY 105: General Chemistry I Fall Classes
IDEA Results: Must be reliable and representative
Progress on Objectives Excellent Teacher

39. What If?
Think about the above results:
1.Same course objectives selected each year?
2.Changes in class size and/or population
3.Change in teaching techniques
4.Other
5.Supplement with additional measures

40. How Can Student
Comments Be Used
Effectively?

41. Sample Student Comments• Huntingdon College
• IDEA Survey Student Comments
• Spring 2011
• Murphy, Maureen – CHEM 106 – MWF 8:00
• · Great instructor!
• · Great instructor. Made concepts easy to understand.
• · Dr. Murphy was an outstanding instructor.
• · Left-handed teachers-step away from board so students on the right side of the classroom can see more!
• · Great class!!!
• Murphy, Maureen – CHEM 106 – TR 8:00
• · The course and approach is fair and straightforward. I believe all courses and
• institutions should have a Dr. Murphy.
• -Quit being so positive-heck, it is 8 am in the morning!
• · Great teacher!! Little hard but wiling to help!!
• · E for the cell= cathode – anode
• · Dr. Murphy has been my favorite teacher in my 2 years at Huntingdon and done
• so teaching one of my least favorite subjects! She’s the best!
• · Dr. Murphy puts a lot of her time in to help her students better understand
• chemistry. She has the most office hours out of all of my professors!
• Murphy, Maureen – CHEM 446 – MW 19:00
• · I really appreciate Dr. Murphy’s concern for her students. She is very open to
• helping whomever needs help and she wants the students to understand the
• material. Her classes are never boring and I always learn a lot. Overall I very
• much enjoyed the class and Dr. Muphy’s teaching and learned a lot.
• · You are an amazing teacher and I appreciate all that you do. If every teacher,
• faculty, and staff member were like you.
• · Dr. Murphy is the best teacher I’ve had in my entire life. She changed my life
• forever. I couldn’t begin to try to name everything she has done for me so I’m not
• going to try. I could never repay her or thank her enough. I love you Dr.
• Murphy, thank you.
• Gets distracted sometimes talking about her husband teaching chemistry also.
Murphy, Maureen – CHEM 409 – MWF 9:15
· Great teacher! Dr. Murphy really cares about seeing us
do well in her class as
well as our life. We each share our own personal
aademic relationship with her.
· Dr. Murphy is the greatest asset at Huntingdon
College. She is an incredible
teacher and cares infinitely about her students. Dr.
Murphy is the only reason I
am where I am today from an academic perspective.
She helped me when I
needed help, and pushed and dragged me through
when I needed dragging. I do
not agree with everything that Huntingdon does, but
having Dr. Murphy around
and involved is easily the smartest thing Huntingdon
has ever done. Thank you
Dr. Murphy, I will never be able to repay you for all that
you have given me.
·
COMMENTS IN BLUE MAY BE
HELPFUL IN IMPROVING
INSTRUCTION

42. Student Comments that
Deserve Attention
• Tests are too hard and do not reflect the
level at which the professor teaches.
• Lets class out early every day-great class!
• Too fast. Slides are read to us in class
and we see 56-75 slides per 60 minutes.
• Cannot (or refuses to) answer our
questions in class.
• Not available or not there during office
hours.

43. SUMMARY
• IDEA Student Ratings of Instruction contain
some useful information for faculty.
• Understanding what the ratings mean and how
they are constructed is important.
• Student perceptions can be useful, but are
limited.
• Supplemental questions can be effectively used
on IDEA forms, especially when paired with
direct assessment of progress on SLOs.
• Evaluation of ratings over time can be useful.
• “Sifted” comments from students can inform
future instruction.

44. YOUR QUESTIONS
??????????

45. Additional Resources
• http://chronicle.com/article/A-Tech-Happy-Prof