Program evaluation 2013

Program Evaluation – Final Report
Program Evaluation- Final Report
Georgia Southern University
FRIT 8435 Program Evaluation
Lara Komanecky & Lisa Witteman
Dr. Carlson
Spring 2013

Table of Contents
I. Executive Summary………………………………………………………………………………………………………………….…4
Purpose and Scope …………………………………………………………………………………………………………………….4
Audience…………………………………………………………………………………………………………………………………….4
Data Collection and Analysis………………………………………………………………………………………………………4
Findings…………………………………………………………………………………………………………………………………..5-6
Recommendations………………………………………………………………………………………………………………………6
II. Introduction………………………………………………………………………………………………………………………………..6
Purpose of the Evaluation…………………………………………………………………………………………………………..7
Design of the Evaluation……………………………………………………………………………………………………………..7
III. Methodology
Methods of Data Collection………………………………………………………………………………………………………..7
On-Task Behavior Chart…………………………………………………………………………………………………………..7-8
Picture Survey………………………………………………………………………………………………………………………….8-9
Pre-and Post Modified KWL for Sixth Graders……………………………………………………………………………..9
Photo Interview Questions for Elementary Students…………………………………………………………………10
Interview Questions for Sixth Grade…………………………………………………………………………………………10
Limitations of Evaluation…………………………………………………………………………………………………………..11

IV. Results……………………………………………………………………………………………………………………………………..12
EvaluationFindings……………………………………………………………………………………………………………………12
Pre and Post -Sixth Grade Modified KWL Intrinsic and Extrinsic Findings………………………………….12
SEN K-2 Visual Intrinsic and Extrinsic Survey Findings………………………………………………………………13
On-Task Behavior Findings………………………………………………………………………………………………………..14
Photo Conference Questions Findings……………………………………………………………………………………..15
V. Conclusions and Recommendations…………………………………………………………………………………………15
VI. References………………………………………………………………………………………………………………………………..16

I. Executive Summary
Purpose and Scope
Interactive white boards (IWBs) are used in almost every classroom in this school district. They
are used across content areas and have a variety of applications and uses. IWBs are currently
used to promote learning in K-6 reading programs. The scope and purpose of this evaluation
was to determine whether IWBs impact students’ attending behaviors and organizational
strategies in K-6 reading programs. Student behavior logs, student interviews, SEN surveys and
Photo “wh” questions were used to determine effectiveness of the IWBs.
Based on the percentage data results of this evaluation, we found that the use of interactive
white boards in K-6 reading programs does improve students attending behaviors and
organizational strategies; therefore, this program should be maintained.
Audience
This evaluation took place from February 22, 2013 to April 5, 2013 in the KW School System and
seeks to inform the stakeholders at KW School System on the impact of IWBs on K-6 reading
programs.
Data Collection and Analysis
Data were collected through behavior logs, interview sessions, self-reporting and surveys of 58
students participating in K-6 reading programs. Behavior logs were tracked during white board
segments, while daily observational data was collected by the evaluator. All of the evaluation
participants were observed for attentiveness during their academic period. In the study,
attentiveness was identified and operationally defined as active learning (i.e., looking at the
speaker, focusing on the whiteboard, utilizing books, clipboards/dry erase boards, manipulating
props and materials). Student’s off task behavior was identified and operationally defined as
disengaged (i.e., playing with clothes, shoes, distracted, looking up at the ceiling, etc.).
Attention to task was discussed at the beginning of each lesson and defined by the students as
well. Data was recorded on a 5 minute-interval tally chart during a thirty-minute lesson. Active
learners received a plus sign (+) and disengaged students received a minus sign (-). Weekly
conferences were held for general education students, during which students completed photo
interview questions assessing information retrieval. Participating sixth grade students
completed pre and post evaluations to determine if there were correlations between IWBs and
students’ utilization of organizational strategies. Participating AU students took part in a three
session visual survey to determine if there were correlations between IWBs and students’
utilization of organizational strategies. Data herein is compiled into several charts, displaying
results. These research methods were used with both an AU self-contained group of students
and an inclusion group.

Findings
The findings of the evaluation are listed below, categorized into four areas.
Student Behavior Logs
*(IWB = Interactive white board, WWB=with white board & WOWB=without white board)
Daily 30 minute behavior logs were utilized by observers, at 5 minute intervals for six
weeks to determine students’ on-task behavior. Both Group A and B were assigned
numbers for student identification. Data on all eight students in Group A were recorded
daily. Data on the 48 students in Group B was randomly taken due to class size. The
qualitative data was tabulated weekly to measure on task behavior with an IWB and
without IWB with a baseline of zero and a goal of 85%. Time on task was reported
higher with the IWB in both the AU self-contained group (WWB=72%, WOWB=65%) and
the inclusion group (WWB=78%, WOWB 52%). Although the researchers did not make
their intended goal of 85%, there was active growth in each group. These data
fluctuations were attributed to outliers of age and development of students, class-size,
pre CRCT and post CRCT variables, and prior knowledge fluency.
Interviews
A modified version of the KWL format was employed before and after the study with the
sixth grade inclusion group to measure the impact of IWB lessons on students’
confidence and expression of detail and interest. The qualitative interview questions
were intrinsic and extrinsic in nature, and open-ended. The responses were rated on a
3-point rubric scale (1 – little detail, 2- some details, 3 – very detailed). Pre-Evaluation
results were; averaged intrinsic responses, (1.75) and averaged extrinsic responses,
(2.25). Post-Evaluation results were; averaged intrinsic responses, (2.5) and averaged
extrinsic responses, (2.66). The results were compared and a percentage growth rate
was tabulated. Intrinsic responses increased by .75 and extrinsic responses increased by
.16. These data validated that IWB lessons contributed to students’ increased
confidence and expression of detail and interest.
SEN Survey
A qualitative SEN survey was used with the AU self-contained group. The survey was
visually designed (the employment of visual answers in place of verbal or written
responses), to measure how the IWB influenced their intrinsic and extrinsic motivation
in correlation with their implementation of organizational and retrieval strategies. An
emotion survey rating scale (1-4) was employed for the intrinsic questions (1-2) and
numbers were assigned for the students’ responses for the extrinsic based questions (3-
6). These data detected the students’ underdeveloped expressive and receptive
language skills, which is common in their pervasive disorder, in their extrinsic based
responses. Not unexpectedly, the percentages for extrinsic based responses yielded < -

.5 regression. In their intrinsic based responses, a significant negative correlation was
discovered in the 1.75 percent growth rate. These data percentages displayed the
participants dislike for conventional means of work tasks and an enjoyment for work
tasks on or including the IWB. The data validates that the employment of the IWB
impacts students’ intrinsic motivation, which theoretically, enables them to implement
organizational and retrieval strategies.
Photo “wh” Questions
The study included utilizing photographs to assess students’ engagement and retrieval
process during IWB lessons. Once a week students were shown a picture of their class
during an IWB reading large group lesson. All students were asked a combination of
questions, some open-ended and story element questions. They were instruction to
give detailed responses. The responses were rated on a 3-point rubric scale (1 – little
detail without explanation, 2- some details without explanation, 3 – very detailed with
descriptive explanation). The results were averaged each week and those averages were
divided into six. The final results were Group A (2.25) and Group B (2.5). Interestingly,
both groups remained relatively consistent with neither displaying a significant increase
or significant regression. With Group A, the photo questions showed that, although the
students could recognize themselves in the pictures and familiar stories, they were
unable to express detailed accounts. With Group B, the students exhibited engagement,
maturity and enjoyed detailing their photos.
Recommendations
1. IWB lessons should be implemented in K-6 reading programs, if they are not already.
2. K-6 reading programs already incorporating IWB lessons should do so at an increased level.
3. KW School System should encourage increased use of IWBs in reading programs.
4. Research should be continued to determine the effectiveness of IWBs on other academic
programs.
II. Introduction
The majority of classrooms in KW School District contain Interactive White Boards (IWBs). These
white boards are often used in reading programs. This program evaluation sought to determine
the effectiveness of IWBs in K-6 reading programs. Two evaluation questions drove this study:
a. Can the incorporation of an interactive white board impact students attending
behaviors during guided reading?
b. Can the incorporation of an IWB increase students’ (general education and SWD)
utilization of organizational strategies?

Purpose of the Evaluation
Stakeholders agreed that the impact of IWBs should be evaluated in K-6 reading programs.
These findings will be shared with faculty and staff to determine future use of IWBs in K-6
reading programs, as well as the use of IWBs in other content areas.
Design of the Evaluation
Two student groups were included in this study. Group A consists of eight self-contained autistic
students. Group B consists of 48 students in inclusion classrooms. Of the 48 students in group B,
19 receive gifted services, 10 receive special education services and one receives both gifted and
special education services. Data collection methods were the same for both groups. Behavior
logs were tracked during white board segments, a modified version of traditional KWL (what you
know, what you want to know and what you have learned) was utilized for interview questions
(Group A was given super symbols to answer interview questions), and photo questions were
used with both groups.
III. Methodology
Methods of Data Collection
Data were collected from February 22, 2013 to April 5, 2013. Behavior logs were tracked for the
duration of the study, six interview sessions took place from March 1st
to April 5th
, self-reporting
for sixth grade students took place on February 22nd
and April 5th
, and SEN surveys were
distributed three times, March 1st
, March 15th
and March 29th
.
Figure A. On-Task Behavior Chart
The chart below was used to record on-task behavior with and without an IWB on all
participants.
With IWB
Days of the Week
Monday
Tuesday
Wednesday
Thursday
Friday
Totals
Average
Number __ +/- Number __ +/-Number __ +/- Number __ +/- Number __ +/-

Without IWB
Days of the Week
Monday
Tuesday
Wednesday
Thursday
Friday
Figure B- Picture Survey
The picture survey below was used for elementary students.
Picture Survey Questions: Student Number____ Time______ Date________
When I use scissors to cut out story pictures then glue stick to glue them in a square, I feel…….
1 2 3 4
When I use the white board and pen to move story pictures to a square, I feel……..
1 2 3 4
I can find the title of a book and the author using a…………
1 2 3
I can see the bigger words, story characters, story settings, when I look at…
1 2 3
Totals
Weekly Average
Number __ +/- Number __ +/-Number __ +/- Number __ +/- Number __ +/-

When I watch stories on the white board and see the big words and pictures, I……
1 2 3
It is easier to use my story checklist at my seat looking at the story on the whiteboard or book
1 2
Figure C - Pre-and Post- Modified KWL for Sixth Graders
Prediction and What I Know- Bubble Head (Intrinsic) Prediction and What I Know-verbally respond (extrinsic)
* Will you learn anything using the interactive whiteboard (IWB)?
1 – little detail 2- some details 3 – very detailed
• What new reading skill will learn a using the IWB?
• Will you be patient taking turns at the board?
• Do you think the IWB will change the way you think
about reading? How?
• Have you ever put your own work on the IWB?
•Who would you tell in another school: a teacher/child to use the
IWB?
• Do you think other children/teachers/parents learn with the IWB?
• Do you think using the IWB for reading will be a good idea?
Why?
• Do you think it won’t be fun to use the IWB? 1 – little detail
2- some details 3 – very detailed
• Would you feel embarrassed to show anyone your work on the
IWB? Why? 1 – little detail 2- some details 3 – very
detailed
• How do you feel about working on the IWB? Why? 1 – little detail
• What grade would learn the best using the IWB?1 – little detail
*How did answering these questions make you feel?

Figure D- Photo Interview Questions for Elementary Students
Interview questions Elementary
Student Number:____ Time:_____ Date:_________
Candid pictures of the students during their reading segment were employed to ask the following
questions….
What are you thinking in this picture?
1 – little detail (Student did not remember specific details) 2- some details (Student remembers 1-4
specific details) 3 – very detailed (Student remembers and describes specific details)
Do you remember the name of the story?
1 – no title 2- half title 3 – full title
What was your favorite part of that story? Why?
1 – little detail without explanation 2- some details without explanation 3 – very detailed with
descriptive explanation Total_____ Average______
Figure E - Interview Questions- 6th
grade
Interview questions-6th
Student Number:____ Time:_____ Date:_________
What are you thinking about in this picture?
When you look at the picture: Do you remember the day off the week?
What lesson is it?
What is going on in the story?
Who was the main character?
Total_____ Average______

Limitations of the Evaluation
Although IWBs are used in various programs throughout the school, this study focuses on their
effectiveness in reading programs only. The number of student participants in this study is also
limited. Due to time constraints, a larger number of students could not be assessed. Data
analysis tables were not preformed due to the limited sample population and relevant past
research was given honorary mention in the appendix for inspiring and guiding the study, but
not directly cited in the evaluation.
IV. Results
Findings
Research showed that IWBs positively impact students attending behaviors and organizational
strategies, as displayed in Pre-and Post-Sixth Grade Modified KWL Chart, SEN K-2 Intrinsic and
Extrinsic Survey chart, on-task behavior chart and photo questions survey chart, found in the
appendices.
Figure F - Pre and Post -Sixth Grade Modified KWL Intrinsic and Extrinsic Findings
0
0.5
1
1.5
2
2.5
3
1
2
3,2,1AveragedRubricScores
1 2
Pre 1.75 2.5
Post 2.25 2.66
Pre and Post Sixth Grade KWL Results
First and Last Week

The Pre and Post KWL Interview data were utilized to determine IWBs’ impact on students’
personal and academic (intrinsic and extrinsic) organizational strategies. These data illustrated a
positive correlation between students’ intrinsic and extrinsic organizational strategies with the
use of IWBs in reading programs.
Figure G - SEN K-2 Visual Intrinsic and Extrinsic Survey Findings
Elementary students’ intrinsic scores significantly improved with the utilization of IWBs in their
reading programs. These data can signify that the students are personally invested in their
academic tasks.
0
0.5
1
1.5
2
2.5
3
3.5
1
2
3
1 2 3
Group A SEN Intrinsic 3.25 3 3.5
Group A SEN Extrinsic 1.5 2 1.5
SEN K-2 Visual intrinsic and Extrinsic Survey
Number of Survey Weeks in

Figure H - On-Task Behavior Findings
All groups showed an increase in on-task behavior with IWB lessons.
0
0.5
1
1.5
2
2.5
3
1
2
3
4
5
6
1 2 3 4 5 6
Group A 2 2 2 2 2.5 3
Group B 2.25 2.5 2.25 2.75 2.75 2.5
Photo Conference Questions
Group A
Group B
Week Number

Figure I - Photo Conference Questions Findings
V. Conclusions and Recommendations
Because research showed improved on-task behavior and improved organizational strategies
with the incorporation of IWBs into K-6 reading programs, the following recommendations are
made:
1. IWB lessons should be implemented in K-6 reading programs, if they are not already.
2. K-6 reading programs already incorporating IWB lessons should do so at an increased
level.
3. KW School System should encourage increased use of IWBs in reading programs.
4. Research should be continued to determine the effectiveness of IWBs on other
academic programs.
5. Research should further explore the correlation between IWBs utilization in the
classroom and its impact on students’ beliefs about learning and metacognition.
Successful implementation of this study, in conjunction with past research and other school
improvement efforts, promises positive correlation with IWB’s and the students’ initiation and
continued self-regulation of attending (active learning) during lesson time, which research
states, improves students’ academic achievement and self-efficacy.
0
0.5
1
1 2
3
4
5
6
1 2 3 4 5 6
Group A NWB 0.56 0.63 0.59 0.65 0.71 0.74
Group A WB 0.62 0.67 0.73 0.77 0.75 0.81
Time on Task: Group B
without IWB
0.4 0.33 0.55 0.65 0.7 0.5
Time on Task: Group B with
IWB
0.75 0.7 0.7 0.85 0.9 0.7
Time On Task - with and without IWB
Week Number
AveragedPercentages-TaskOnTask

Reference Page
Honorary Mention
The following are articles that supported and gave perspective to the Evaluation Study.
Carnahan, C., Williamson, P., Hollingshead, A., & Israel, M. (2012). Using technology to support
balanced literacy for students with significant disabilities. Teaching Exceptional Children,
45(1), 20-29
Frauenberger, C., Good, J., & Keay-Bright, W. (2011). Designing technology for children with special
needs: bridging perspectives through participatory design. CoDesign, 7(1), 1-28. doi:
10.1080/15710882.2011.587013
McKenna, M., Labbo, L. & Reinking, D. (2003) Effective use of technology in literacy Instruction, in: L.
Morrow, L. Gambrell & M. Pressley (eds), Best Practices in Literacy Education, 2nd
edn. (New
York, The Guilford Press).
Solvie, P. (2007). Leaping out of our skins: postmodern considerations in use of an electronic
whiteboard to foster critical engagement in early literacy lessons. Educational Philosophy and
Theory, Vol. 39, No. 7, 2007. doi: 10.1111/j.1469-5812.2007.00312.x

Program evaluation 2013

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (19)

En vedette

En vedette (19)

Similaire à Program evaluation 2013

Similaire à Program evaluation 2013 (20)

Plus de lisawitteman

Plus de lisawitteman (20)

Program evaluation 2013