Ocean Discovery Institute Ocean Discovery Institute is a nationally recognized leader in providing science education to students in under-served communities. The Institute's work creates significant impacts for their students, with dramatically improved high school graduation rates and college acceptances. In 2012, Ocean Discovery Institute was one of two organizations selected nationally to receive the Presidential Award for Excellence in Science, Mathematics, and Engineering mentoring. In collaboration with the Rokenbok Educational Foundation, Ocean Discovery Institute developed, tested and evaluated the Rokenbok Bridge Challenge curriculum for Middle and High school students in 2011 - 2012.

1. Ocean Discovery Institute/Rokenbok
Curriculum Pilot, Summer 2012
Evaluation Report
Prepared for Ocean Discovery Institute
September 19, 2012
by
Roxanne Ruzic, EdD
Ruzic Consulting, Inc.
10601 Tierrasanta Blvd., Suite G136
San Diego, CA 92124
rox@roxanneruzic.com
(858) 384-7237

2. 2
Ocean Discovery Institute, Rokenbok Curriculum Pilot 2012, Evaluation Report
Prepared by Roxanne Ruzic, September 19, 2012
Introduction
During the summer of 2012, representatives from Ocean Discovery Institute and Ruzic
Consulting, Inc. collaborated to conduct a small pilot evaluation of a new curriculum for
high school students that utilizes Rokenbok toys to teach principles of engineering
through instruction coupled with hands-on, collaborative bridge building activities. The
pilot evaluation was designed to investigate the ways in which students worked together
and engaged with the curriculum activities using the Rokenbok toys, as well as how
successful the curriculum was in helping students master the engineering principles
presented.
Methodology
We piloted the full, four-lesson curriculum with 20 students, 10 male and 10 female,
over the course of two days during a low-cost summer enrichment science program
offered by a nonprofit organization in Southern California. Students who participated in
the pilot ranged in age from 14-17 and represented a wide range of ethnicities.
Students were divided into teams of three or four by the enrichment program directors,
rather than Ocean Discovery Institute instructors, and stayed with these groups
throughout the full, four-lesson unit, with two exceptions. One student arrived late and
was assigned to a group part way through the first lesson. Two students from another
single group were absent for the fourth lesson. The remaining student from that group
was reassigned to another group for that final lesson.
During the pilot evaluation, data were collected in five ways:
1. Observations of student activities and student conversations during the lessons;
2. Informal interviews with students during the lessons;
3. Examination of artifacts, such as worksheets and Rokenbok toy bridge models,
created by students during the lessons;
4. Short content exams given to students three times during the full unit (before the
first lesson, midway through the full unit, and after the final lesson);
5. Surveys of students at the conclusion of the full unit.
Findings
We identified five primary findings from the pilot evaluation:
1. The curriculum using Rokenbok toys was highly engaging for students;
2. The curriculum encouraged high-level problem solving and collaboration;
3. The majority of students improved their understanding of engineering principles
presented in the unit;
4. Instructor guidance was a critical component of the student learning experience;
5. Students enjoyed the experience.
Each of these findings is discussed in more depth below.

3. 3
Ocean Discovery Institute, Rokenbok Curriculum Pilot 2012, Evaluation Report
Prepared by Roxanne Ruzic, September 19, 2012
The curriculum using Rokenbok toys was highly engaging for students
“My favorite part of this experience was when we were testing out our bridges. The
suspense in the air as the weights were being placed was kind of funny.”
—Female pilot evaluation student
• All student groups were able to design and build a full bridge during a single
45-minute class period using the Rokenbok toys.
• Students were able to prototype rapidly while still creating bridges that
differed from one another using the Rokenbok toys.
When students entered the room for the first activity, there was little interaction.
Students did not appear to know each other well and—with the exception of a single
group—sat quietly, not talking with each other, not looking at each other, and not
interacting. In the short initial introduction to the unit, most students did not volunteer to
speak.
Once students were presented with the first activity, however, all the students in the
room engaged almost immediately with the task of working together to choose and build
a bridge with the Rokenbok toys, modeled after one of the types of bridges presented.
Not all students participated in the same way, and the dynamics in each group were
different. Some students put pieces of the bridges together, some watched and gave
suggestions or direction to their teammates, and some alternated between participating
in building and giving suggestions. Students sometimes took on different building tasks,
and sometimes worked on a single piece of the bridge together. Students in some
groups talked extensively with each other, while other groups spoke barely at all. All
participants except one (95%), however, were fully engaged in the process of building
the bridge throughout the entire lesson. This pattern of total engagement by all students
except one during bridge building was consistent across all four lessons, including
when members were moved or added to new groups. In addition, the student who was
partially rather than fully engaged during the first two lessons also engaged more fully
when her group shrunk to only one other member during the third lesson.
During the bridge breaking process, students again were focused entirely on the
activity, with all eyes in the room on each bridge being tested each time. There was a
sense of excitement and suspense, and there was cheering or laughter each time a
bridge broke during the first day. During the second day, when bridges were able to
hold significantly more weight, the bridge breaking process took longer. Three additional
students were less engaged during parts of the longer wait for each bridge to break, but
the room remained almost entirely silent throughout the bridge breaking process until
each bridge broke.
We did not test potential variations in the curriculum empirically, and therefore cannot
make definitive claims about the causes of these high levels of engagement. We
hypothesize that a number of aspects of the activity we witnessed may have
contributed. These include the requirement that students work in teams; the use of
hands-on, visible manifestations of the engineering problem being considered; the

4. 4
Ocean Discovery Institute, Rokenbok Curriculum Pilot 2012, Evaluation Report
Prepared by Roxanne Ruzic, September 19, 2012
specification of a desired end-goal (a bridge that supported the most weight); testing in
the form of competition; and the use of building tools that enabled students to prototype
rapidly and test variations in design while still creating structures that differed from one
another.
Requiring that students work on a concrete, visible, well-defined problem may have
allowed students to more easily picture and discuss their conceptual activity and remain
clear about their goals and progress. Requiring that students work in teams and engage
in public competition may have provided motivation through interpersonal interaction
and the desire to succeed in a public forum. Using tools that allowed for rapid
prototyping may have made the building periods more motivating for students because
they could create their designs relatively quickly. The potential for variability in design in
the building tools have made both the design and testing phases more interesting for
students because they could make authentic choices and see the effects of those
choices.
The building materials used with the curriculum unit during the pilot study consisted of
pre-formed Rokenbok toys: plastic pieces including beams, blocks, risers, braces, and
supports. With these toys, students were able to construct their bridges rapidly, rather
than having to create their bridges brick-by-brick. During the pilot study, all student
groups were able to design and build a full bridge during a single 45-minute class period
using the Rokenbok toys. In addition, students had time to experiment during the
process. We observed members of each team creating and taking apart portions of
their bridges throughout each activity as they tested, discussed, and refined their bridge
designs on the fly.
The fact that the Rokenbok toys used during the activities were able to fit together in
different ways may have made the bridge breaking portions of the curriculum more
interesting for students as well. During the pilot study, different groupsʼ final bridge
designs varied from one another, even when all groups were asked to work on the
same type of bridge during a particular lesson. This variation in final designs may have
made the tests to see which bridge would hold the most weight more suspenseful than
if all the bridges tested had been essentially the same.
The curriculum encouraged high-level problem solving and collaboration
“My favorite part was getting to build the bridges because they challenged our group
and made us work as a team.”
—Male pilot evaluation student
• The bridge building curriculum using Rokenbok toys fully and consistently
engaged all students but one in deep reasoning and problem solving
activities directly related to the content being presented.
Many school activities that engage students fall into one of two categories: assignments
that engage students but do not require them to think deeply, and assignments that
engage students in activities that do not focus directly on the academic content of a
particular unit.

5. 5
Ocean Discovery Institute, Rokenbok Curriculum Pilot 2012, Evaluation Report
Prepared by Roxanne Ruzic, September 19, 2012
For example, students are sometimes asked to write down information individually,
follow directions using individual sets of tools, or work as members of a group to repeat
a pre-determined process using a set of tools for the full group. These types of lessons
often lead to high levels of engagement while students participate in hands-on activity,
but the lessons themselves often do not require the students to think deeply about the
content being presented. Instead, the focus is on following directions and completing
the task.
Alternately, students are sometimes divided into groups and asked to work on projects
or create presentations with more freedom. These types of activities also often lead to
high levels of engagement while students focus on completing the project, but the
activities often do not require students to deeply engage with the content being
explored, as when students spend time engaged in tasks such as looking for images
and videos to complete a presentation rather than focusing on making sense of new
content or solving authentic problems.
Unlike these examples, the bridge building curriculum using Rokenbok toys tested in
this pilot evaluation fully and consistently engaged all students but one in deep
reasoning and problem solving activities directly related to the content being presented.
Every group, over each of the four lessons, spent 100% of the time for each activity
deeply engaged in the process of deciding how to build the different kinds of bridges
and why. We observed no off-task activities during the building process, and noted that,
with the exception of the one student already mentioned, 100% of students were fully
engaged in the process with other group members throughout each of the four units.
The work we witnessed was truly collaborative, with students creating plans in their
notebooks, discussing options, building, testing, taking designs apart and re-building, all
with the full participation and involvement of all group members throughout. The bridge
building tasks were intellectually challenging without being prescriptive, and students
were fully focused on those tasks. The Rokenbok toys used during the unit supported
this curricular approach, providing structure and constraints while still allowing for
creativity in the final structures.
While we did not test variations in the curriculum empirically, we hypothesize that this
combination of constraints and flexibility within a hands-on curriculum focused on a
concrete task may have provided the opportunity for students to engage thoughtfully
and authentically with their designs as opposed to following a lock-step procedure or
engaging with activities outside the core learning goals.
The majority of students improved their understanding of engineering principles
presented in the unit
“My favorite part was just learning about bridges in general and getting to build each
type and the learn the flaws/strengths of each type.”
—Female pilot evaluation student
• The majority of students demonstrated an increased understanding of
engineering principles presented in the curriculum unit after participating in
the four lessons using the Rokenbok toys.

6. 6
Ocean Discovery Institute, Rokenbok Curriculum Pilot 2012, Evaluation Report
Prepared by Roxanne Ruzic, September 19, 2012
Before the first lesson, 7 of 19 students (37%) who took the pre-assessment were not
able to identify the three different kinds of bridges—beam, arch, and truss—presented
in the unit. By the end of the second unit, 100% of students were able to identify all
three.
Similarly, before the first lesson, no student was able to accurately describe the
dissipation of forces in the three different kinds of bridges presented in the unit. Before
the first lesson, 8 of 19 students (42%) were unable to accurately identify any of the
forces acting on any type of bridge presented in the unit.
By the end of the full unit, 17 of the 20 students (85%) had demonstrated an increased
understanding of dissipation and the forces acting on the bridges. After the final lesson,
3 of 16 students who took the final assessment (19%) were able to accurately depict
how dissipation functions in an arch bridge; 7 of 16 students (44%) how dissipation
functions in a truss bridge; 9 of 16 students (56%) how dissipation functions in a beam
bridge; and 2 of 16 students (13%) how dissipation functions in all types of bridges
presented in the unit.
Instructor guidance was a critical component of the student learning experience
“My favorite was building the truss bridge. I liked it because it was challenging and
seemed like the strongest.”
—Female pilot evaluation student
• Instructor guidance was important in helping students understand the
engineering principles presented in the unit and how to design effective
bridges using the Rokenbok toys.
An important aspect of the curriculum evaluated in this pilot study was the guided
nature of the instruction. In addition to the short informational sessions at the beginning
of each lesson and the structure created by the specific bridge building activities, the
instructor created additional support and structure for student learning by asking
thoughtful questions of the students throughout the bridge building process. As students
were involved in the hands-on group building activities, the lead instructor moved about
the room, asking open-ended questions of the students about their designs. On multiple
occasions, we observed the conversation and building activities shift immediately in
response to these questions. The questions did not seem designed to test studentsʼ
knowledge or check understanding by eliciting a correct answer for all members of the
group to hear. Instead, the questions seemed designed to shift student focus to
important but neglected aspects of the engineering process in an open-ended way. In
most cases, the instructor did not wait for an answer to her question before moving on
to the next group. Instead, she examined the bridge and the building process and then
posed a question or questions that she left the group to discuss and consider.
Questions asked in the pilot (and included as examples in the curriculum unit)
addressed such topics as where studentsʼ bridges were weak and why; why students
chose particular design elements and how those elements affected bridge strength and
stability; and how students might use the engineering principles presented in the unit to
strengthen their designs.

7. 7
Ocean Discovery Institute, Rokenbok Curriculum Pilot 2012, Evaluation Report
Prepared by Roxanne Ruzic, September 19, 2012
Students enjoyed the experience
“Great thing to do in a science or engineering class!”
—Male pilot evaluation student
• A significant proportion of students found it challenging to work with the
Rokenbok toys.
In surveys at the conclusion of the program, students were upbeat about the unit, with
100% of students reporting that they enjoyed the experience. In fact, 5 of the 16
students (31%) who completed the final survey asked that the session be longer or that
they be allowed to complete additional activities. Students cited the hands-on nature of
the activity, the chance to learn more about bridges and the principles behind them, and
the opportunity to collaborate and work as a team as positive aspects of the curriculum.
Students had few suggestions for improvements, though six students explicitly
mentioned that they struggled when working with the Rokenbok toys. During the lessons,
we observed members of every team having difficulty putting pieces together and pulling
pieces apart. In addition, several groups of student did not seem to distinguish between
the 30-degree and 60-degree connectors, which led to unexpected issues with at least
three bridge designs.