Yael Kali and Tamar Ronen-Fuhrman: Making Expert Design Knowledge Useful for Novice http://www.ld-grid.org/workshops/ASLD11

1. Making Expert Design Knowledge Useful for Novices
Yael Kali (yaelk@edtech.haifa.ac.il)1
Tamar Ronen-Fuhrmann (tamarrf@gmail.com) 2
1) University of Haifa
2) Technion – Israeli Institute of Technology
In the past decades, much design knowledge has been gained by expert educational technology designers, and accumulated via
design research projects. Learning scientists have sought various ways to make this knowledge available and useful for other
educational technology designers, and particularly for novices in this field. These efforts are part of a trajectory which views
education as a design-science (Collins et al. ,2004), or even more broadly, as one of the sciences of the artificial (Simon, 1969). In
such fields generalizations of common examples are often articulated, to enable their application in other settings. Following the
work of Alexander (1979), and his vision for articulating a “design pattern language” in architecture, learning scientists have
developed several manners to articulate general design guideline for curricular design. Three main types of guidelines that have
been developed are: (a) design narratives (Hoadley, 2002; Linn & Hsi, 2000, Mor & Noss, 2008), (b) design principles
(Herrington, 2006; Kali, 2006, 2008; Linn, Bell, & Davis, 2004; Merrill, 2002; van den Akker, 1999), and (c) design patterns
(Goodyear, 2005; Goodyear & Retalis, 2010; Linn & Eylon, 2006; Mor & Winters, 2007; Retalis, Georgiakakis, & Dimitriadis,
2006).
Unfortunately, efforts to translate expert tacit knowledge into practical design guidelines, such as those mentioned above,
have often failed to serve as useful aids for novice designers (McAndrew & Goodyear, 2007). It appears that in order for novices
to take advantage of such guidelines, a pedagogical framework is required. To overcome this challenge, we developed, in a
previous study (Kali & Ronen-Fuhrmann, 2011), a pedagogical model aimed at assisting graduate students in education to design
technology-enhanced curriculum modules, which utilizes a set of design guidelines called the Design Principles Database (DPD)
(Kali 2006; 2008).
The model was developed in a designed-based research methodology with three iterations. In each iteration, a course that
was based on the pedagogical model was implemented with students. Data was collected and analyzed, and design decisions were
made to improve the model for the next iteration (Kali & Ronen-Fuhrmann, 2011). The two authors of this paper served as
teachers in the three implementations (as often the case in design-based research projects).
A Pedagogical Model for Teaching Educational Technology Design
The pedagogical model was embedded in a design course which combined theoretical and practical aspects of educational
technology design. It’s final version includes three main elements, and reflects a unique application and integration of three
frameworks: (a) the well-known Analysis, Design, Development, Implementation, Evaluation (ADDIE) model (Dick, Carey, and
Carey 2001), (b) the studio approach to design teaching (Hoadley & Kim, 2003; Schon, 1983), and (c) the use of the DPD (Kali
2006; 2008) as a framework of design guidelines.
a) The ADDIE model was used as follows: In the Analysis stage students selected contents from a scientific discipline they knew
well and had teaching experience with. They conducted a needs-analysis and a content-analysis to focus on a specific
pedagogical challenge in this area. The Design stage was expanded to include three additional non-linear iterative stages:
Brainstorm Activities, Build Flow, and Design Features. Students brainstormed ideas for activities that would potentially assist
learners1 gain skills and knowledge required for understanding the contents of the module they designed. Then, they built a
flow of activities, and designed features showing in detail how each activity would be viewed by a learner, including a screen
layout, interactive elements, and instructions. Instead of the Develop stage of the ADDIE model, students were required to
design a detailed mockup of their module. For the Evaluation stage students were required to present their modules in class and
provide extensive feedback to each other. Based on this feedback, and additional comments from the instructors, they
conducted a second design cycle.
b) All course meetings took place as ‘design studio’ sessions. Students worked in groups of two or three students. At key stages
each group presented their latest version of the artefact, and received feedback from peers and instructors.
c) The DPD was embedded into students’ work process. This Web-based infrastructure, was designed to support researchers and
technology-based curriculum designers share and synthesize their design knowledge (Kali, 2006; 2008). The shared design
knowledge is accumulated in the DPD in the form of general design principles that are connected to example instantiations in
various pedagogical settings (elementary, secondary and tertiary educational settings, in various subject matters). Students in
the course were required to use the DPD at four points in the process: Analysis, Brainstorm Activities, Build Flow, and Design
Features.
Research Goal
This research builds on two earlier studies that explored student learning with the pedagogical model described above. The first,
(Ronen-Fuhrmann, Kali, & Hoadley, 2008), showed that there is an important added value in engaging graduate students in
designing their own technology-based curriculum modules; while working on their design projects, students became more aware
of gaps between what was defined as their “theoretical epistemologies about learning” (ideas expressed during general discussions
1
We use the term Students to refer to the graduate students who designed the modules, and Learners to refer to potential users of those modules.
1

2. about design, usually representing a socio-constructivist approach) and their “applied epistemologies about learning” (ideas
reflected in artifacts they created, which tended to apply more transmissionist approaches), and were able to reduce these gaps. In
this manner, students’ epistemologies about learning became more coherent – an important outcome for students in education,
whether or not they intend to design curriculum materials. The second study (Kali & Ronen-Fuhrmann, forthcoming), showed that
in the technology-enhanced educational modules they designed, students tended to stay at an abstract level and had a difficulty to
translate their pedagogical rationales and design ideas into concrete features. Thus, by the end of the course many artefacts stayed
at an immature level. As the pedagogical model was refined to attend to student challenges (such as coping with the open-ended
nature of the task and making complex design-decisions with limited peer-feedback), it also better supported them in developing
the skill to concretize their design ideas and translate these ideas into features in mature learning environments. Concretization
was described as a crucial skill for novices to progress in a design knowledge novice-expert continuum. In order to better
understand how expert design knowledge, such as the knowledge in the DPD can assist novice educational technology designers,
the goal of the current research was to explore the relationship between students’ development of concretization skills and their
ability to reduce their epistemological gaps, in the context of the educational technology course explored.
Methodology
We used a case-study methodology to examine students’ learning processes and their development of design knowledge
throughout the course. A collective case-study approach–often referred to as “multiple case study” (Stake, 1994)—was
implemented. This approach is aimed at providing insights into an issue or problem or to refine a theory by exploring similarities
and patterns between several case-studies. In this research, each group of 2-3 students, who worked on one design project during
the course, was defined as a case-study.
The study was conducted with 14 groups (33 graduate students) who participated in three enactments of the Designing
Educational Technologies course. Most students had some experience in teaching or were active science teachers. They had some
experience in designing curricula but most of them had no experience in designing technology-based learning modules. In order to
characterize student learning in each of the iterations student documents were collected at various stages of the course. These
documents included formal design artifacts students were required to create (including their final mockup), as well as informal
notes and sketches students created to discuss their ideas with peers and with us. These artefacts were analyzed using two rubrics;
the first, entitled a Maturity Of Design Artefact (MODA) rubric (Kali & Ronen-Fuhrmann, forthcoming), was used to evaluate the
degree to which students were able to translate their design ideas into design artifacts (Table 1 and Figure 1); the second, entitled
“epistemology rubric” (Table 2) was used to examine the epistemological changes that students went through during the course.
Table 1: Maturity Of Design Artefact (MODA) rubric (Kali & Ronen-Fuhrmann, forthcoming)
Stage in Degree of Maturity Required in the Design Representative Artefacts or Expressions
Design Artefact
Process
Analysis 1. Only general pedagogical ideas about “It’s very important to build activities that would be relevant and interesting to the
the module should be expressed in learner”
this stage
(Except from a discussion of one of the groups in the analysis stage)
Brainstorm 2. A collection of design ideas for the “Learning throughout the whole module should follow a specific inquiry question”.
Activities module. The ideas should only
(Excerpt from a discussion of one of the groups regarding their design of a biology
generally refer to the way a learner
module. They planned to design the activities around an inquiry question but were not
might act in the module.
concerned at this stage about the nature of this question).
Build Flow 3. Graphical or verbal description of a set “First we should show them [the learners] the story about the family tree, then have
of activities, with an indication of which them review the algorithm for scanning the tree, and then use the simulation”
activity should take place before or
(Excerpt from a discussion regarding the design of a module for high-school computer-
after another.
science learners)
Figure 1a shows a sketch of the way students envisioned an activity they planned for a
Design 4. Ideas should be translated to actual
module in genetics. Learners in this activity were required to decide whether they can
Features features and presented in a way that
donate blood to a kid with cystic fibrosis.
shows how a learner might interact
with the module.
(As reference, see Figure 1b showing stage 6 – Mockup iteration 2).
Mockup: 5. Initial learning environment – A
Iteration 1 mockup of the module showing some
of the activities, with instructions for
learners. An initial navigation scheme
should be present.
Figure 1b shows a sketch of one screen (from about 20 screens of the mockup which
Mockup: 6. Mature learning environment – A
were developed by this group with a similar level of detail) of a module designed for
Iteration 2 mockup of the module showing most
teaching logical thinking for middle school math students. The buttons at the top and
of the activities with clear instructions
side of the screen are part of the whole learning environment’s navigating scheme.
for learners. A clear navigating
scheme should be represented.
2

3. (a) Design artefact showing level 4 of maturity (b) Design artefact showing level 6 of maturity
Figure 1. Examples of artefacts showing levels 4 and 6 of maturity in the MODA rubric.
Table 2: Epistemology Rubric
Dimension Low Medium High
Learner activity Passive: e.g. learner Active: e.g. learner
E.g. learner clicks on
The degree to which students expressed ideas that support active reads or views manipulates
links.
engagement of learners within a technology-based learning environment. information. variables
Collaboration Group work is not
Collaboration is
The degree to which the students supported using technology in ways that Individual learning supported by
intrinsic to the activity
enable learners to learn from each other technology
Content accessibility No effort to connect Motivational aspects Motivational aspects
The degree to which students expressed views that support making the contents to student are extrinsic to are intrinsic to
contents of a learning environment accessible to learners. world activities activities
Combining the Two Rubrics to Map Findings
Initial analysis of the findings showed that using each of the rubrics described above, we can distinguish between two patterns of
learning. Using the MODA rubric, we found that one pattern was represented by groups who had difficulties in translating their
design ideas into concrete artefacts (they were slow in
developing concretization skills). The maturity level of
their artifacts at various stages of the design process was
lower than the level required at that stage (see Table 1). On
the other hand there were groups whose pattern did not
show any difficulty with the concretization and were
sometimes even ahead of the required level in the design
process. This enabled us to refer to the dichotomy: Low
versus High pace of concretization skills acquisition.
Using the epistemology rubric, we were able to
clearly distinguish between one pattern, in which groups of
students showed a gap at beginning stages of the semester,
as described above, versus another pattern of those who
showed a coherent epistemology throughout the semester.
This enabled us to refer to the dichotomy: Coherent versus
Non-coherent pattern of group epistemology. Using these
two dichotomies, we developed a four-quadrant matrix
(Figure 2) to map our findings regarding the relations
between maturity/concretization and epistemology.
Figure 2. The four-quadrant concretization/epistemology matrix.
3

4. Outcomes and Discussion
Following an in-depth analysis of each of the 14 case-studies, in which we used both the MODA and the epistemology rubrics
using all the data sources, we were able to map the cases into the four-quadrant matrix. Thus, two of the cases were mapped in
quadrant 1, another two in quadrant 2, three more in quadrant 3, and 7 cases—more than half of the students—were mapped in
quadrant 4. Additionally, the anaysis of each of the cases’ patterns of learning revealled that groups that were classified as
belonging to quadrant 4 significantly reduced their episteomogical gaps throughout the semester, whereas groups that belonged to
quadrant 3 only did so to a small extent. We argue that the high pace of their acquisition of concretization skills (expressed in the
maturity of their artifacts) was an important factor in enabling groups in quadrant 4 to reduce their epistemological gaps. To
support this claim, we describe in detail one case-study representing and illustrating the learning processes of groups that were
classified as belonging to quadrant 4.
Illustrating Learning Processes in Quadrant 4: The case of I,S&E
I,S&E designed a technology-based module designated for high-school computer-science learners. Their module focused on
recursive algorithms for scanning data-structure trees. One of the features they designed, from very early stages of the design
process was an animation that demonstrates a certain algorithm for scanning a tree. Their (potential) learners were required to
solve problems that utilize the demonstrated algorithm.
Analysis of the design artifacts they produced at various stages of the design process using the MODA rubric (left graph
in Figure 3) indicates that this group’s acquisition of concretization skills was of a high pace (high pace was defined as a slope
that is higher or equals to 0.75, where each stage of the design course was numbered consecutively starting with “Analysis=1”).
I,S&E had come up with the idea of the animation as early as the Analysis stage (in which they were still not required to suggest
ideas for activities). They continued at a “normal”, or “required” pace (see dotted line in Figure 3 - left graph) in the Brainstorm
Activities and Build Flow stages. When required to design features, they were still struggling with their flow of activities, but they
gradually progressed until they reached level four of concretization in their final mockup.
The analysis of IS&Es’ learning process using the epistemology rubric (Figure 3, right side) revealed that at the
beginning of the semester, in general discussions about educational technologies, each of these students expressed ideas that we
classified as high level of sophistication with regards to epistemology (level 3 in each of the dimensions of the epistemology
rubric). However, as can be seen in figure 3, there was a large drop at the Analysis stage, with respect to the Learning Activity and
Content Accessibly dimensions, which continued with a drop of the Collaboration dimension at the Build Flow stage. These drops
represent the gap described earlier, between “theoretical” and “applied” epistemologies. Specifically, when IS&E began to design
their animation, it required learners only to passively watch the animation, and there was no attempt to make the contents more
accessible. Collaborative aspects were minimal (a forum was designed for Q&A). Gradually, as this feature was revised following
discussions with peers and instructors, and following the use of the Design Principles Database, this feature became a manipulable
tool, which enabled learners to solve problems by exploring various ways to scan given trees, as well as their own trees. Our
analysis of their final mockup, using the epistemology rubric was as follows: Learning Activity = 3 (learner manipulates
variables); Content Accessibility = 2 (motivational aspects were eventually at an intermediate level); Collaboration = 3 (activities
that required learners to solve problems in tasks created by their peers were designed). Thus - we interpreted their learning process
as representing a major reduction of their groups’ epistemological gap. The dotted line in left graph of Figure 3, which represents
the average between the three dimensions, illustrates this decrease of the epistemological gap.
5 3
Concratization skill Activity
Expected Pace Collboration
Accessibility
4
Epistemology avrage
Level of Concratization
Epistemology
3
2
2
1
0 1
Theory Analysis Brainstorm Flow Features Mockup‐1 Final Theory Analysis Brainstorm Flow Features Mockup‐1 Final
Design course stages Design course stages
Figure 3. The four-quadrant concretization/epistemology matrix.
Our findings illustrate that as students concretized their design ideas and represented them in sequences of activities, they
exposed their pedagogical way of thinking to others. This enabled them to negotiate and reexamine their thinking with peers and
instructors and to compare the design solutions they came up with, with those of experts, as represented in the DPD. The exposure
of ideas, induced by the concretization, brought to identification of gaps between students’ views about how people learn, and
pedagogical notions expressed in artifacts they designed at initial stages of the course (Ronen-Fuhrmann et al., 2008). As students’
artifacts became more concrete, they also represented more advanced pedagogical views of learning. The gaps were reduced as a
result of refinements students made throughout the design process. Thus, in the context of educational technology design, we view
concretization as: (a) an essential skill in the process of gaining design knowledge, and (b) a way to assist students to reflect and
reduce gaps in their understanding about learning theory. Our pedagogical model proved as a productive manner for novices to
use expert design knowledge, in the form of design principles and feature in the DPD to guide their design process.
4

5. References
Alexander, C. (1979). The timeless way of building. New York: Oxford University Press.
Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. Journal of the Learning
Sciences, 13(1), 15-42.
Goodyear, P. (2005). Educational design and networked learning: Patterns, pattern languages and design practice. Australasian
Journal of Educational Technology, 21(1), 82-101.
Goodyear, P & Retalis, S. (2010). Learning, technology and design. In Goodyear, P & Retalis, S (Ed.), Technology-enhanced
learning: design patterns and pattern languages. (pp. 1–27), Rotterdam: Sense.
Herrington, J. A. (2006). Authentic e-learning in higher education: design principles for authentic learning environments and
tasks. In T. C. Reeves & S. Yamashita (Eds.), Proceedings of World Conference on E-Learning in Corporate,
Government, Healthcare, and Higher Education 2006 (pp. 3164-3173). Chesapeake, VA: AACE.
Hoadley, C. (2002). Creating context: Design-based research in creating and understanding CSCL. In G. Stahl (Ed.), Computer
Support for Collaborative Learning 2002 (pp. 453-462). Mahwah, NJ: Lawrence Erlbaum Associates.
Hoadley, C., & Kim, D. (2003). Learning, design and technology: The creation of a design studio for educational innovation. In
A. P. d. Reis & P. Isaías (Eds.), Proceedings of the IADIS international conference e-Society 2003 (pp. 510-519). Lisbon,
Portugal: IADIS Press.
Kali, Y. (2006). Collaborative knowledge building using the Design Principles Database. International Journal of Computer
Support for Collaborative Learning, 1(2), 187-201.
Kali, Y. (2008). The Design Principles Database as means for promoting design-based research. In A. E. Kelly, R. A. Lesh & J. Y.
Baek (Eds.), Handbook of design research methods in education: Innovations in science, technology, engineering, and
mathematics learning and teaching. Mahwah, NJ: Lawrence Erlbaum Associates.
Kali, Y., & Ronen-Fuhrmann, T. (2011). Teaching to design educational technologies. The International Journal of Learning
Technology (IJLT), 36(2), 129-149.
Kali, Y., & Ronen-Fuhrmann, T., (forthcoming). Concretization of design ideas in the context of educational technology design.
The Journal of Instructional Science.
Linn, M. C., Bell, P., & Davis, E. A. (2004). Specific design principles: Elaborating the scaffolded knowledge integration
framework. In M. C. Linn, E. A. Davis & P. Bell (Eds.), Internet Environments for Science Education (pp. 315-340).
Mahwah, NJ: Lawrence Erlbaum Associates.
Linn, M. C., & Eylon, B.-S. (2006). Science education: Integrating views of learning and instruction. In P. A. Alexander & P. H.
Winne (Eds.), Handbook of educational psychology (2nd ed., pp. 511-544). Mahwah, NJ: Lawrence Erlbaum Associates.
Linn, M. C., & Hsi, S. (2000). Computers, Teachers, Peers: Science Learning Partners. Mahwah, NJ: Lawrence Erlbaum
Associates.
McAndrew, P., & Goodyear, P. (2007). Representing practitioner experiences through learning design and patterns. In H.
Beetham & R. Sharpe (Eds.), Rethinking pedagogy for a digital Age: Designing and Delivering E-learning (pp. 92-102).
Oxon: Routledge.
Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43-59.
Mor, Y., & Winters, N. (2007). Design approaches in technology enhanced learning. Interactive Learning Environments, 15, 61-
75.
Mor, Y., & Noss, R. (2008). Programming as mathematical narrative. International Journal of Continuing Engineering Education
and Life-Long Learning (IJCEELL) 18 (2), 214-233.
Retalis, S., Georgiakakis, P., & Dimitriadis, Y. (2006). Eliciting design patterns for e-learning systems. Computer Science
Education, 16(2), 105-118.
Ronen-Fuhrmann, T., Kali, Y., & Hoadley, C. M. (2008). Helping education students understand learning through designing.
Educational Technology, 48(2), 26-33.
Schon, D. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.
Simon, H. A. (1969). The sciences of the artificial. Cambridge, Massachusetts: MIT.
Stake, R. E. (1994). Case studies. In N. K. Densin & Y. S. Lincoln (Eds.), Handbook of Qualitative Research (pp. 236-247).
Thousand Oaks, CA: SAGE Publications.
van den Akker, J. (1999). Principles and methods of development research. In J. v. d. Akker, N. Nieveen, R. M. Branch, K. L.
Gustafson & T. Plomp (Eds.), Design methodology and developmental research in education and training (pp. 1-14).
Dordrecht, the Netherlands: Kluwer.
5