Dnle final project

Science Team
Designing a New Learning Environment Course
A Plug-in for Science
Enhancing the 21st century education:
The intelligent tutoring spirit
Science team proposal
Piso-MOOC
in for Science-oriented MOOCs (PiSo
Enhancing the 21st century education:
The intelligent tutoring spirit
Science team proposal
16 December 2012
oriented MOOCs (PiSo-MOOC)

Science Team Piso-MOOC
Science team - Designing New Learning Environments
Venture Lab Fall Course 2012

Summary
1. Our Rationale....................................................................................................................................... 5
2. Our System function and features ...................................................................................................... 5
2.1. State of the art ............................................................................................................................. 6
2.2. Technological requirements......................................................................................................... 6
2.3. Learning environment based on scientific research .................................................................... 7
2.4. Higher order learning (implementation of Bloom’s taxonomy)................................................... 7
2.5. Learning sequence (curriculum sequencing)................................................................................ 7
3. Our Target Audience and Learning Conditions ................................................................................... 7
3.1. Target audience............................................................................................................................ 7
3.2. Student with disabilities............................................................................................................... 8
3.3. Student profiles............................................................................................................................ 8
4. Our Business Model/implementation.................................................................................................8
4.1. Required technological and human resources............................................................................. 8
4.2. Business model............................................................................................................................. 8
4.3. Contingency planning................................................................................................................... 9
Appendix I - Prototype description: Bradford assay laboratory exercise ............................................... 9
I.1 Automated tests: when and why? ................................................................................................. 9
I.2 Automated assessment.................................................................................................................... 10
I.2.1 Testing a sequence of tasks....................................................................................................... 10
I.2.2 Learning the right gesture......................................................................................................... 13
I.2.3 Memorizing formulae................................................................................................................ 14
I.3 Evaluation as a teaching methodology ........................................................................................ 15
I.3.1 Memory target: Bradford assay, error bars formula............................................................. 15
I.3.2 Test 1: Drag-and-drop ........................................................................................................... 15
I.3.3 Test 2: MCQ of increasing difficulty ...................................................................................... 16
I.3.4 Test 3: Cloze tests.................................................................................................................. 16
I.4 A need for critical thinking in experiments: the implementation of a virtual laboratory............ 18
I.4.1 Virtual laboratories should include a student model............................................................ 18
I.4.2 Human vs. Automated evaluation of experimental choices ................................................. 18
I.4.3 Collaboration in virtual experiments..................................................................................... 19

I.4.4 Usual learning environment features.................................................................................... 20
Appendix II - Additional resources ........................................................................................................ 21
II.1 Links towards video tutorials ...................................................................................................... 21
II.2 Google presentation link............................................................................................................. 21
II.3 Prezi link ...................................................................................................................................... 21
II.4 Link to the mock up..................................................................................................................... 21
II.5 Bibliography................................................................................................................................. 21

1. Our Rationale
Basic understanding of science and learning to learn have been recognized as the key
competencies for the 21st century (The key competences for European citizens in the knowledge
society).
As described in this document, “competence in science refers to the willingness to use the body of
knowledge and methodology employed to explain the natural world in order to identify questions and
to draw evidence-based conclusions”, while “learning to learn is pursuing and persisting in learning,
organising one’s own learning, including through effective management of time and information, both
individually and in groups”. Moreover, critical thinking and problem solving skills have been labeled
with the highest ranking among the skills with increasing demands in today’s society in both USA and
EU (Emerging Skills and Competences - A transatlantic study). Those skills include effective
reasoning, systems thinking, making judgements and decisions based on the critical analysis of
evidence and identifying and asking significant questions that lead to better solutions.
Training the new generation of experts according to these guidelines is therefore the global
aim of education systems throughout the world. Including scientific training in the curriculum could
provide a basis for the development of key competencies for 21st century professionals. Unfortunately,
many students switch from STEM majors early in their educational career, and this attrition rate hasn’t
changed much in the past few decades. Often, even students studying sciences lack a deep
understanding of the theory behind the techniques that are used in their respective fields, have
problems organizing their learning and show inadequate critical thinking and problem solving skills. A
possible cause for this problem could be insufficient or poorly implemented laboratory exercises.
Instead of deepening the understanding, laboratory exercises encourage repetition, following the
procedures without understanding and offer no challenge or possibility for problem-based inquiry. In
addition, high costs of laboratory equipment and consumables greatly inhibits any innovation in this
type of learning (learning by doing and learning through research).
Our proposal describes a novel learning environment based on scientific evidence offering an
experimental training in a virtual laboratory as an added value to existing campus-based theoretical
courses or recently extremely popular MOOCs (Massive Open Online Courses). Our design would
offer a tool for teachers of laboratory sciences to engage students outside of the laboratory. This
could serve as a preview to a laboratory exercise or as a review of a laboratory exercise. In addition, in
environments that do not have access to a science lab, our design can serve as a way for students to
learn laboratory procedures without a large investment in resources. Moreover, our modular design
allows for high flexibility and evolvability to suit any content and learning objectives, with special
emphasis on the development of deep understanding of material, critical thinking and learning to learn
competencies. Finally, the flexibility of our design could allow for implementation of this learning
environment to any discipline and educational level, allowing the teaching of meta-competences in a
broader sense to a much wider audience.
2. Our System function and features
We believe that laboratory exercises are crucial for the development of scientific thought and
understanding. Our proposal enhances the experience for the students and the instructors despite
increasing demands on instructors due to class size and time limitations. Our design aims to offer
teachers a way to reach larger numbers of students, as well as to engage those students outside of
labs and classrooms.

We base our system on the needs we identified for modern science teaching:
• clear visualizations of laboratory procedures
• easy memorization of lab tasks, supported by audio and video material, and repetition
• critical thinking and understanding of the procedures prior to class, using assessment tools
and exercises
• large-scale and low-cost open and accessible tools
• intuitive, fun, and learner-centered learning environment that engages students in content and
develops crucial skills
2.1. State of the art
The recent success of Massive Open Online Courses has highlighted the importance of
developing online based learning tools. MOOCs enable teachers to efficiently deliver pedagogical
content and assignments. Nevertheless, the efficiency of this learning approach is often questioned,
since it is not clear what the benefits of learning a topic through a MOOC are. In MOOC science
classes, remembering and developing critical thinking are essential. However, to our knowledge, there
is no automated procedure to assess precisely what students remember from the course, and how
well they would perform in a situation where they have to apply their knowledge to real problems, by
having to design experiments by themselves for instance. We will describe a tool that assesses
students’ automatically, with a focus on memory and creativity. In many ways, the tool belongs to a
large family of quiz makers like Hot potatoes, Adobe Captivate, and so many others. The difference
lies in the fact that we add a memory component to it; our philosophy is very close to the one that
underlies cognition-based learning platforms like Memrise or Iknow.jp. The main difference lies in the
focus; Memrise is designed towards learning vocabulary in foreign languages, i.e. word lists. Memrise,
permits little control over the nature of tests, and in most quiz makers, there is no control upon the
sequence and the frequency of testing. We would therefore combine the advantages of classic quiz
makers and Memrise-like tools in one platform to create a powerful online learning tool. Moreover, we
would like this tool to be used in the context of virtual experiments in order to help students acquire
critical thinking over what they do. Several virtual laboratories have been developed in order to
perform low cost experiments, but as of yet, they have had very limited success, partly because the
value they add to the learning process is unknown. In our virtual laboratory, there will be a strong
focus on student evaluation in order to avoid this problem.
To enhance the learning experience for the learner, we plan to integrate knowledge maps and
game elements. Knowledge maps have already been used to guide self-directed learners in Khan
Academy Math lessons , where the implementation was done using Google maps overlays.
Gamification in education is gaining more and more visibility and impacts course design and
curriculum development (see infographic), both in online systems like the Khan Academy, but also in
off-line systems, such as a school-based A Quest to Learn. We hope to provide a learning
environment where the learner has instant feedback of its progress towards mastery and where we
offer reminders and recommendations of suitable activities to enhance both memorization and
understanding of the content. We hope to create an environment where both instructors and students
can set specific learning goals and where every action taken towards a learning goal is recognized by
giving points and badges to motivate the learner. In addition, we plan on integrating existing social
networks to allow users to share their success stories, as well as to compete in challenges with other
users. These elements will hopefully provide an interesting and dynamic social learning environment
that would attract users to use the system more frequently.
2.2. Technological requirements
Our design would be optimized for online learning, requiring a stable Internet connection, but an offline
version would be made available for students with unreliable Internet access. The learning

environment is designed to be used on most common learning devices, including tablets, smartphones
and personal computers. social networking will be integrated to.
2.3. Learning environment based on scientific research
To implement our design, we draw on research in both intelligent tutoring systems and
computer adaptive testing. Intelligent tutoring systems has been a very intensive field of research in
the seventies (Corbett et al. 2008). Though recently research has waned, findings in the field have
been applied to various topics, ranging from programming (Sykes 2007, Dadic 2010) to physics
(Schulze 2000) and medical studies (Suebnukarn 2006, Kazi 2007) . The computer adaptive testing
approach we draw on (Gerson 2005, Georgiadou 2006) follows the philosophy of cognitive tutoring
systems that highlight the importance of the student cognition, including memory and misconceptions
(Aderson 1990, Shute et al.1995, Colace 2006, Feng 2012, Branstadter 2012). Regarding virtual
laboratories, we refer to the inquiry based pedagogy that has been encouraged from the seventies
(Holt et al. 1969, OCDE 2006 Kloser 2012, Brownell 2012). The actual implementation of this
pedagogy with numeric technologies has been described in the virtual laboratories literature (Zumbach
2006, Cobb 2009, Girault 2012).
2.4. Higher order learning (implementation of Bloom’s taxonomy)
Our system will address the following learning objectives:
• remembering will be enforced through cognition-based features, such as memorization
• understanding will be evaluated through quizzes
• applying will be presented through problem-based exercises
• analyzing will be developed through comparing the results of virtual experiments
• evaluating will be trained by choosing and prioritizing the steps of procedures in a virtual
experiment
• creating will be taught through experimental design and planning of learning activities
• learning will globally enhanced through mastering both content-specific knowledge and
acquiring meta-competencies (critical thinking, creativity, learning to learn)
2.5. Learning sequence (curriculum sequencing)
Our learning platform is aimed at fostering memorization and learning through trials and error
via virtual experiments. It will include an authoring tool to enable teachers to design and implement
their own pedagogy. Teachers and self-learners will be responsible for the learning sequence that will
be implemented, as some people prefer to experiment before memorizing whereas others prefer the
other way round. Ultimately, however, learners following the same module will be evaluated in the
same way.
3. Our Target Audience and Learning Conditions
3.1. Target audience
We are targeting students attending MOOCs or institutions where practical laboratory
experience is not available for financial or other reasons. Even in developed countries, the expense of

laboratory exercises is a limiting factor to having each student perform a laboratory exercise in a cost-
efficient manner. Initially, our design would be geared towards university undergraduates in scientific
fields via their instructors who would be able to design an optimal learning environment (by either
customizing their own module or using an “off-the-shelf” version). Our design could also be
implemented in schools or for a general audience, and can be customized for use in other fields that
might benefit from memorization, procedural planning, and automatic evaluation modules.
3.2. Student with disabilities
The complementary use of videos with narratives, subtitles, and supporting material will allow equal
access to students with disabilities, particularly hearing or vision disabilities. The customizable design,
which can be modified to suit individual learning pace, offers a nurturing environment for students with
learning disabilities, including autism spectrum disorders.
3.3. Student profiles
The problem of developmental-appropriateness of our design is solved by the fact that the
instructor will create the content, ensuring optimization for the students’ learning stage. Subtle
modifications can be easily introduced to teach the same content to different education levels or
developmental stages of students. More specifically, in the beginning of each virtual exercise the
student will be presented with an evaluation of previous knowledge and they will be asked to auto-
evaluate their motivation and learning skills. According to both content-related aspects and self-
directed learning level, the student can be offered different content depths and more or less complex
activities, and/or have a learning path tailored towards individual objectives and learning styles.
4. Our Business/Implementation Model
4.1. Required technological and human resources
Development of our design requires curriculum designers, educators and programmers to form a solid
interdisciplinary team. The team should also have an expert who would be offering workshops to
educate instructors on how to use our design and integrate it in the existing framework.
4.2. Business model
Although we would be a non-profit entity, we would use a for-profit model to increase the
business’s self-sufficiency and reduce our need for outside donors. We would offer two tiers of
service: for instructors interested in experimenting with the platform, a limited version would be free;
for instructors interested in using a fully featured version for academic classes, we would offer
renewing yearly and quarterly subscriptions that would track the academic year. Outside
philanthropists could also purchase subscriptions for other groups of users. For example, a charitable
organization might sponsor a subscription to the platform for a school in need.
We hope to initially launch through a crowdfunding source like Kickstarter, which would allow
us to present a prototype (and demonstrate value to potential users) before making large
investments. This funding would allow us to develop the platform and create several stock modules
that would both demonstrate to potential users what the platform is capable of and be products
investors could immediately implement, if desired. As our module bank grew, we would open the
platform up for subscriptions, giving discounts and other rewards to initial investors. One of the goals
would be to foster instructor interaction through the platform to enable the exchange of knowledge
between instructors and promote the diffusion of innovation. Modules created by any instructor would

be public by default, and instructors would have the ability to discuss, adopt, and adapt modules
created by other instructors. Once subscribed, instructors can create as many modules as they need
and for as many different courses as they desire, which would incentivize subscribers to invest in
using the platform. We would also issue regular tutorials for both students and instructors on how to
make their experience using the platform as valuable as possible. We plan to gain additional revenue
from “swag” sales and events between other potential partners in the learning space.
4.3. Contingency planning
Although subscription models can work well once implemented, probably the most difficult part
would be demonstrating the value of the platform to initial subscribers. Many crowdfunded projects fail
because the product was ultimately more expensive to develop than anticipated or because the
business could not transition into a sustainable business model.
To avoid this problem, we plan on building collaborative partnerships with private and public
organizations, including governments and ministries of education interested in enhancing the current
educational system, as well as private organizations that would be interested in increasing their
visibility through backing up projects such as ours.
In the long-run, attaining a critical mass of users to our product would be essential for success. To
build a growing and active community, we will invest a significant amount of time in communications
and public relations, primarily through social networks and targeted advertising, to ensure the influx of
interested instructors and students.
Appendix I - Prototype description: Bradford assay laboratory
exercise
For the purposes of our demonstration, our target audience is college freshmen or sophomores with
some basic chemistry background.We have focused on one specific laboratory procedure, Bradford
protein assay, to illustrate how our proposed system works. Bradford assay is used to measure the
total protein concentration in the solution. In this assay, a dye Coomasie blue shifts from its red form to
the blue form once bound to proteins. The change can be quantified by spectrophotometry - the
increase of absorbance at 595 nm is proportional to the concentration of proteins in the sample.
Bradford assay laboratory exercise offers several higher-order concepts the students should learn,
including experimental design, statistical analysis, standard error, using a reference curve, critical
thinking and collaboration.
I.1 Automated tests: when and why?
Evaluating student skills is one of the main issues associated with online learning platforms. This
evaluation can be either automatic or done by humans. In the latter case, the main problem is
scalability. Human evaluation implies that either an instructor or a peer spend some time evaluating
students outcomes. An instructor can evaluate only a limited number of students and peer evaluation
also has its drawbacks. Although it is possible to assess whether homework has been submitted and
whether evaluation criteria have been followed through peer evaluation, most peers, particularly for
sciences courses, do not have the knowledge required to assess whether a skill has been acquired.
There is therefore a strong need to improve automated evaluation techniques, especially for scientific
skills.
Many skills cannot be assessed through automated evaluation. For instance, there is no automated
way to assess the quality of an argument or of a demonstration. Automated evaluation is therefore
restricted to a subset of skills, and it is mostly based on tests like multiple choice questions, cloze tests,

matching tests, among others. This approach is widely followed in MOOCs like Udacity, edX, and
Coursera.
Here, we will discuss the best uses of automated tests for assessing scientific skills. We will use the
Bradford assay as an example.
A test may have different objectives. Here are three different aims of a test:
• Evaluating the student before the beginning of a course. This approach allows an instructor to
assess whether a student has the prerequisites to attend the course and also provides the
instructor with an opportunity to adapt their teaching methods to the needs of the students.
• Evaluating the student after a course. This type of assessment may help determine whether
specific skills have been acquired.
• Evaluating during the course. This can provide an opportunity to check whether the concepts
or procedures, taught through tutorials for instance, have been correctly understood.
Evaluations can also be viewed as teaching methods as they require memory recollection and
therefore strengthen skill acquisition. This philosophy underlies sites such as Memrise. We will
consider two types of skills: cognitive and procedural. Both types of skills are needed to perform a
Bradford assay. Procedural skills include the correct gestures necessary to perform the assay, like
preparing dilutions and manipulating chemicals. Cognitive skills include mathematical computations,
such as applying calculus and proper formula use.
Figure 1: Left, Bradford Assay video tutorial on Youtube. Right, theoretical step-by-step
explanation (Adobe Captivate, thanks to Dr. Karen Champ)
The first step, before evaluation, is the passive assimilation of knowledge through lectures, video
tutorials or any kind of document. To illustrate this aspect of the work we designed some introductory
lectures to the Bradford Assay explaining its context, lab procedures and the correct use of formulae.
In addition to these explanations, students will go through a video tutorial showing the whole Bradford
assay. After having watched the videos, students can proceed to the evaluation itself.
I.2 Automated assessment
I.2.1 Testing a sequence of tasks
A Bradford assay is a sequence of tasks, beginning with the dilution of the original solution
and ending with the calculation of the protein concentration of an unknown sample.
Before assessing whether students know how to perform each individual task, an instructor may want

to evaluate if the students know the correct sequence of the tasks that are to be performed. One easy
way to assess this automatically is to use a drag-and-drop test. In this case, a name would be given
to each particular task. This would include names such as "dilution", "calculation of error bars", etc.
The tasks are then presented to the student in a random order, and the students are required to “drag
and drop” (i.e. rearrange) the tasks to put them in the correct order. This exercise makes sense in a
number of situations where the sequence is not obvious. However, in the case of the Bradford assay,
this type of assessment is not challenging enough due to the limited number of tasks. Common sense
is sufficient to know that one cannot create the calibration curve before having done the dilutions.
Figure 2:Learning a sequence of tasks: the drag-and-drop matching test
Moreover, the fact that a student can rearrange the tasks in the correct order does not mean
that they would be able to recollect all of these tasks. For example, a student may be able to place
tasks D, C, B, A in the correct order A, B, C, D if asked to do so. However, if asked to perform the
Bradford Assay, the student may forget task C and propose the sequence A, B, D. In this case, tasks
are in the proper order but one is missing. The student is, therefore, not able to perform the Bradford
assay by himself. This test is too easy to be valuable
In comparison, it can be difficult to use a cloze test to assess whether a student knows a specific
sequence. In a cloze test, the student must type exactly (or nearly exactly) what the computer expects.
In some cases, a small range of answers are acceptable. One difficulty is that the designer of the
cloze test cannot easily predict the way in which students will refer to a given task, as well as what
they will consider to be a task to be included versus a subtask that should not be included. Even if the
student knows the total number of tasks he is expected to list, it is unlikely that the student’s answer
will match the computer’s expectations. This is a major drawback of current automated tests. However,
we can use the cloze test as a final test and assume that students can learn by heart our way of
referring at the different steps of the experiment.
One possible approach to avoid this cloze test issue is to rely on a sequence of multiple
choice questions. With simple MCQs, users must choose the correct answer to a question from among
four (or more) different answers. After submitting the answer, the user is usually provided feedback, at
least whether his answer was correct or not. With a sequence of MCQs, the approach is slightly
different. This is because the user needs to answer multiple, interconnected MCQs before submitting
their responses. First MCQ: What is step 1? Second MCQ: What is step 2? Third MCQ: What is step 3?
After selecting an answer for each question, the user submits these for grading. If there is only one
correct answer to a given test, there is a 1/96 chance (one out of 4x4x4=96 ) of randomly getting the
correct answer if there are four answer choices for each included question. The higher the number of
MCQs that are included in the sequence, the lower the probability that the correct answer will be
selected randomly.

Figure 3: Learning a sequence of tasks: the MCQ
This approach also has drawbacks. Since this test is more difficult than the previously
described drag and drop test, it is more reliable for determining whether the student really knows the
sequence. However, since there are many potential incorrect answers, it is hard to infer the associated
misconception from those incorrect answers. Correspondingly, it is difficult to guide accordingly.
Intelligent tutoring systems research indicates that this is a common issue associated with constraint-
based models. With constraint-based models, the answer provided by the student must meet certain
constraints in order to be accepted as correct. An easy way to deal with this issue is to run through the
sequence proposed by the student and stop at the MCQ where the mistake occurred. This is an easy
way for a computer to infer what the corresponding misconception is. This type of issue has been
addressed in previous papers (Viswanathan 2005).
One might argue that few differences exist between a sequence of MCQs and a list of MCQs,
but this is not entirely true. When considering a list of MCQs, the MCQs are independent from one
another. With a sequence of MCQs, all of the MCQs are interconnected. They all have to be
answered correctly for the student to demonstrate that the sequence is known. Moreover, with a
sequence of MCQs, a given question may depend on the answer of the previous MCQ, i.e., the
second MCQ will depend upon the answer to the first MCQ. This approach is referred to as a
Computer Adaptive Test or CAT, and it is widely used to assess the fluency of a student in a language.
Since the use of CAT in experimental sciences is still subject to debate, we will not discuss this further.

Figure 4: Learning a sequence of tasks: the cloze test
I.2.2 Learning the right gesture
The whole Bradford assay can be decomposed into a sequence of individual gestures, and
each gesture can be evaluated individually. Evaluating whether a student performs the right gestures
is easier than evaluating cognitive skills. Even practical tasks such as the dilution or the correct use of
a pipette can be assessed through automated tests. The most rigorous way of evaluating student
gestures would be to have an instructor analyze a video of the student practicing the gesture, but if we
want to stick to automated evaluation, there are a handful of approaches. First, a video tutorial
showing the whole Bradford assay with the different gestures would be shown, with oral explanations
or subtitles. Since this video can be pretty long, from five to ten minutes, there are two ways to test.
Either the test takes place during the tutorial, an approach that is followed by many MOOCs like
Udacity and Coursera, or it takes place after it.
The first kind of test is to show a video gesture without any comment, and to ask through a MCQ
whether the gesture was performed correctly or not, and if not what is wrong. Let us call this kind of
test the wrong gesture analysis. Another way of using the MCQ is to show four different ways of
performing it, and ask the student to assess which video shows the correct gesture. MCQ is perfectly
fit for this approach, since there are not so many different ways of failing a given task. The few most
common mistakes can be identified and students can be trained to detect them through the approach
we described, and therefore avoid them.

Figure 5: MCQ and choice of the right gesture based on videos
We believe this approach can be more efficient to assess student skills than real-world, in-
person courses. In in-person courses, there are usually around five to ten students per instructor, if not
more. It is impossible for an instructor to correct all of the students at the same time; as a result, many
students may perform an incorrect gesture without being noticed by the instructor. Moreover, for some
tasks, even close attention from the instructor is not enough to detect whether the task is done
correctly. A typical example in the case of the Bradford assay is the use of the pipette. When sampling
a liquid with a pipette, one has to avoid sampling too much liquid by pressing the button too much.
This is a very common mistake among students that decreases measurement precision.
It is difficult to assess whether a student is making this mistake just by looking at him. A properly
designed MCQ would be more efficient in this particular case to assess whether the students know
what the right gesture is.
I.2.3 Memorizing formulae
Performing the Bradford assay implies more than just performing the right gesture. Students
have to calculate volumes in order to do the dilution properly. Then they have to create a calibration
curve by measuring the absorbance of samples of known concentrations of proteins. Once the
calibration curve is done, they have to measure the absorbance of the sample of interest and to
deduce its concentration in protein. This last step involves calculus skills since the computation of the
error bars involves the use of a fairly complex formula.
Both MCQ and cloze tests are usually used in this situation. A student may be asked to give a number
through a cloze test. It can be a volume, a concentration, or something else. There are two options.
Either this value is exact, or it is within a range of values. The latter case is not very common, though
possible, as most cloze tests require a perfect match between a student's answer and the expected
answer. The advantage of using cloze tests for numerical values is that it is easy to assess
automatically whether the student got the right answer. This value can be complex enough to prevent
the student from finding it without applying perfectly the formula. Moreover, it can be a way to assess
whether a student can do complex calculations without making mistakes. It is widely used in
international tests of mathematics, computer science, and other similar disciplines.

The drawback of this approach is that it is not easy to assess whether a student reasons correctly. Or
on the other hand, a student may know the formula but make mistakes in the calculation.
It is hard to assess through cloze tests whether a student knows the formula since there are different
ways of writing a formula, from the choice of the symbol to represent a variable to the order of the
variables within the formula. MCQ and drag-and-drop can be an intermediary solution.
I.3 Evaluation as a teaching methodology
We have discussed some different methods of automatic assessment for the sequence of
tasks that are to be performed in a Bradford assay, the right gestures, and the correct use of formulae.
We would like to highlight that evaluation can also be a learning technique. Active recollection of
memories through these tests is a more efficient way of strengthening a memory than reading or
watching tutorials. This is one of the basis of the learning by doing philosophy. Websites like Memrise
and Iknow.jp are based on this principle. In this approach, memories are strengthened through tests of
increasing difficulty (drag-and-drop and MCQ being relatively easy tests and cloze tests being more
difficult).
For instance, here is the formula that describes the error bar associated to the measurements in the
Bradford Assay.
I.3.1 Memory target: Bradford assay, error bars formula
(variables in http://en.wikipedia.org/wiki/Calibration_curve)
Say we want students to memorize this formula. The easiest test is a drag-and-drop, where students
have to place variables in the right place.
I.3.2 Test 1: Drag-and-drop
Here is a view of the drag and drop test.

The number of possible combinations in this particular case is 6! The first issue is that there is no
difference between these two :
This kind of issue is manageable at this stage because there are only two different options, but the
bigger the formula is, the trickier this issue becomes. The cloze test is almost impossible to apply to
literal formulas due to the many different ways of writing it. This is the reason why the validation of a
formula through a cloze test is easier through its numeric application.
I.3.3 Test 2: MCQ of increasing difficulty
The second kind of test relies on MCQs. In this test, students have to find the right place
It is possible to increase the difficulty of the test by increasing the number of distractors in the MCQ
(distractors are wrong answers in a MCQ), or making hard distractors, i.e. wrong answers pretty
similar to the right answer.
I.3.4 Test 3: Cloze tests
Finally, the cloze test is the most reliable test to assess whether the student actually knows the
formula. As we discussed before, the numeric application is the easiest way to check whether the
formula is known. If we want to stick to the literal formula, the order of the formula has to be somehow
imposed to avoid the issue we evoked previously of various correct literal writings.
Another problem is the issue of the definition of variables through letters like a, m, n. If the student is
given both the names of variables and the structure of the formula, then this cloze test is not much
more than a drag-and-drop test from the cognitive point of view. There should be two sequential tests.
In the first one, students have to define the variables that are to be used.
• Cloze test, step 1: letters to represent the variables
The student has to give the different letters of the formula: m, n, k, etc.
• Cloze test, step 2: definition of the variables.

For a given letter, m, the student has to write its meaning: m = slope of the line
• Cloze test, step 3: structure of the formula
Once variables are defined, students have to write the formula. A part of the structure of the formula
can be given to help, like the square root. Students will have to follow an imposed structure of the
formula for the cloze test to work.
• Cloze test, step 4: Numeric application
Once the formula is accepted, they have to apply it, with given values for variables.
These tests, drag-and-drop, MCQ, and cloze tests can be applied to various contexts. We took this
formula as an example because it is one of the most common objects to memorize, but there are
many other objects that can be memorized this way.
These tests are very common among memory orientated learning platforms. The variations
among these platforms like Memrise, Iknow.jp, Mnemosyne, and quizlet, lie in the sequence and the
frequency of these tests. For a given memorization task, students start with easy tests, and then tests
become increasingly difficult as students provide correct answers. Regarding frequency, it is important
to keep in mind that the fact that a student successfully answered a test at time t does not mean that
they can do it correctly a couple of hours later. It is important to make the difference between short-
and long-term memory. For instance, in Memrise, there are time frames for a given object during
which it is not possible to test it. It is only after having successfully answered the most difficult test at
different times (t0=0, t=t0+3h, t=t0+3 days) that the object is considered lodged in long-term memory.
What Memrise does for language learning, we hope to do for science learning. Basically, Memrise
helps students make elementary associations between words, i.e., it helps student learn words lists.
Users can only choose the words lists, but do not have any control upon the choice of the tests, the
sequence nor the frequency and impose tests that absolutely meaningless in science. Our memorizing
tool is therefore a mix between quiz makers, like Hot Potatoes, and memory-based tools like Memrise.
Figure 6: Associating cognition and competence to quizzes

An additional functionality that could be add to this kind of tool is the notion of skill or competence. It is
of course subject to debate, and we will here present a very simple way of looking at it. In our case, all
the tests that we want to design will be associated with a skill. For instance, the formula cloze test is
associated with the skill “knowing how to compute error bars in calibration curves.” Skills can be
themselves nested to higher order skills. In our case, “how to compute error bars in calibration curves”
is nested in the skill “how to perform a bradford assay”. From the practical point of view, the user, a
teacher, should be able to add this notion of skill as a tag of a test, and create a hierarchy among tags
to account for skills being nested in one another. From the student point of view, a list of skills appears
in his dashboard, and the more tests he completes successfully, the more the associated skills are
considered acquired.
I.4 A need for critical thinking in experiments: the implementation of a
virtual laboratory
Although memorizing is an essential part of the learning process, it cannot alone develop creativity
and critical thinking. Tools must be developed to simulate of simple experiments. In the context of
science, these tools are often virtual laboratories.
For instance, the Mastering suite from Pearson offers virtual textbooks in various scientific domains,
ranging from engineering to biology. Some virtual laboratories are associated to these textbooks:
microscopy, molecular biology, systematics, ecology, genetics. They allow students to explore the
world of molecular biology through costless virtual experiments. Such environments can be pretty
realistic. Usually based on Flash, they allow students to manipulate virtual chemicals and biological
material through a drag-and-drop approach. Advanced virtual laboratories allow a certain degree of
freedom, experiments can fail if the experimental choices are inappropriate.
I.4.1 Virtual laboratories should include a student model
However, the main drawback of these virtual laboratories is that there is no associated student model.
Students play around with the buttons, but it is difficult to know what the student is thinking from this
“Push the button” approach. Playing randomly with virtual chemicals does not foster critical thinking
either. Moreover, another issue is the artificial intelligence behind the simulator, which is usually
simple even in the case of advanced virtual laboratories. Creating an authoring tool to enable teachers
to simulate virtual experiments is a challenging task.
I.4.2 Human vs. Automated evaluation of experimental choices
There are two approaches to a robust virtual laboratory: a fully automated virtual laboratory,
and a human-based virtual laboratory. In the human-based approach, students propose an experiment
virtually, a tutor reviews it, and then proposes an outcome that the students have to analyze. This
approach solves the issue of building a complex simulator, but raises the issue of scalability since it
implies finding competent tutors. Under this approach, the virtual laboratory is not much more than a
platform to exchange documents, pdfs, videos, etc.
The main difficulty with completely automated simulators is that replicating the complexity of
experimental environments is resource-intensive, due to the many combinations of chemicals and
experimental conditions (like temperature and time) that students might manipulate. We highlight the
fact that this simulator is not supposed to model biological or chemical processes; rather, it is about
learning and remembering. If the models underlying the simulator are too complex to be understood by

students, then we miss the point. We suggest giving students the freedom to modify only one or two
parameters at a time, such as temperature or choice of chemical. In some ways it would perform like a
MCQ, but instead of having a feedback in the form of “Correct answer” or “Wrong answer”, the
feedback would represent the outcome of the experimental choice in the form of an image, a graph, or
other result. This kind of approach already exists; in Hot Potatoes is an example. Based on these
results, the students would have either to change the experiment condition, either to go further in the
experiment. In the latter case, the next question would depend upon the previous experimental choice.
Every aspect of an experiment could be divided into small parts and taught so that students get
mastery of the whole process. To avoid people answering randomly and not taking the output
seriously, there should be somehow a penalty if they do too many bad experimental choices. A
commonly followed approach is to associate a cost to any experimental choice, whether it is relevant
or not, like in Mastering Biology virtual labs. It would give students a better idea of how a real
experiment works, and would avoid the “randomly push the button” approach that we often see in
online learning. Instructors could also impose budgetary restraints on students to require them to think
about the costs of performing a given experiment.
Finally, unlike in most existing virtual laboratories, student choices would be recorded,
regardless of their relevance. After having completed the virtual experiment and understood all of its
aspects, we could test what they have learnt and memorized from it by following the same memory
based approach we have described previously. This virtual experiment simulator is not much more
than a quiz maker with complex feedback and pre-designed paths to go from one experimental choice
to the other. Of course, there will be an authoring tool associated so that teachers can design any
experiment they like. This authoring tool makes the difference between our virtual lab and existing
stand-alone solutions.
I.4.3 Collaboration in virtual experiments
We have described here a very individualistic approach of learning through the virtual lab. We
could add a social learning aspect to it. Students will work in teams and have to design the experiment
together, within budgetary constraints, as in the individualistic approach. They could gather in online
study rooms like in getstudyroom.com to design the experiment together. Some links towards existing
solutions like Study room should be included in the virtual lab to avoid having to develop features that
already exist elsewhere.

Figure 7: Fostering collaboration: links towards Study room or other existing online solutions
I.4.4 Usual learning environment features
Of course, student activity, whether they work on the memorizing aspect or on the exploration
part of the virtual laboratory, should be recorded and displayed on a dashboard. They could choose
topics to learn or experiments to do, and could also earn medals when learning. Ongoing topics and
experiments are stored in the student’s library. We could also think about memorizing or experimental
design competitions like in Memrise. Basic social network tools like a mailbox, finding friend, and
eventually tutors would also be available. Since these aspects of the platform are not specific to our
project, we only mention these as future possibilities. They are mentioned here because some of them
are in the mock up.
Figure 8: Dashboard

Appendix II - Additional resources
II.1 Links towards video tutorials
Serial dilution in Bradford Assay: http://cfscience.net/DNLE/Serial_Dilutions/Serial_Dilutions.htm
Providing the context: http://cfscience.net/DNLE/Protein_PreLab/Protein_PreLab.htm
Bradford assay procedure: http://cfscience.net/DNLE/Bradford_Assay/Bradford_Assay.htm
Video tutorial showing the real Bradford assay: http://www.youtube.com/watch?v=5It_AEOTYoM
II.2 Google presentation link
https://docs.google.com/presentation/d/1JAODI4rssuSR9VXw-1NUYHiCCsUMo-
Zw_kOeoHkFn8E/edit
II.3 Prezi link
http://prezi.com/ohf_brwdy54g/science-team-final-project-for-
dnle/?auth_key=77084dcb9f84fa7f6071f63939cca38ed5d06c8f&kw=view-ohf_brwdy54g&rc=ref-
17313360
II.4 Link to the mock up
Our project Mock up
https://www.dropbox.com/sh/mdbhboj0db16q6b/VVcAMcAncB
II.5 Bibliography
European Commision, 2012, COMMISSION STAFF WORKING DOCUMENT ;Assessment of Key Competences
in initial education and training: Policy Guidance Accompanying the document Communication from the
Commission Rethinking Education: Investing in skills for better socio-economic outcomes The key competences
for European citizens in the knowledge society
Google developers. (2012, September 27). Retrieved December 15, 2012, from Java Scripts Overlays:
https://docs.google.com/document/d/1r8cJ1ponVvuLdaJcfwbR5d3GlmlbHEnVpwYw0grpE5M/edit#
Hanne Shapiro,John René Keller Lauritzen, Pat Irving, 2011.Emerging Skills and Competences- A transatlantic
study , Emerging Skills and Competences - A transatlantic study
Khan Academy. (n.d.). Retrieved december 15, 2012, from Khan Academy: http://www.khanacademy.org/
Anderson Cognitive modeling and intelligent tutoring Artificial Intelligence 42 (1)p. 7-49 (1990)
http://act-r.psy.cmu.edu/papers/119/CogMod_IntTut.pdf
Kristina Brandstädter, Ute Harms, Jörg Großschedl Assessing System Thinking Through Different Concept-
Mapping Practices International Journal of Science Education 34 (14) p. 2147-2170 (2012)
http://dx.doi.org/10.1080/09500693.2012.716549
Brownell et al. Undergraduate Biology Lab Courses: Comparing the Impact of Traditionally Based “Cookbook”
and Authentic Research-Based Courses on Student Lab Experiences. Journal of College Science Teaching 41:4
(2012)

Cobb et al. The Learning Gains and Student Perceptions of a Second Life Virtual Lab. Bioscience Education 13:1
(2009)
Corbett, et al. Intelligent Tutoring Systems Springer Berlin Heidelberg5091 (6) p. 31-40-40 (2008)
http://www.springerlink.com/content/b4197184w8667748/
Dadić. Intelligent Tutoring System for Learning Programming. Intelligent Tutoring Systems in ELearning
Environments Design Implementation and Evaluation p. 166 (2010)
http://www.igi-global.com/viewtitlesample.aspx?id=45547
Feng et al. Student modeling in an intelligent tutoring system. Intelligent Tutoring Systems in ELearning
Environments Design Implementation and Evaluation p. 208 (2010)
http://nth.wpi.edu/pubs_and_grants/papers/2008/Feng_Stude...
Gershon Computer adaptive testing. Journal Of Applied Measurement 6 (1) p. 109-127 (2005)
http://www.ncbi.nlm.nih.gov/pubmed/15701948
Elisabeth Georgiadou, Evangelos Triantafillou, Anastasios A Economides Evaluation parameters for computer-
adaptive testing. British Journal of Educational Technology 37 (2) p. 261-278(2006)
http://www.blackwell-synergy.com/doi/abs/10.1111/j.1467-8...
Girault Characterizing the Experimental Procedure in Science Laboratories: A preliminary step towards students
experimental design. International Journal of Science Education 34:825-854 (2012)
Holt, C. E., Abramoff, P., Wilcox, L. V.,& Abell, D. L. (1969). Investigative laboratory programs in biology: A
position paper of the commission undergraduate education in the biological sciences. BioScience, 19, 1104–1107.
H Kazi A Diverse and Robust Tutoring System for Medical Problem-Based Learning Supporting Learning Flow
through Integrative Technologies162 p. 659-660 (2007)
Kloser et al. Integrating Teaching and Research in Undergraduate Biology Laboratory Education PloS Biology
9:11 (2011)
Model et al. Modell, H. I., & Michael, J. A. Promoting active learning in the life sciences classroom: Defining the
issues. Annals of the New York Academy of Sciences, 701, 1–7. (1993).
OECD (Organisation for Economic Co-operation and Development). (2006). Evolution of student interest in
science and technology studies (Policy Report). Paris: Author.
Schulze Andes: An active learning intelligent tutoring system for Newtonian physics THEMES in Education 1 (2) p.
115–136 (2000) http://www.public.asu.edu/~kvanlehn/Stringent/PDF/00ANDES...
Shute et al. Intelligent tutoring systems: Past, present and future. Handbook of Research on Educational
Communications and Technology. Scholastic Publications. (1995)
Suebnukarn A Bayesian approach to generating tutorial hints in a collaborative medical
problem-based learning system Artificial Intelligence in Medicine (2006) 38, 5-2
Sykes A Prototype for an Intelligent Tutoring System for Students Learning to Program in Java
Advanced Technology for Learning 1 (1) http://www.actapress.com/PaperInfo.aspx?paperId=20654 (2004)

Viswanathan A Comparison of Model-Tracing and Constraint-Based Intelligent Tutoring Paradigms The
International Journal of Artificial Intelligence in Education 15:1 (2005)
Zumbach et al. Learning Life Sciences: Design and Development of a Virtual Molecular
Biology Learning Lab. Journal of Computers in Mathematics and Science Teaching, 25, 281-
300. (2006)

Dnle final project

Recommandé

Recommandé

Contenu connexe

Similaire à Dnle final project

Similaire à Dnle final project (20)

Plus de Matthieu Cisel

Plus de Matthieu Cisel (18)

Dernier

Dernier (20)

Dnle final project