![WHICH PROBLEM TYPE IS MOST DIFFICULT FOR BEGINNING ALGEBRA STUDENTS? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
Similaire à Continuous Improvement in Teaching and Learning – The Community College Open Learning Initiative
Similaire à Continuous Improvement in Teaching and Learning – The Community College Open Learning Initiative (20)
Dernier
Dernier (20)
Continuous Improvement in Teaching and Learning – The Community College Open Learning Initiative
- 1. Continuous Improvement in Teaching and Learning: Candace Thille, Director OLI
- 3. Algebra Student Results: Story Problems are Easier!
- 4. The Expert’s Blind Spot 0 10 20 30 40 50 60 70 80 90 100 Elementary Teachers Middle School Teachers High School Teachers % Making Correct Ranking (which problems hardest) Nathan, M.J. & Koedinger, K.R. (2000). Teacher’s and researchers beliefs of early algebra development. Journal of Mathematics Education Research, 31(2), 168-190 Expert intuitions about student difficulties are often wrong, systematically biased
- 6. Goal directed practice and targeted feedback
- 8. Developing component skills and knowledge, and synthesizing and applying them appropriately
- 9. Feedback: Changing the Effectiveness of both Learners and Faculty
- 13. “ The Killer App” feedback loops for continuous improvement
- 14. Learning Curve Analysis on Stoichiometry Data
Notes de l'éditeur
- The goal of DFA was to explore students ability to work with different representational formats. What underlying, implicit knowledge is needed to be successful. Data on student thinking is critical to effective instruction … example In March of their first year of Algebra which is most difficult for beginning Algebra students? We surveyed algebra teachers and math education researchers non this question and they thought story problems.
- But that isn’t true. Teachers think their students are going to translate the problem then solve it, but that is not what they do. Almost as if the algebra is a foreign language. Teachers do not remember that they learned.
- The expert’s blind spot was originally illustrated in a study by Ken Koedinger and Mitchell J. Nathan. They used actual student performance in answering a number of high school mathematics problems to establish the varying difficulty of those problems. They then asked three different groups: high school mathematics instructors (the most expert), middle school mathematics instructors, and elementary school mathematics instructors (the least expert) to rank the problems according to the degree of difficulty they believed high school students would have with them. The results shown in this graph show the “experts blind spot.” By far, the high school instructors were the least able to adopt the novice’s perspective. This fact has profound implications for how we prepare instruction. Specifically, it means that, as we become more and more expert in a subject, we can afford to depend less and less on our intuitions about the ways in which novices understand that same knowledge. Some implications Dr. Koedinger has drawn from these studies are: Instruction should explicitly bridge from students' informal, verbal forms of understanding to more powerful formal, symbolic forms of understanding. We have to help students make sense of formalisms and see how they connect to everyday life and common sense. Beware of “expert blind spot” - Educators have biases that can lead to inaccurate conclusions about student difficulty. We need data-driven design
- OLI is an open educational resources project that began in 2002 with a grant from The William and Flora Hewlett Foundation. Like many open educational resources projects, ours makes its courses openly and freely available. However, our courses are not collections of material created by individual faculty to support traditional instruction. While our courses are often used by instructors to support classroom instruction, the most challenging goal of the project is to develop online courses that are the complete “enactment of instruction”. OLI online courses are designed to support an individual learner, who does not have the benefit of an instructor, to learn a subject at the introductory college level. A fundamental goal of the project is to provide open access to these courses and course materials to learners world wide. The goals of providing greater access to education and improving the quality of education are, for us, inextricably tied together. In addition to our mission to improve access and quality, we also have a practical and research driven motivation for making these courses openly and freely available. As you will see in this presentation, our use-driven design process depends on the courses being used by a large number of students with varied background knowledge, relevant skills and future goals. The continued development and evaluation process of OLI courses depends on the growing community of use and research and development.
- <DEMO MinitTutor in Engineering Statics course> This tutor is in a section of the Engineering Statics Course on Summing Force Vectors, and helps students learn how to determine the sum of concurrent forces by resolving them into components. It is intended to be an opportunity for students to do a &quot;self-check&quot; to make sure they understand the concept. The student is presented with a graphical representation of the problem and asked for the answer. If the student is unsure of the procedure for solving the problem, the first hint provides a link which, when clicked, expands the tutor into the various steps needed to solve the problem. The tutor provides scaffolding to support the student to learn the steps of the procedure when needed. The hints and feedback given by the tutor change depending on which part of the exercise the student is attempting. The tutor recognizes when a student has used the scaffolding and hints and when the student gives the correct answer after having used the scaffolding and hints, the tutor suggests the student to try another problem. The graph, the problem statement, hints, feedback and answers are dynamically-generated. The student can work through the tutor multiple times, receiving a different problem each time, until the student is confident that he or she understands the concept and has developed fluency with the procedure. This provides the student with virtually unlimited opportunities for supported practice.
- The mini tutors used throughout OLI courses such as the ones we just saw in the economics course, are built on the 20 years of work that has been done at Carnegie Mellon on cognitive tutors. The mini-tutors in OLI courses are not full cognitive tutors in that they do not have full production rule sets or student models but their behavior is similar to a cognitive tutor for the given problem they are intended to tutor.
- A challenge in chemistry education is that students can be quite proficient at solving the mathematical problems in chemistry textbooks without being able to flexibly apply those tools to novel chemistry phenomenon in which their application would be useful. Our goal in the OLI chemistry course is to bridge f rom mathematical procedures to chemical phenomena through use of the Virtual laboratory and to bridge from chemical phenomena to real world through scenario based learning . Prior to designing our course, we observed that students typically solve traditional chemistry text book problems via a shallow ends-means analysis, by matching the information given in the problem statement with the equations they can pull from the chapter text. <DEMO Virtual Lab and Chemistry Course> To address this and other issues in chemistry education, rather than the traditional approach of teaching the abstract mathematical skills of chemistry out of context, the OLI chemistry course situates the learning in an authentic investigation that addresses real world applications and asks students to approach chemistry problems as a chemist would approach them. The OLI unit on stoichiometry is situated in a real world problem of arsenic contamination of the water supply in Bangladesh. We address the challenge of connecting the mathematical procedure to use in chemistry by replacing traditional textbook problems with problems to be constructed and solved in the virtual chemistry lab. We use the virtual chemistry lab to create learning environments with ill-structured, ambiguous problems that require flexible application of procedural knowledge. The virtual chemistry lab provides opportunities for students to interact with the environment by exploring and manipulating objects, wrestling with questions and designing experiments. This approach promotes deeper learning and lets different students solve problems in different ways. At each step of exploring a solution to the arsenic contamination problem, the student is introduced to and practices one of the target stoichiometric concepts or skills. In the very first step, determining the level of arsenic contamination in a sample of well water, the student uses the Chemistry Virtual Lab to analyze the sample and compare the level of arsenic to the acceptable levels set by the World Health Organization. In order to evaluate the safety of the water, the student must either understand the concept of the “mole” and apply dimensional analysis, composition stoichiometry and solution stoichiometry. If the student does already understand these concepts or can’t demonstrate mastery of these procedures in the context of solving the problem, the student is directed into an instructional sequence that includes demonstrations, worked examples and minitutors.
- The most powerful feature of web-based instruction is that it allows us to embed assessment into every instructional activity and use the data from those embedded assessments to drive powerful feedback loops for continuous evaluation and improvement. By “feedback” we mean information derived from student activities that is used to influence or modify further performance. Feedback to students: In the case of feedback to students, we refer to corrections, suggestions and cues that are tailored to the individual’s current performance and that encourage revision and refinement. Many learning studies have shown that students’ learning improves and their understanding deepens when they are given timely and targeted feedback on their work (Butler & Winne, 1995; NRC, 2004.) Regarding the timing and frequency of feedback, the best learning outcomes occur when feedback comes immediately after the students’ response but not before the student is ready to revise his or her understanding (Corbett & Anderson, 2001). All OLI courses include frequent opportunities for students to assess their own learning and receive immediate context specific feedback. Fortunately, we benefit from inheriting some of the best work done in the area of online tutoring from Carnegie Mellon and University of Pittsburgh faculty. Many OLI courses feature Cognitive Tutors and “mini-tutors” that give students feedback in the problem solving context. A Cognitive Tutor is a computerized learning environment whose design is based on cognitive principles and whose interaction with students is based on that of a (human) tutor - i.e., making comments when the student errs, answering questions about what to do next, and maintaining a low profile when the student is performing well. This approach differs from traditional computer aided instruction in that traditional instruction gives didactic feedback to students on their final answers whereas the Cognitive Tutors and “mini-tutors” provide context specific assistance during the problem solving process. Feedback to Instructors In traditional face-to-face classrooms, the feedback loop between student learning activities and instructor activities is always cycling unmediated by technology. The traditional instructor-student feedback loop works well in classrooms that have an expert teacher and a not unreasonably heterogeneous class. For example, the instructor recognizes a need for additional explanation or practice by listening students' questions or by seeing confused looks on students' faces. In an online environment, this dynamic feedback loop is broken. The richness of the data we are collecting about student use and learning provides an unprecedented opportunity for keeping instructors in tune with the many aspects of students’ learning. Ideally, OLI courses can assist instructors in addressing the challenges they confront as a result of the increasing variability in their students’ background knowledge, relevant skills and future goals. Creating an effective feedback loop to instructors using the OLI courses is our current area of research. Some of the research questions we are exploring in this area are: What measures best capture when students “get it” ? How do we construct an accurate representation of students’ knowledge? What is the most effective way to present this information to instructors? How can we best support instructors in making use of such information?
- This is a brief description of how OLI courses are used in Hybrid mode. The class meets on Mondays and Wednesdays at 10AM. Rather than assigning students homework from a traditional textbook, On Wednesday the instructor assigns the students to work through an OLI module before Sunday evening at 10PM. As the students work in the OLI module, they receive support and feedback from the system and the system records information about their work for the instructor.
- At 10PM on Sunday night, the instructor logs into the system and review the system reports on the students work.
- See notes to slide 9 for more detail on feedback to students and instructors. The most powerful feature of web-based instruction is that it allows us to embed assessment into every instructional activity and use the data from those embedded assessments to drive powerful feedback loops for continuous evaluation and improvement. By “feedback” we mean information derived from student activities that is used to influence or modify further performance. Feedback to Course Designers During the design process and during use, we continuously evaluate the courses by studying data from student use and learning. With the students' permission, we digitally record interaction level detail of student learning activities in all OLI courses and labs. The student learning data is stored in a standard SQL database and can be mined using standard query tools. We analyze the student activity data to learn how students are using the material and the impact of their use patterns on learning outcomes. We also use student log data to evaluate and iteratively refine parts of the course. For example, by examining the data from students working through our Causal and Statistical Reasoning course, we observed that students were engaging in all of the learning activities we had designed but still failed on a target skill of “building causal response structures.” We constructed six additional learning activities and mini-tutors designed to support students to understand and practice this target skill. The following semester we analyzed the student data again to confirm that students were using the new activities and that using the activities resulted in the students’ learning the target skill. In the first run of the statistics course at a community college, we learned that students needed more support in monitoring their own learning. In other words, students must become conscious of their thinking processes. This is called metacognition (Matlin, 1989; Nelson, 1992). One way to do this is to require students to explicitly monitor, evaluate, and reflect on their own performance, and provide them with feedback on these processes. Another is to model the process for students, by showing them how we approach problems, question our strategies, and monitor our performance. In addition, we can provide a series of explicit prompts or questions that ask them to monitor and evaluate their performance. With sufficient practice students should eventually internalize these processes and use them without the need for external aids. Analyzing the data logs to see where student were having challenges and working with faculty from the community college our team at Carnegie Mellon modified the course in the following two ways: We added additional “did I get this activities” to the end of the current set of activities. At the end of the “did I get this?” activity, the student is prompted again with the question “did you get this?” and if their answer is “not yet” then they are presented with additional practice problems. If their answer is “yes” they can move on. We added prompts for the students to evaluate their own competence on the stated learning outcomes for the section of the course just completed. The prompts for the students to self evaluation are stated in student-centered measurable words. Rather than asking students to “rate their understanding of..” we ask, “Evaluate your ability to perform each of the following tasks. Selecting 1 means &quot;I could do this not at all&quot;, selecting 3 means &quot;I could do this while relying on the course material&quot; and selecting 5 means &quot;I could do this perfectly on an exam.” Likewise, the learning outcomes/tasks are stated in student-centered measurable words : “Summarize and describe the distribution of a quantitative variable in context: a) describe the overall pattern, b) describe striking deviations from the pattern. “ Feedback to Learning Science Researchers Some OLI courses also serve as part of the research environment for the Pittsburgh Science of Learning Center (PSLC). Learning researchers affiliated with the PSLC can embed experimental manipulations in OLI courses to test specific learning theories. The researchers then analyze the data collected by the OLI logging service using the PSLC datashop tools. The PSLC datashop has created a number of tools specifically designed to generate meaningful displays of student learning data. Our Learning environments both build on what we know about learning and serve as a platform in which new knowledge about human learning can be developed and further refined. For example, in our environment we can explore the assistance dilemma- How should learning environments balance information or assistance giving and withholding to achieve optimal student learning? How best to achieve this balance remains a fundamental open problem in instructional science. We emphasize the need for further science to yield specific conditions and parameters that indicate when and to what extent to use information giving versus information withholding forms of interaction.
- This is a learning curve graph of the knowledge components in the stoichiometry tutor (OLI Chemistry Course). The Y-axis is the “Assistance Score” (the number of hints the students requests and the number of incorrect answers they select) for the selected knowledge component and the X-axis is the opportunity number. The curve is trending downward which means that the students needed fewer hints and gave fewer incorrect answers as they progressed through the material; shows that learning is occurring. This is a graph for all knowledge components together. The tool also allows us to look at graphs for each individual knowledge component and to identify knowledge components such as “set denominator value of Avagadro’s number” which may show a learning curve that is not trending so neatly downward and may indicate a need for revision of the teaching approach.
- As you know, OLI is a more than a technology. It is a set of strategies for course design, development, delivery and evaluation. What is unique about OLI? Use learning science research results to inform course design Use learning science research methods to unpack the cognitive tasks we want students to learn and to design the instructional intervention Collect data to provide feedback loops to students, instructors, course design team, learning science for continuous evidence based improvement I will demonstrate in a case study what difference this makes – but before I do, Ken, can you say something about the role of the learning science/learning scientist in our design process and why we can expect that to make a difference. We know that the traditional process of having every instructor design their own courses is incredibly inefficient, What may be less obvious is that the traditional process is often ineffectual. So... we pull together a development team and use learning science research results and methods. ---------- A lot of places have teams with instructional designers what is different about having instructional designers and Learning Scientists? Instructional designers operate on a formulaic level as opposed to operating from a level of deep science of learning. Unfortunately many of the formulas are not based on evidence but on opinion.
- A recent example of this hybrid model is in an accelerated learning study we conducted.
- Our next project is to Collaborate with Community Colleges and use the OLI development and evaluation model to address the challenges of post secondary success.
- The OLI course design process puts all of the pieces together. We start with what we know about human learning from the learning sciences. We put together teams of faculty content experts, learning scientists, human computer interaction experts, software engineers, teachers. We do an initial design using the course design triangle. As the course is being used, we collect data and use that data to inform the next iteration of the course design and to refine the underlying learning theory. The use and evaluation supports the progressive refinement of the course and of our understanding of human learning. information and communication technologies can now be used to provide meaningful, actionable feedback to students, instructors, instructional designers, and learning scientists that simply is not available in the traditional teaching as a “ solo sport ” model. Thus far these technologies have not been widely used for such purposes. When they are, the long hoped for transformational impact of technology on education becomes a reality..
- The vision for the PSLC starts with the broad goals of understanding human learning and using that understanding to improve education. More specifically, we are working to fill the chasm that currently exists between learning science research and educational practice. On the practice side, the large international and racial achievement gaps that persist today are indicators that educational practice is not as effective as it could be. On the science side, that costly large-scale randomized control trials have had such a low hit rate is an indication that the science behind most of these trials has not been sound and reliable enough (I’m a believer in RCTs, but we need more science to built up to them!). 2) In other words, while scientists and practitioners have produced many ideas for educational improvement, these ideas tend not to have a firm enough scientific foundation, neither theoretically nor empirically. 3) What we need is smaller more focused basic research studies performed in the field that can produce a practical theory. We need theories and methods that can prune out instructional hypotheses that are either ineffective or unreliable before millions are spent on field trials that find no effect. 4) Toward this end, the PSLC has set out to better “identify the conditions … learning”. We are supporting field-based rigorous studies of instructional principles across multiple domains. We are building upon cognitive and computational theories of learning and making use of advanced technologies to further science and support dissemination. Not long before the PSLC was conceived, a National Research Council book (NRC, 2002) expressed a dire need for “rigorous, sustained scientific research in education”.
- “ changing many variables” is not the same as too noisy or lots of uncontrolled variables -- such variation if in both control & experimental condition, through good randomization, can be dealt with through strong treatments & good size N. Large RFTs in NSF IERI and dept of ed involve treatments that are more than one variable or instructional principle from control -- hard part of a roadmap but are that long blind walk in search of the north pole
- Donald E. Stokes was professor of politics and public affairs in the Woodrow Wilson School of Public and International Affairs at Princeton University. Pasteur's Quadrant Basic Science and Technological Innovation. The synergistic relationship of OLI and PSLC at Carnegie Mellon exemplifies one of the core ideas that it is possible and desirable to think of research and application in OERs as synergistic enterprises. PSLC research in OLI environments explores theoretical issues about learning in contexts that really matter; and, when we work on OLI course design problems we can often frame them so that our work helps make progress on fundamental theoretical issues in learning science. Summary of idea and context: Over fifty years ago, Vannevar Bush released his enormously influential report, Science, the Endless Frontier, which asserted a dichotomy between basic and applied science. This view was at the core of the compact between government and science that led to the golden age of scientific research after World War II--a compact that is currently under severe stress. In this book, Donald Stokes challenges Bush's view and maintains that we can only rebuild the relationship between government and the scientific community when we understand what is wrong with that view. Stokes begins with an analysis of the goals of understanding and use in scientific research. He recasts the widely accepted view of the tension between understanding and use, citing as a model case the fundamental yet use-inspired studies by which Louis Pasteur laid the foundations of microbiology a century ago. Pasteur worked in the era of the &quot;second industrial revolution,&quot; when the relationship between basic science and technological change assumed its modern form. Over subsequent decades, technology has been increasingly science-based. But science has been increasingly technology-based--with the choice of problems and the conduct of research often inspired by societal needs. An example is the work of the quantum-effects physicists who are probing the phenomena revealed by the miniaturization of semiconductors from the time of the transistor's discovery after World War II. On this revised, interactive view of science and technology, Stokes builds a convincing case that by recognizing the importance of use-inspired basic research we can frame a new compact between science and government. His conclusions have major implications for both the scientific and policy communities and will be of great interest to those in the broader public who are troubled by the current role of basic science in American democracy.
- What is Olnet? – open learning network – is a 3 year initiative funded by The William and Flora Hewlett Foundation. It is a partnership between the Open University of the UK and Carnegie Mellon University building on the experience that we each have gained in developing and researching Open Educational Resources (OER). The Hewlett foundation that has invested more than $90m in establishing OER wants to find out more about benefits – what is the evidence? How should people learn with them? What issues does the community still need to solve? OLnet will develop a networked community of researchers and practitioners – offering them support, events and a chance to contribute evidence and questions. OER acts as a unifying theme that will generate sub-issues that need to be considered. Projects will carry out different streams of research looking at such things as design, collaborative learning and the developing world. Funded fellowships will bring in external expertise and offer a program of exchanges and support for research ideas. Researching Open Educational Resources (OER) has proved a complex and interesting challenge – data is hard to grasp not least because of the very openness which makes OER valuable. But also because it is not just the products that matter but also the process and the role OER has in sparking changes. Our aim in OLnet is to start to use the “collective intelligence” in the larger OER community to get more out of the activity by recording and reflecting on it.