Chounta@paws
•Télécharger en tant que PPTX, PDF•
1 j'aime•205 vues
This document summarizes the research of a postdoctoral researcher studying tutorial dialogue systems. It discusses developing a student model to guide tutorial dialogues according to students' needs. The researcher aims to analyze tutorial dialogues, build a coding scheme to identify levels of support, and integrate the student model into a dialogue system. Challenges include a lack of data per student skill and predicting student knowledge in real-time. Preliminary results applying regression and Bayesian models to existing datasets show Bayesian Knowledge Tracing performs comparably to or better than the regression models with less overfitting. Future work involves implementing the system for real-world use and further refinement.
Recommandé
Recommandé
Contenu connexe
Similaire à Chounta@paws
Similaire à Chounta@paws (20)
Plus de Irene-Angelica Chounta
Plus de Irene-Angelica Chounta (18)
Dernier
Dernier (20)
Chounta@paws
- 1. Hello world!
- 2. My baby steps… PhD: Methods and tools for the evaluation of collaborative learning activities using time series (2014, University of Patras) Greece!!!
- 3. Growing up…
- 4. Here we are! Postdoc @ HCII since April, 2016!
- 5. “Linking Dialogue with Student Modeling to Create an Enhanced Micro-adaptive Tutoring System” University of Pittsburgh (LRDC) & CMU (HCII)
- 6. Dialogue as a means for learning • We aim to develop an adaptive tutorial dialogue system, guided by a student model that will support students in learning physics
- 7. Research questions • RQ: What makes tutorial dialogues successful? Teachers’ adapt the level of discussion to the student’s “zone of proximal development” (Vygotsky)
- 8. Research questions • RQ: What makes tutorial dialogues successful? Teachers’ adapt the level of discussion to the student’s “zone of proximal development” (Vygotsky) • RQ2: How do tutorial dialogues adapt to different student characteristics and prior knowledge? Degree of Teacher Control (van de Pol) Contingent Tutoring (Pino-Pasternak) Cognitive Complexity (Nystrand, Graesser)
- 9. An example would be nice…. RQ: What minimum acceleration must the climber have in order for the rope not to break while she is rappelling down the cliff? (You do not have to come up with a numerical answer. Just solve for "a" without any substitution of numbers.) Chip: a = f / m T: what's f ? Chip : f = mg T: just mg ? how many forces act ont he climber ? Chip : mg + T T: is mg down or up ? Chip : down and T is up T: ok so now solve for a again plugging in T and mg RQ: What minimum acceleration must the climber ……. Dale: 500/55 kg=a m/s^2 T: I don't agree - that's the acceleration that just the pull from the rope would produce (well once the units are straightened out it would be). Think a little more. What is the general rule for finding acceleration from forces? Dale : F/m=a T: and what is the F there? Dale : tension? T: No.. the F in F=ma is always the net force on the object (or group of objects). The vector sum of all the forces on the object. I prefer to say "Sum of F= ma" because it's easier to get it right. So.. if she is sliding down and the rope is just short of breaking, what is the *net* force on her? High performer
- 10. Research Objective To integrate a student model into a tutorial dialogue system in order to guide the dialogue more effectively according to students’ needs “more effectively”
- 11. • Analyze human-to-human tutorial dialogues • Build a coding scheme to operationalize Level of Support The mechanics of tutorial discussions “we need to identify and group the features of dialogic discourse to differentiate the levels of support”
- 12. Method of the study 3 human-to-human dialogues on Physics / 1 per overall learning gains level [low/medium/high] Level of Control [3-step scale] Question Category [18 types] Level of Specificity [3-step scale] Contingent Tutoring [binary] LOS Coding Scheme
- 13. Coding scheme - Application • 4 coders • 3 dialogues [low/medium/high] • 19 tutor turns • Introduction to the coding scheme • Rating handbook & template
- 14. Coding scheme - Results Dimension Fleiss’ Kappa p-value Level of control 0.404 4.13e-11 Question category 0.395 0 Level of specificity 0.141 0.0245 Contingency 0.0764 0.415 Lessons learned: Still unclear how teachers effectively regulate the level of support
- 15. Coding scheme: Lessons learned • Not easy to interpret the goal of the intervention • One intervention, multiple goals • Crucial features: – New content – Feedback Information / Information meant to push student forward – Degree of detail
- 16. Coding scheme Adaptation and Evaluation Before After Level of Control Information related to student’s answer (Backward/Forward) Hints Provision Question Category Question Category Level of Specificity Feedback on Correctness Information related to feedback Contingency Contingency
- 17. Application and Evaluation • 10 human-to-human tutorial dialogues (Physics) – 3 High, 3 Low, 4 Medium • 2 raters per dialogue • The raters were given a tutorial on the coding scheme and detailed instructions Dimensions Cohen's kappa D1. Information Provision (B) 0.871 D1. Information Provision (F) 0.843 D2. Hints Provision 0.843 D3. Feedback on correctness 0.826 D4. Information related to feedback 0.764 How to author dialogues
- 18. So, you’ve coded it… Now what? Remember the Research Objective? “To integrate a student model into a tutorial dialogue system in order to guide the dialogue more effectively according to students’ needs” Use a student model to decide on what step and which piece of dialogue to give next
- 19. The line of reasoning *Jordan, P., Albacete, P., & Katz, S. Exploring Contingent Step Decomposition in a Tutorial Dialogue System. Can you please tell me what is the vertical net force on the arrow? What does that mean with respect to the arrow’s vertical acceleration? RQ: “Suppose the archer is standing at the edge of a high cliff and shoots his arrow perfectly horizontally with an initial velocity of 50 m/s. Neglecting air resistance, how will the vertical velocity of the arrow vary during its flight until it eventually hits the ground? “
- 20. Move through the line of reasoning *Jordan, P., Albacete, P., & Katz, S. Exploring Contingent Step Decomposition in a Tutorial Dialogue System. We need an assessment of the specific skills(KCs) involved in the specific steps (nodes) (Conati, 2010) RQ: “Suppose the archer is standing at the edge of a high cliff and shoots his arrow perfectly horizontally with an initial velocity of 50 m/s. Neglecting air resistance, how will the vertical velocity of the arrow vary during its flight until it eventually hits the ground? “ High Support: I disagree. Look at Newton's second law, Fnet = m*a. Is there a vertical net force acting on the arrow when it leaves the bow? In our case, the Fnet is the force of gravity and is applied on the arrow in the vertical direction. So Fnet = mg and mg = ma Low Support: Recall that you just said that the force of gravity is applied on the arrow. Would this force produce an acceleration on the arrow as it leaves the bow?
- 21. A couple of things to consider… • How to model The Model? • And how to evaluate it?
- 22. How do we model the Model? • Regression models • Bayesian networks • Rule-based models • Time-series models • Overlay models (using NLP)
- 23. What challenges we face • Not enough observations per skill
- 24. What challenges we face • Not enough student input
- 25. What challenges we face • Not enough observations per skill • Not enough student content • Real-time assessment and prediction requires efficiency
- 26. What challenges we face • Not enough observations per skill • Not enough student content • Real-time assessment and prediction requires efficiency
- 27. How do we model the Model? • Regression models • Bayesian model (essentially Bayesian KT) • Rule-based models • Time-series models • Overlay models (using NLP)
- 28. Regression models • Advantages – Good match for our datasets (multiple skills per step) – Easy to implement /maintain (for new skills, scenarios) • Disadvantages – Multicollinearity issues might affect performance (i.e. when the predictors highly correlate with each other) – Overfitting – Never used for real-time predictions before! *Cen, H. (2009). Generalized learning factors analysis: improving cognitive models with machine learning. *MacLellan, C. J., Liu, R., & Koedinger, K. R. (2015). Accounting for Slipping and Other False Negatives in Logistic Models of Student Learning.International Educational Data Mining Society.
- 29. Bayesian models • Advantages – Good way to describe complex mechanisms – Provide better understanding/reasoning for diagnosis • Disadvantages – Need of a pre-defined network for every scenario/activity – Difficult to adapt/maintain (when new skills are added) BUT - Bayesian Knowledge Tracing has been used “online” for inferring mastering of “main” skills per step… *recent work on individualization/introducing student- specific parameters into BKT *Yudelson, M. V., Koedinger, K. R., & Gordon, G. J. (2013, July). Individualized bayesian knowledge tracing models. In International Conference on Artificial Intelligence in Education (pp. 171-180). Springer Berlin Heidelberg *Pardos, Z. A., & Heffernan, N. T. (2010, June). Modeling individualization in a bayesian networks implementation of knowledge tracing. In International Conference on User Modeling, Adaptation, and Personalization (pp. 255-266). Springer Berlin Heidelberg.
- 30. Evaluation • How to identify the best student model (or models)? – Model parameters (when available) – aic, bic – Error rates (RMSE, MAE) and Learning curves (Corbett, et.al.,1995) – Students’ input – observe students’ behavior, interviews etc. – Performance indicators[e.g. time, complexity]/cost
- 31. Work in progress: Early application of modeling techniques on existing data • 3 datasets 1 super set Skills(Dynamics, Revoice, Summary) 217 skills • We applied Regression & BKT on all datasets
- 32. Work in Progress: Results (per study) Dynamics (N=5272) BKT AFM PFM IFM AIC 6816 4946 5029 4946 BIC 7922 6812 6895 7200 RMSE 0.458 0.396 0.401 0.399 Revoice (N=4613) BKT AFM PFM IFM AIC 6621 3954 3908 3913 BIC 7759 5746 5700 6117 RMSE 0.439 0.396 0.391 0.392 Summary (N=6209) BKT AFM PFM IFM AIC 7110 5821 5773 5797 BIC 8456 7861 7813 8302 RMSE 0.425 0.392 0.390 0.391 *The Akaike information criterion (AIC) and the Bayesian information criterion (BIC) are measures of the relative quality of statistical models for a given set of data.
- 33. Work in Progress: Results (super set) Global (N=15644) BKT AFM PFM IFM AIC 7110 13103 13123 13056 BIC 8456 17229 17249 17836 RMSE 0.425 0.42 0.421 0.419 Prediction time (sec) 0.125 0.253 0.223 0.229
- 34. Notes • Overfitting – For regression models: The more skills, the more overfitting – For BKT: practically the same even for large datasets • Error – For regression models the error increases along with the dataset size – For BKT, practically the same (or even better…) – For large datasets, error(regression)≈error(BKT) • Execution Time – Regression models almost double than BKT
- 35. Tutor architecture and Student Model answer UpdateProfile ( student) Compute Next Steps’ Probabilities Tutor Next step Next step Profile(student) Evaluation (answer) Run Rules Student Model Next Steps alternatives
- 36. Future Work • Implementing the infrastructure for real-time studies • Field trials in lab and in classroom • Further refinements… But
- 37. The end. If you want to know more, get in touch! NSH 2602K ichounta@cs.cmu.edu (plus cat pictures!)
Notes de l'éditeur
- I was born and raised in Greece where I studied electrical and computer engineering and did my phD in the university of patras. My phd was on CSCL and in particular I studied the use of time series to model and assess collaboration quality for collaborative learning activities
- After my phD, I moved to germany where I worked as a postdoc at the university of duisburg-essen with prof. Ulrich Hoppe and the Collide group. During my time there, we focused on the use of learning analytics for mirroring and guiding purposes. In particular we studied the use of network analytics to support communication in MOOCs and the use of interaction analysis to support coordination in maker events & hackathons.
- On April 2016 I moved in Pittsburgh and ever since I am a proud postdoctoral researcher at the HCII institute at the Carnegie Mellon University Here I work on a collaborative project between the university of pittsburgh and cmu
- The project’s goal is to link dialogue with student modeling in order to create an enhanced micro-adaptive tutoring system. I am very lucky to work with excellent researchers such as pamela jordan, sandra katz and particia abacete from the learning research and design center of the university of pittsburgh and bruce mc laren from cmu
- In particular we want to use an intelligent tutor to support students who learn physics through dialogue. In contrast to other ITS and cognitive tutors where the student has the initiative and the tutor responds to student input by providing recommendations and feedback, in our system the tutor is responsible for guiding the student through the learning scenarios. So what we practically aim to do is to build a physics story teller that will guide the student through adaptive lines of reasoning. And by “adaptive” we mean that we want to adapt the dialogue both in terms of structure but also in terms of content and depending on the student’s characteristics. The project is building on top of the successful practice of RIMAC that is a physics tutorial dialogue system developed by the university of pittsburgh and has been widely used to support school students. What is particularly interesting and innovative in this project is that -to our knowledge- no other adaptive dialogue tutor has used a student model to guide the student appropriately.
- There are two fundamental research questions that guide our project. The first one is: what makes tutorial dialogues successful. One can argue that learning is a social process and therefore the interaction between the teacher and the student can benefit and scaffold learning. What is also very important to remember here is that the teachers have the ability to assess their students’ performance and knowledge and to adapt the conversation to the right level in order not to set the bar too high but at the same time challenge the student, that is to address the student within her zone of proximal development where actually learning takes place.
- Our second research question focuses on what changes during the adaptation of the tutorial dialogues to different characteristics? What are the specific factors and dialogical features that adapt to match the zone of proximal development A lot of research has been done in this field and some studies pointed out important dimensions that affect tutorial dialogues such as the level of teacher control, the contingent tutoring structure of dialogues and the cognitive complexity
- These are two dialogues from two different students on the same reflection question and with the same tutor. The differences between those dialogues are obvious even by visual inspection. The one on the left is limited, using few words and symbols while the one on the right is extended and elaborate, the teacher uses terminology and provides explicit feedback, challenges the student to try harder etc. For the record, the dialogue on the right belongs to a high performing student
- To come back to our research, our objective is to integrate a student model in order to guide tutorial dialogue more effectively. And this brings us to our second question: what means “more effectively”?
- In order to explore deeper the mechanics of tutorial discussions we analyzed human to human tutorial dialogues in order to identify the features of dialogic discourse that change to support students of different levels. Additionally, we attempted to build a coding scheme to operationalize –using these features- the level of support, i.e. the way we need to address a student during a tutorial dialogue in order to successfully scaffold the learning process.
- As a first attempt, we analyzed 3 h2h dialogues that belong to different learning gains. For this analysis we applied different coding schemes that came up from our literature review. In particular we code our dialogues on four dimensions, the level of control which describes how much new content the teacher provides and how much control the teacher has over the students’ answer (I,e, whether he gives our the answer or tries to elicitate it etc. The level of specificity - this dimension refers to whether the tutor provided specific and focused information to the student. The contigent tutoring dimension - contingent tutoring takes place when the tutor challenges the student with questions or comments that are at or above her potential and non-contingent tutoring happens when the tutor poses questions and tasks that are lower than the student’s potential. The question category – 18 question categories (verification, concept completion etc) to code teacher’s intervention
- This is an example of what our coding scheme looked like
- However the results were discouraging. It was still hard for us to define how teachers effectively adapt the dialogue to the appropriate level for tutoring and in addition, we also discovered several factors that we didn’t take into account in this first approach
- These are some of the lessons learned: The Analysis revealed that experts usually had different opinions about what the goal of the intervention was. In some cases, they even stated that most likely the teacher didn’t have a specific goal but was instead trying to assess the student’s knowledge state. Frequently, the experts stated that a specific intervention served multiple goals. Teachers often provided feedback and guidance in one dialogue turn and this caused mismatches to the codings From the analysis we also identified important features that should be include into the coding schema such as the amount of new content, the provision of feedback and the degree of detail the teacher gives and requires from the answer.
- Provide appropriate, adaptive dialogic support: What is appropriate for whom? guidelines for feedback provision
- So now, we have a rough idea what “effective support” means in terms of tutorial dialogues and what are the features of tutorial discourse that are adapted to match students levels of knowledge and understanding. And we know how *roughly* to write dialogues for students of different knowledge levels. However our research objective is to use a model in order to guide students through the dialogue and what this means is that we need of a student model that will decide the optimal line of reasoning based on student’s performance and skills
- What we see here is an extract of the plan network of a tutorial dialogue for responding to the RQ, taken from Pam’s, Sandy’s and Patty’s recent paper where they argued when it is helpful to expand the dialogue and when not. I use it here to describe the application of the student model (SM). The dialogue always begins with a reflection question that the tutor sets to the student. Depending on the student’s answer we choose how the tutor should respond and where should the tutor move the dialogue choosing from a pool of alternatives
- By initiating a special dialogue depending on the current assessment of the student (Low/Medium/High) – and for that we need an overall assessment of the student profile (i.e. not only when it comes to particular skills but the general picture) –macro-assessment and/or By choosing appropriate nodes, i.e. node 11 instead of node 4 (again depending on the level of the student) – and here we need an assessment for the skills involved in this step – micro-assessment An additional question here is whether these question “levels” also represent concepts of different levels (this is where it links to Patricia’s and Dennis’ hierarchies). This is something that should be answered by the content experts and the teachers. To my understanding, when you decompose a concept you move down to more basic concepts. I do not know whether this is the case here.
- These are the modeling approaches that we have taken into consideration, so far. First, I will go through the ones that I do not consider as appropriate or good enough and my reasonings.
- These are the modeling approaches that we have taken into consideration, so far. First, I will go through the ones that I do not consider as appropriate or good enough and my reasonings.
- .
- The idea here is to propose a general architecture in which any of the models can be inserted and tested. The model here is implemented as a “plug-in” for the tutor (on the right), using as input the student profile, the evaluation of student answer in the current turn, the possible next steps (entry points). Depending on the type of the model, additional information on the entry points might be required (i.e. related KCs etc). As an output, the model will provide an updated profile of the student and the proposed next step.