Evaluating Instructional Technology Program
•Télécharger en tant que PPTX, PDF•
1 j'aime•1,571 vues
Graphics to support my presentation to Northern Illinois educators at the NFI Conference in Galena, IL , November 2014
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (18)
Similaire à Evaluating Instructional Technology Program
Similaire à Evaluating Instructional Technology Program (20)
Plus de University of Wisconsin - Superior
Plus de University of Wisconsin - Superior (20)
Dernier
Dernier (20)
Evaluating Instructional Technology Program
- 1. Evaluating Instructional Technology Programs NFI Conference November 6, 2014 Galena, IL
- 2. Program evaluation is essential in today’s high stakes accountability environment
- 3. Education is the only business still debating the usefulness of technology. Schools remain unchanged for the most part, despite numerous reforms and increased investments in computers and networks. • U.S. Secretary of Education Rod Paige, quoted in National Educational Technology Plan, 2004
- 4. Why Program Evaluation • Demonstrate program effectiveness to administration and Board of Education • Improve the implementation and effectiveness of programs • Better manage limited resources • Document program accomplishments • Justify current program funding or support the need for increased levels of funding • Demonstrate positive and negative effects of program participation • Document program development and activities to help ensure successful replication
- 5. Potential Aspects of Instructional Technology Programming Student Achievement Student Growth Student Engagement Student Behavior Cost Effectiveness Instructure Effectiveness Professional Development Hardware Reliability Time on Task
- 6. CoSN’s Elements • Devices • Networks • Systems • IT Spending • Support • Online Learning From CoSN, KPI, 2014.
- 7. From CoSN, KPI, 2014.
- 8. Some Flaws with CoSN KPIs
- 9. Factors for Evaluation from Intel From Intel in partnership with ROCKMAN ET AL, 2007.
- 10. Technology Facets • Hardware • Software • Administrative Software • Service and Support • Technology Staff Development • Integration into the General Instructional Program • Integration into the Special Education Instructional Program • Instructional Technology Courses • Technology Facilities • Internet Presence From Baule, 2001.
- 11. IT Program Evaluation: Following the Correct Steps Determine project goals & objectives to be measured ~ Key Performance Indicators Determine criteria (or norms) to measure success Determine measurement period(s) Determine who will collect the data and how it will be collected Conduct an analysis of the data & present your results
- 12. How to Measure Success • Compare to Benchmarks • Criterion Referenced • Rubrics can work well here • Measure Growth • Norm Referenced • Qualitative Measures
- 13. Evaluation Design Models • Experimental Design (Possible in some cases using control and experimental groups) • Quasi-experimental design • Non-experimental design (Comparison of variables within a single sample) • Qualitative methods (Interviews, observations and descriptive data) From Intel in partnership with ROCKMAN ET AL, 2007.
- 14. For a 1:1 Program • What would you want to measure? • How would you measure each?
- 15. What to Measure What will you measure? How (What is the measurement tool)? When (Annually, Quarterly, etc.) Success will equal what?
- 16. A Student Engagement Example • Goal to increase student engagement through the implementation of 1:1 technology • How will you measure student engagement? • Survey data? • Attendance? • Observation?
- 17. Better Student Engagement “The use of todaysmeet resulted in the participation of 100% of the students. So many students are too shy to share aloud, but a discussion board gives them an opportunity to express themselves without feeling as self-conscious.” “The discussion board then served as a quick-reference. I could quickly and easily see and address any misconceptions and provide reinforcement of how accurate the students were.” Dana Rosenquist, 7th grade language arts teacher
- 18. Example: How to measure? Technology & Learning 1:1 Computing Guidebook, 2005
- 19. Example: How to measure? Technology & Learning 1:1 Computing Guidebook, 2005
- 20. Example: How to measure? Technology & Learning 1:1 Computing Guidebook, 2005
- 21. 1:1 Tablet Program • Each 7th & 8th Grader has a ASUS Droid Tablet • Teachers and students are using Google Apps for Education (GAFE) to produce much of their work • Teachers, parents and students all have access to student work via Schoology, a learning management system or LMS
- 22. Improving Student Motivation and Engagement Success Indicators • A decrease in office referrals, detentions and suspensions • A decrease in the number of days absent • An increase in homework completion Results • Reduced from 138 to 28 • 45.8% decrease in days absent • Completion increased from 59% to 76.2%
- 23. Increase Student Achievement Success Indicators • Increase MAP and ISAT scores • Increase the use of formative assessment via Schoology • Increase RTI interventions for struggling students Results • 77% of students met benchmarks in reading; 68% in math ~ highest rate in district • 100% of 7th grade staff reported an increase • The delivery of accommodations and modifications through the use of the tablet has been more than we could have asked for.
- 24. Reduce Ongoing Instructional Costs Success Indicators • Reduction in the paper budget • Decrease in staff absences • Long term reduction in textbook costs as we move to digital resources Results • Saved 30% of paper budget in first year • Staff absences decreased by about 2/3s • Undetermined at this point
- 26. 1:1 Evaluation Resources • 12 Questions to ask about devices & carts • http://www.nbcusd.org/administration/superintendent/12Que stionsChoosingTabletandLaptopCarts.pdf • Intel/Rockman Evaluation Toolkit • http://www.nbcusd.org/administration/superintendent/FullSe t_of_One-to-One_Evaluation_Toolkit.pdf • Orange County, NC 1:1 Program Evaluation • http://www.orange.k12.nc.us/1to1/Program_Planning_Evalu ation.pdf • Technology & Learning 1:1 Computing Guidebook • http://www.techlearning.com/techlearning/events/techforum 06/LeslieWilson_onetoone2.pdf
- 27. Performance Management Resources • CoSNs KPI • http://cosn.org/key-performance-indicators-kpis • Information Technology Infrastructure Library (Best Practices) • http://www.itil-officialsite.com/ • ISTE Standards and Performance Indicators • http://www.slideshare.net/mictwell/iste-nets-and-performance- indicators-for-teachers • ISTE Essential Conditions • http://www.iste.org/standards/essential-conditions
- 28. Rubric Websites • Rubistar • http://rubistar.4teachers.org • iRubric • http://www.rcampus.com/indexrubric.cfm • Teacher Planet • http://www.sites4teachers.com/ (search for rubric or assessment generators) • How & When to Use Rubrics • http://pareonline.net/getvn.asp?v=7&n=3
- 29. Don’t forget to submit the Administrator Academy homework by November 14, 2014 Questions to me at sbaule@nbcusd.org or 815-765-9431
Notes de l'éditeur
- Pre-post assessment of changes in outcomes. In this design, outcomes are compared before and after an intervention to assess impact. Inferential statistics, including t-tests, ANOVAs, and chi-square tests, are used to determine if pre-post differences are the result of chance. This design may compare pre-post outcomes with a group that is receiving the intervention to a similar group that is not, or participants may be randomly assigned to a treatment and non-treatment condition. A pre-post design can provide rigorous, scientifically-based evidence of impact. • Quasi-experimental comparison to other groups. A quasi-experimental design compares outcomes from two groups that have been matched on a predetermined set of characteristics, such as location, gender distribution, student test scores, or years of teacher experience. This design is not as rigorous as a randomized study, but can identify initial evidence of impact that leads to additional research. • Non-experimental methods. In a non-experimental study, researchers compare variables within a single sample. For instance, researchers may correlate student attitudes toward technology with engagement in classroom activities. Non-experimental studies can identify the kinds of variables that may influence the impact of 1:1 computing programs. They can also confirm the expected paths to impact that are described in the logic model. • Qualitative methods. Qualitative studies tend to be more descriptive in nature, collecting more in-depth data to understand what is happening within specific contexts. The goal of qualitative studies is not so much to generalize to other settings, but rather to gain a rich understanding of what is being studied. The studies use interviews, observations, and other descriptive data to look at the implementation of a program and its impact. Case studies are a common format. While quantitative and experimental studies tend to describe what changed as a consequence of a program, qualitative methods describe the process, or how and why the changes take place in the way that they do.Qualitative is better at expressing what not why