Learning Analytics and Knowledge (LAK) 14 Education Data Sciences
1. Education Data Sciences
Framing Emergent Practices for Analytics
of Learning, Organizations, and Systems
Philip J. Piety
Ed Info Connections
ppiety@edinfoconnections.com
Daniel T. Hickey
Learning Sciences,
School of Education
Indiana University
dthickey@indiana.edu
MJ Bishop
Center for Innovation and
Excellence in Learning & Teaching
University System of Maryland
mjbishop@usmd.edu
1
3. Four Big Ideas
1. Sociotechnical paradigm shift
2. Notion of Education Data Sciences (EDS)
– Academic/Institutional Analysis
– Learning Analytics/Educational Data Mining
– Learning Analytics/Personalization
– Systemic Instructional Improvement
3. Common features across these communities
4. Framework for EDS
3
6. The Educational Data Movement
Understanding how the organizational model of
education is similar to/different from other fields is key
to understanding the educational data movement.
1980 – 1990 - 2000 - 2010
Finance
Manufacturing
Retail
Health Care
Education
6
7. The Educational Data Movement
Understanding how the organizational model of
education is similar to/different from other fields is key
to understanding the educational data movement.
1980 – 1990 - 2000 - 2010
Finance
Manufacturing
Retail
Health Care
Education
7
13. Systemic/Instructional Improvement
“In many ways, the practice of data use is out ahead of research.
Policy and interventions to promote data use far outstrip research
studying the process, context, and consequences of these efforts.
But the fact that there is so much energy promoting data use and
so many districts and schools that are embarking on data use
initiatives means that conditions are ripe for systematic, empirical
study.”
Coburn, Cynthia E., and Erica O. Turner. "Research
on data use: A framework and analysis."
Measurement: Interdisciplinary Research &
Perspective 9.4 (2011): 173-206.
13
19. COMMON FEATURES & FACTORS IN
EDUCATIONAL DATA SCIENCES
A unified perspective for
Educational Data Science
19
20. Five Common Features in EDS
1. Rapidly changing
- Indicative of sociotechnical movement
2. Boundary issues
- All communities touch on other communities
3. Disruption in evidentiary practices
- Big data is disrupting all the sectors
4. Visualization, interpretation, and culture
- Dashboards, representations, APIs, open data
5. Ethics, privacy and governance
- FERPA & COPPA
20
21. Four Factors that Make All
Educational Data Unique
• Human/social creation
–Most requires human manipulation
• Measurement imprecision
–Reliability issues are huge
• Comparability challenges
–Validity creates “wicked problems”
• Fragmentation
–Systems can’t talk to each other
21
24. Recognize Social/Temporal Levels
Timescale
Context
Targeted
Educational
Content
Time
Frame
Format of
Educational
Evidence
Appropriate Formative
Function for Students
Ideal Formative
Functions for Others
Immediate Curricular
Activity
(lesson)
Minutes Event-oriented
observations (Informal
observations of the
enactment of the activity)
Discourse during the
enactment of a particular
activity.
Teacher: Refining discourse during the
enactment of a particular activity.
Close Curricular
Routines
(chapet/unit)
Days Activity-oriented quizzes
(semi-formal classroom
assessments)
Discourse following the
enactment of chapter,
quiz.
Teacher: Refining the specific curricular
routines and providing informal
remediation to students.
Proximal Entire
Curricula
Weeks Curriculum-oriented
exams (Formal classroom
assessments)
Understanding of primary
concepts targeted in
curriculum.
Teacher/curriculum developer: providing
formal remediation and formally refining
curricula.
Distal Regional/Na
tional
Content
Standards
Months Criterion-referenced tests
(external tests aligned to
content standards)
Administrators: Selection of curricula that
have the largest impact on achievement in
broad content domains.
Remote National
Achieve-
ment
Years Norm-referenced external
tests standardized across
years (ex: ITBS, NAEP)
Policy makers: Long-term impact of
policies on broad achievement targets.
24
25. Digital Fluidity
State Longitudinal
Data Systems
District Data
Warehouses and
Teacher Evaluation
Systems
Learning
Tools-Driven
Analytics
School
Teams
School
Leaders
District
Curriculum
District
Leaders
Teacher
Planning
Individual
Students
State
Analysis
25
26. Values in Design
Infrastructure and
Tools Context
Organizational and
Political Context
•routines
•access to data
•leadership
•time
•norms
•power relations
Processes of
data use
•noticing
•interpreting
•constructing
implications
•data components
•linkages
•time span covered
•Infrastructure
boundaries
•data quality
•technology
features
26
28. Four Big Ideas
1. Sociotechnical paradigm shift
2. Notion of Education Data Sciences (EDS)
– Academic/Institutional Analysis
– Learning Analytics/Educational Data Mining
– Learning Analytics/Personalization
– Systemic Instructional Improvement
3. Common features across these communities
4. Framework for EDS
28
29. Education Data Sciences
Framing Emergent Practices for Analytics
of Learning, Organizations, and Systems
Philip J. Piety
Ed Info Connections
ppiety@edinfoconnections.com
Daniel T. Hickey
Learning Sciences,
School of Education
Indiana University
dthickey@indiana.edu
MJ Bishop
Center for Innovation and
Excellence in Learning & Teaching
University System of Maryland
mjbishop@usmd.edu
29
Notes de l'éditeur
Phil is with ed info connections in DCI am with IUMJ Bishop is the director of CIELT for the university system of Maryland
Let me summarize the four main points in our paper and in this talk.Paradigm shift in conception of data. It is a sociotechnological one.I will then argue that this movement involves four distinct fields and that together these fields might be most productive if merged into one broader notion of EDSWe then discuss five common features that unite the currently disparate EDS communities WE then
The scientific community describe by Kuhn are giving way to highly contextual and current paradigms and educational data is advancing more quickly than we know what do with it.This has created fragmentation that limits knowledge sharing and advance with different communities, different names, and different practices.It is not just that technology is driving the society but the society is driving the technology
We Hold These Truths to be Self Evident:Digital tools create vast quantities and categories of data: This datastorm is like nothing we have ever seen beforeQualitative Shift from institutional control of data, think personalized learning, people controlling their own learning. Bring your data with you like in health care.There is just so much more knowledge Th
The points here are twofold:There are parallels with other fields Even though we argue that education is different and has special properties, the movements in broad terms are similar and (very importantly) there are those who expect that education's transformation will parallel other fields. We are not arguing this, but there are parallels and this causes us to re-examine what about education is fundamentally different, unique.Education is both coming to this transformative stage late and with expectations to quickly catch up and so the time other fields have had to develop mature infrastructures was greater than education has leading to stresses in the process.Late to the game but that means that people expect EDS to be like finance and that is not goingThis is big and irrevesable.: OK, now we are in an age of big data and analytics and there is a lot of promise, a lot of potential, and a lot of need. At the same time, the field needs to develop a professional corps to deal with the new opportunity and challenges. It is not the
Now, this plugs my book. In doing so we can say that this book provides a broad argument for some of what to consider in this new paradigm, although there is more of a focus on K-12 than there could be. Specifically, the book:Compares education to other fields Explains using education data is necessary/difficultSynthesizes strands of education/organizational researchUsing design science and learning sciences framework
Traditionally called Institutional Research. Began as extension of analytic work and reporting many IHEs have been doing for years. FOCUSED ON INSTITUTIONAL NEEDSEarly warning systemsRecently, moving forward with new analyses including:Who gets accessHow they are admittedWhere they progress through the systemWhat is occurring in finance, fundraising, administration, grants management, etc.Moverment toward open standards (OAAI
Traditionally called Institutional Research. Began as extension of analytic work and reporting many IHEs have been doing for years. FOCUSED ON INSTITUTIONAL NEEDSEarly warning systemsRecently, moving forward with new analyses including:Who gets accessHow they are admittedWhere they progress through the systemWhat is occurring in finance, fundraising, administration, grants management, etc.Moverment toward open standards (OAAI
Largely from US with direct support No Child Left Behind (NCLB)Test focusedLots of talk about DDDMLimited by its data—lets just call it test driven decision making or even single test driven decision making.Ten years in an millions of dollars later we don’t have very much—But the notion of State Longitudinal Data Systems is big and includes more than test scoresNo conferences, no journals, some special issues but it is quite fragmentedlots of jobs and income for testing companiesSpecial issue of journalNo evidence of systemic validity, lots of evidence of negative consequenceIs getting a lot of attention because they will be using test scores to evaluated teacher performance
Largely from US with direct support No Child Left Behind (NCLB)Test focusedAccumulating research literature across many education communitiesLots of talk about DDDMLimited by its data—lets just call it test driven decision making or even single test driven decision making.Ten years in an millions of dollars later we don’t have very much—But the notion of State Longitudinal Data Systems is big and includes more than test scoresNo conferences, no journals, lots of jobsSpecial issue of journalNo evidence of systemic validity, lots of evidence of negative consequenceIs getting a lot of attention because they will be using test scores to evaluated teacher performance
Both Learning Analytics & Educational Data Mining (EDM) emerged around same time (2005)Have similar roots in digital learning environments. Now share many aspects. Learning Analytics was more rooted in Includes use of data from LMSData Mining is cognitive tutors and videotamesBig focus at NSF right now (Big Data in Education)Example of broad convergence
How many of you think that these should merge?Lot of parallels with Learning Sciences and CSCL
Macro or “learner” levelNow focus on “adaptive” mode considering:Time to revisit ATI, the availability of data may change things. Information-driven personalization
Historically learners differences was cognitive (Hence the early Aptitude Treatment Interaction Work
These four areas are examples of the kinds of emerging data-intensive activities that work across the different parts of the data sciences
1. Rapid changing indicative of sociotechnical movementLike the steam engine and the printing press, lots of innovation across multiple sectors2. Boundary IssuesFor example, flipped classrooms stretch across systemic improvement and personalization, early warning systems stretch across EDM and Institutional Analytics.3. Disruption in evidentiary practicesAcross all four of these communities, there have also been questions about how to use different kinds of information that were previously not available; how to make high quality inferences using the different kinds of evidence in ways appropriate to the context4. Visualization, interpretation, and cultureAcross these four areas we also see the emergence of issues around visualization and interpretation of information. Visualization is usually the lead element in these discussions as different representational schemes. These include “dashboards” that rank and sort individuals and other similar tools used to make sense of the vast amounts of diverse information available in these four communities. 5. Ethics and privacyAcross all four of these areas are issues of ethics and privacy; how the collection and use of the information about learners and teachers can be done responsibly while also safeguarding the privacy of those whose information is captured.
Appreciate the Distinctive Character of Education DataIn important ways this field is like the other areas where data are used in other domains such as health care, finance, and industry. In some other ways, educational data has unique properties. These unique properties include:Human/social creation. Unlike most other fields that use data, much of educational data requires human manipulation, which increases the possibility of error and manipulation. Some have focused on cheating and gaming of the system in the area of tests, but this property is actually much more pervasive and affects areas like special education planning and school improvement plans as well as assessments. Measurement imprecision. Educational data can be rife with issues of precision, especially when assessments of student learning or systemic capabilities are used. Compared to blood pressure readings or financial transactions, educational assessments are noisy. They can be sensitive to student background, instructional techniques, circumstances of testing, and the likeComparability challenges. Comparisons across different areas of educational data can be sometimes impacted by contact variation. For example, different schools are often compared for many different kinds of analyses. However, programmatic variation often occurs from school to school and those programmatic differences may not always be apparent in the data streams. Fragmentation. The world of educational data is fragmented. Many different organizations hold parts of educational information and there are still incomplete and partially adopted technical standards which impacts the ability to link some data without specific extra work. There are a number of efforts to create interoperable data standards. While progress has been made in these areas the road forward will be difficult as the governance of educational data is highly decentralized owing to the US Constitution’s delegation of authority for education to states and across the states there are many different approaches to state-district interactions and almost 20,000 district and charter providers.
The message here is that these six fields (not counting computer science) are all influential in the EDS, but computer science is more so because it is generative with the development of different innovations that will end up rippling through other fields and then to EDS. Roy Pea was behind this thinking.
The concept here is that the same artifacts can serve multiple purposes. This ties back nicely to the paradigm shift because historically data were developed for single purposes and often externally. Now, data is generated from many places and being used for many purposes.
This is really the sociotechnical thesis that argues that the artifacts – any artifacts – are not neutral. They are both adopted and adapted. They shape contexts and so the implications of them are far beyond their singular instrumental use. For example, think of NCLB testing. These are more than simply tests that measured student achievement. They cast a huge shadow over following years and were exploited by conservatives to help break the public’s confidence in public schools
The Gates Data Quality Campaign likes to use the phrase data should be like a flashlight not a hammer. I go further to say it is like lenses, imperfect, but generally better than nothing. Educational data is rarely truth, but often an indication of something to pay attention to.
Let me summarize the four main points in our paper and in this talk.Paradigm shift in conception of data. It is a sociotechnological one.I will then argue that this movement involves four distinct fields and that together these fields might be most productive if merged into one broader notion of EDSWe then discuss five common features that unite the currently disparate EDS communities WE then
Phil is with ed info connections in DCI am with IUMJ Bishop is the director of CIELT for the university system of Maryland