Initial plans for a dissertation on creating an assessment toolkit for the purposes of grading college and university students in networked learning settings
2. Period in human
history in which
traditional industry
of the industrial
revolution has been
replaced by an
economy based on
digital information
and processes.
Photo: “Where is the Internet?” Flickr user Lea,
https://www.flickr.com/photos/violinha/
3. People and
information
sources are linked
together in a
global network,
allowing for
unprecedented
and disruptive
forms of
communication. Photo: “Virtual Marketing, Shirley Chan http://09820006s.blogspot.com/
4. Consume but also
actively contribute.
Affiliation Formation
Creative Expression
Collaborative
Problem Solving
Contributing to
Information
Circulation
Photo: “Michael Wesch Keynote” by Flickr user Guilia Forsythe
https://www.flickr.com/photos/gforsythe/
5. The new economy and culture have sparked calls for
educational reform among a variety of stakeholders.
21st Century skills
New Media skills
Digital skills
Photo: “What’s in my bag?” by Flickr user Brandy Shaul, https://www.flickr.com/photos/zoologist/
6. Takes advantage of:
The open, distributed
spaces of the web
The participatory, creative
nature of digital tools
The power (reach) of
networked
communication
The power (potency,
innovation) of networked
thinking
Photo: “Network” by Simon Cockell
https://www.flickr.com/photos/sjcockell/
7. Networked Learning emerged from informal
learning environments, where formal assessments of
student learning are not common and not
necessarily valued.
8. Traditional formalized learning assessments are not
consistent with networked learning pedagogies.
Networked Learning Traditional Assessments
Values learning as a process Values knowledge as acquired
object
Values collaborative work Value individual work
Values remixing, synthesis Values “original” work
9. To create and
validate an
assessment toolkit
for grading student
performance in
networked learning
courses in higher
education
environments.
Aligned
Agile
Valid
Digital
10. Networked
Learning Is
Becoming
More
Popular In
Higher
Education.
Higher
Education
Requires
Formalized
Student
Assessments.
Techniques Or
Protocols Must
Be Developed
That Provide
Formalized
Assessment
Without
Compromising
Pedagogical
Alignment.
11. Networked Learning (noun):
Connectivist-based learning
that takes place on
distributed,
openly networked, &
digital platforms
12. Connectivist-based
learning
that takes
place on
distributed,
openly
networked, &
digital
platforms
Distributed: Consists of more than one
platform, supporting different types of
activities.
Openly networked: Occurs on the
open web, which allows students to
interact with a larger community &
provides more opportunities for
students to make connections between
the “spheres of learning.”
Digital: Internet-based
13. Knowledge: a set of connections formed by
actions and experience distributed across
networks of human and non-human nodes. It is
dynamic, changing quickly based on context. It is
interpreted by the individuals who are making the
connections. It rests in the diversity of opinions.
Learning: the process of making decisions
through the connection of information sources.
(Downes, 2006)
14. Core skills include the ability to:
▪ Choose what to learn & interpret it based on a
lens of shifting reality
▪ Nurture & maintain personal connections for
continuous learning.
▪ See connections across fields, ideas, and
concepts.
Ability to learn through this process is more
critical than current knowledge base.
(Siemens, 2004)
15. Developing &
Maintaining
Personal Learning
Networks
Curating,
Critiquing, &
Organizing Data
and Data Sources
Connecting
Concepts & People
Across Spheres of
Learning
Sharing with
Personal Learning
Network
Transforming
Concepts into a
New Product
17. Participation
Individual Extent to which students contribute to
the social stream
Transactivity Extent to which students refer to and
build on each other’s contributions
Transliteracy Extent to which students move across
types of digital media
Content
Engagement
Sharing Extent to which students share
resources or information
Construction Extent to which students use
information to solve problems within
the same situation in which the
information was presented
Transformation Extent to which students
collaboratively remix or repurpose
information to create new knowledge
18. (Fact: Digital platforms record the actions of its
users and store them in ways amenable to analysis
and visualization)
**Preliminary Research of Social Media Analytics
from a Networked Learning Course provided
potential factors for student assessment.
19. Networked
Learning Principle
Assessment
Principle
Operationalization
Blogs Twitter
Establishing and
Maintaining a
Personal Learning
Network (PLN)
Participation
(Individual)
#Posts,Comments #Tweets
Network Degree Centrality
Curating, Critiquing,
Organize Data and
Data Sources
Content
Engagement
(Sharing)
Keywords
#Links
Keywords
#Links
Connecting or
coordinating people
and concepts over
space, time, and
spheres of learning a
Participation
(Transliteracy) Ratio: Posts, Comments, Tweets
Participation
(Transactivity)
#Links
Classmate Mentions
#Retweets, Mentions, Replies, Links
Network Betweeness Centrality
Content
Engagement
(Construction)
Keywords
Links (Content)
Keywords
Links (Content)
Transforming data
into new products
Content
Engagement
(Transformation)
Content Content
Sharing new product
with PLN
Participation
(Individual) #Posts
#Tweets
Network Degree Centrality
Participation
(Transactivity)
Links (Content)
Classmate Mentions
#Mentions, Replies, Links
Network Betweeness Centrality
20. How can networked learning be documented for the
purposes of student assessment?
How can social media analytics be used to inform
student assessment in networked learning
environments?
How does the use of social media analytics relate to a
qualitative content analysis of the same data?
How can social media analytics be integrated into a
larger student assessment toolkit for higher education
faculty?
22. Exploration of Student Data in Network Learning
Environments
▪ Descriptive Statistics
▪ Quantities of student participation in the form of posts, comments,
retweets, replies, mentions, etc.
▪ Social Network Analysis
▪ Characterization of interactions between students within the community
▪ Content Analysis
▪ Identification of key themes within the discourse
▪ Discourse Analysis
▪ Characterization of interactions between keywords, posts containing
keywords, and students using keywords.
23. Secondary Data Analysis
#ThoughtvectorsMOOC
VCU undergraduate 2014 summer course
95 Student Participants and 30 Open Participants
Students were required to blog daily and comment on each
others’ posts – 98% of activity took place on WordPress
Over 2309 student posts and 419 open participant posts
Twitter discourse was encouraged but with limited student
participation – approximately 4000 tweets total
#ConnectedCoursesMOOC
Digital Media Literacy Research Hub course on teaching
connected courses with over 250+ active participants
Weekly blogging encouraged (over 1880 posts)
Active Twitter discourse among students
25. Networked
Learning Principle
Assessment
Principle
Operationalized (Variables) Measurement Tool
Blog Twitter
Establishing and
Maintaining a
Personal Learning
Network (PLN)
Participation
(Individual)
Number: Posts, Comments Number: Tweets,
People Followed
TAGS-WP/Excel
Curating, Critiquing,
Organize Data and
Data Sources
Content
Engagement
(Sharing)
Keywords
Number: Links
Keywords
Number: Links
KBDex
TAGS-WP/Excel
Product Assessment b
Connecting or
coordinating people
and concepts over
space, time, and
spheres of learning a
Participation
(Transliteracy)
Ratio: Posts, Comments, Tweets TAGS-WP/Excel
Participation
(Transactivity)
Number: Links
Classmate Mentions
Number: Retweets,
Mentions, Replies,
Links
TAGS-WP/
Excel/NodeXL
Content
Engagement
(Construction)
Keywords
Content: Links
Keywords
Content: Links
KBDex
Product Assessment b
Transforming data
into new products
Content
Engagement
(Transformation)
Content Content
KBDex
Product Assessment b
26. Construct Validity:
▪ Qualitative Content Analysis of post and comment text
▪ How do descriptive statistics, social network analysis, and
discourse analysis metrics relate to a thematic analysis of student
posts and comments – with an emphasis on networked learning
principles?
Internal Validity:
▪ Statistical Analysis
▪ Looking for opportunities to assess internal validity through
factor analyses and Chronbach alpha testing
▪ Potential correlation with post-course surveys meant to measure
“connectedness”
▪ Potential correlation with final course grades
27. This is an exploratory process which will need
continued study, replication, and refinement.
Some student data obtained from courses
not consistent with formal higher education
settings.
Results and usefulness of toolkit are
impacted by instructor’s devotion to making
networked learning a priority in the learning
environment.
28.
29. Downes, S. (2006). Learning networks and connective knowledge:
Discussion paper #92. Instructional Technology Forum. Retrieved from:
http://it.coe.uga.edu/itforum/paper92/paper92.html
Oshima, J., Oshima, R., & Matsuzawa, Y. (2012). Knowledge building
discourse explorer: A social network analysis application for knowledge
building discourse. Educational Technology Research and Development,
60(5), 903–921. doi: 10.1007/s11423-012-9265-2
Siemens, G. (2004, December 12). Connectivism: A learning theory for
the digital age. Retrieved from:
http://www.elearnspace.org/Articles/connectivism.htm
Strijbos, J.W.(2011). Assessment of (computer-supported) collaborative
learning. IEEE Transactions on Learning Technologies, 4(1), 59-73.
Notes de l'éditeur
Since the beginning of the
The Internet (specifically Web 2.0) has created a networked culture defined by unprecedented and disruptive communication capabilities
Social media facilitated by the Internet is unique, linking “the many to the many.” It allows for exponential, rapid, global spread of information; television and radio all for unidirectional broadcasting, telephones and emails allow for information exchange in relatively small circles
While it is not technically ubiquitous, it is becoming increasingly more common; 80% of Internet users and 25% of worlds total population participate in social media
In other words, they annotate, curate, combine, imitate, counter, support, criticize, and share what they consume.
Network and Participatory have combined to change the way we understand and assess information.
Networked Data Saturation
Participatory Challenge to traditional hierarchies of who has the right to generate valid knowledge
Cormier (2008): “The traditional method of expert translation of information to knowledge requires time – time for expertise to be brought to bear on new information, time for peer review and validation – that could make the knowledge itself outdated by the time it is verified. Information is coming too fast for our traditional methods of expert verification to adapt.”
Effective members of the networked world understand that they are members of a network who play an active role in transmitting, collecting, and transforming information for the creation of knowledge
The networked world and its participatory culture have inspired new learning theories, pedagogical frameworks, and learning formats that capitalize on the affordances of the web and aim to teach students how to capitalize on them. By affordances I mean that they take advantage of:
Networked Learning was developed in informal learning , where assessments of student learning are not traditionally used or valued; rather, effectiveness of curriculum or instructional design are most likely to be judged by student completion, student satisfaction, or student perceptions of learning.
The purpose of this study is to create and begin to validate an assessment toolkit for the grading of student performance in higher education networked learning courses that:
Is consistent with the theoretical underpinnings of networked learning and collaborative learning assessment
Takes advantage of the technological affordances of the digital platforms on which these courses take place.
Provides student assessment options for higher education faculty that are easy to perform and interpret, as well as pedagogically-aligned, agile, and valid.
There is evidence that these networked learning environments are becoming more popular in higher education settings (CCK08, CCK09, DS016, Thoughtvectors)
Higher education environments require formalized student assessments, but traditional assessments are not consistent with the pedagogical stances or instructional formats of networked learning
Therefore, new assessment toolkits need to be developed to aid in the assessment of student learning in networked learning environments when they are applied in formal higher education settings.
Networked Learning (Networked Learning, Connected Learning, Distributed Learning)
Occurs on distributed openly networked digital platforms
Distributed: More than one platform that supports different types of activities
Openly networked: occurs on the open web, which allows for interaction with a larger audience and more opportunities for students to make connections between the “spheres of learning”
Digital: Internet-based
Knowledge is a set of connections formed by actions and experience distributed across networks of human and non-human nodes. It is dynamic, changing quickly based on context, interpreted by the individuals who are making the connections.
Learning is the ability to make those connections
Note that these principles are offered in a wonky circle – they are nonlinear and iterative.
The CSCL assessment scholars call this concept “Knowledge Creation” a combination of Knowledge Acquisition and Participation metaphors for learning. It takes into account the paradoxical tensions of social versus individual and acquisition versus participation in assessment
Preliminary Research consisted of the literature review and an exploration of social media analytics collected from a VCU networked learning course, #Thoughtvectors, that had already taken place
Social media analytics
Analytics is the discovery and communication of meaningful patterns in data. Especially valuable in areas rich with recorded information, analytics relies on the simultaneous application of statistics, computer programming and operations research to quantify performance. Analytics often favors data visualization to communicate insight.
All digital platforms record the actions of its users. Learning analytics is a field of educational research that capitalizes on the “big data” collected by learning platforms to provide formative feedback for instructors and students.
In the case of #Thoughtvectors, it meant looking at the available TAGS data (Twitter Analytics Google Spreadsheet) and WP (WordPress) Analytics
Tool Kit: A collection of instruments, tools, or protocols
is a library that includes specifications for routines, data structures, object classes and variables. The practice of publishing APIs has allowed user information to be shared from web communities to third parties for the purpose of studying or applying user data.
Application Programming Interface (API)
Wordpress plugins allow for the searching and copying of timestamped and url-identified content from blog posts and comments
Microsoft Excel runs web queries to retrieve text or data from API
Descriptive Statistics and Keyword searches
NodeXL is a Microsoft Excel template that allows for easy Social Network Analysis
KBDeX (Knowledge Building Discourse eXplorer) allows for the analysis of keywords in relationship with each other, posts in which they were found, and students who used them. (Oshima, Oshima, Matsuzawa, 2013).