Unraveling Multimodality with Large Language Models.pdf
Applying cognitive science to education
1. ECC2012-13
Cognitive studies meet education
A NEW FIELD OF APPLIED RESEARCH TO EDUCATION AND ITS
PERIMETER
Its meaning
Its reasons
A BIT OF HISTORY
What’s new
2. ECC2012-13
ECC2012
Birth of a new field
Policy-making needs
science. (Alberts 2010)
2009-
2010
2002- 2000-
2006 2005 2008 2011
3. ECC2012-13
ECC2012
Birth of a new field
Mind, Brain, Education USA: Internationally:
Harvard 1999-2006 OECD-CERI:
2003 Conference at Graduate School Brain and Learning
the Accademia of Education Project (Della Chiesa)
Pontificia (Gardner &
Fischer)
2006-2012 Annual MBE Teachings in Battro (Argentina),
Summer School, Graduate Koizumi (Japan),
Biannual Conference Schools in the Goswami (UK), Spitzer
USA,(Texas (Germany), Léna
Austin, …) (France), Dehaene
2007 MBE International (France), Wolf (USA),
Society and Journal McDonnel Geake (Australia),
Foundation Strauss (Israel), …
(Bruer)
4. ECC2012-13
ECC2012
Birth of a new field
Educational Educational Neuroscience & Education USA
Neuroscience UK
Neuroscience Vanderbilt University
Cambridge (McCandliss)
Trends in University
Graduate School Oregon Institute of
Neuroscience (Goswami)
and education Neuroscience (Posner)
Journal (Spitzer) Center for
Educational
Neuroscience –
Birbeck, IOE, UCL
(Bell, Thomas,
Butterworth, …)
5. ECC2012-13
ECC2012
Birth of a new field
Neuroeducation UK Neuroeducation Neuroeducation USA
Europe:
2000-2008 Seminar Dana Foundation Art
TLRP EARLI Sig 22 & Brain Initiative/
Neuroeducation
(Biannual Neuroeducation
2011 Royal Conferences)
Institution Johns Hopkins
University Graduate
Bristol (Howard- School (Hardiman)*
Jones), UCL Neuroeducation
London (Frith, Quebec: NY University
Blakemore, (Brabeck)
Butterworth, …), Neuroeducation
Cambridge Quebec –
Conferences, SfN – Neuroeducation
(Goswami), … Journal Summit (Carew)
6. ECC2012-13
ECC2012
Birth of a new field
Science of Learning New Learning Learning
Centers Program NSF Sciences Sciences
USA
Istitute for ISLI
Life Center learning and International
(Bransford, Kuhl, …) brain sciences, Society of the
Washington learning
(Kuhl, Meltzoff) sciences
(Sawyer)
…
7. ECC2012-13
ECC2012
Perimeter
Biology Cognitive Education Technology
science
Neuroscience Cognitive Educational Computer
psychology, psychology science
evolutionary
psychology
Cognitive Information Social sciences Robotics
neuroscience sciences
Genetics Developmental Learning and Emerging
neuroscience psychology transfer studies technologies
Social Instructional
psychology, design, wisdom
anthropology of practice
8. ECC2012-13
ECC2012
¤ (Fischer et al. 2007)
¤ Human beings are unique in their ability to learn through
schooling and diverse kinds of cultural instruction.
¤ Education plays a key role in cultural transformations: it
allows members of a society, the young in particular, to
efficiently acquire an ever-evolving body of knowledge and
skills that took thousands of years to invent.
¤ It is time for education, biology, and cognitive science to join
together to create a new science and practice of learning
and development. The remarkable new tools of biology and
cognitive science open vast possibilities for this emerging
field.
9. ECC2012-13
ECC2012
¤ (Meltzoff et al. 2009)
¤ Homo sapiens is also the only species that has developed
formal ways to enhance learning: teachers, schools, and
curricula.
¤ Neuroscientists are beginning to understand the brain
mechanisms underlying learning and how shared brain
systems for perception and action support social learning.
Machine learning algorithms are being developed that
allow robots and computers to learn autonomously. New
insights from many different fields are converging to create a
new science of learning that may transform educational
practices.
10. ECC2012-13
ECC2012
¤ Learning sciences is an interdisciplinary field that studies
teaching and learning.
¤ Learning scientists study learning in a variety of settings,
including not only the more formal learning of school
classrooms but also the informal learning that takes place at
home, on the job, and among peers. The goal of the
learning sciences is to better understand the cognitive and
social processes that result in the most effective learning,
and to use this knowledge to redesign classrooms and other
learning environments so that people learn more deeply
and more effectively.
¤ The sciences of learning include cognitive science,
educational psychology, computer science, anthropology,
sociology, information sciences, neurosciences, education,
design studies, instructional design, and other fields.
¤ (Sawyer 2008, p. xi)
11. ECC2012-13
ECC2012
¤ (Bransford et al 2000, p. 4)
¤ Research from cognitive psychology has increased understanding of the
nature of competent performance and the principles of knowledge
organization that underlie people's abilities to solve problems in a wide
variety of areas
¤ Developmental researchers have shown that young children understand a
great deal about basic principles of biology and physical causality, about
number, narrative, and personal intent,
¤ Research on learning and transfer has uncovered important principles for
structuring learning experiences that enable people to use what they have
learned in new settings.
¤ Work in social psychology, cognitive psychology, and anthropology is
making clear that all learning takes place in settings that have particular
sets of cultural and social norms and expectations and that these settings
influence learning and transfer in powerful ways.
¤ Neuroscience is beginning to provide evidence for many principles of
learning that have emerged from laboratory research, and it is showing
how learning changes the physical structure of the brain and, with it, the
functional organization of the brain.
¤ Emerging technologies are leading to the development of many new
opportunities to guide and enhance learning that were unimagined even
a few years ago.
12. ECC2012-13
ECC2012
Perimeter
Knowledge & Design For better Everywhere
learning
Understanding Design better Learn more Learning that
of cognitive environments deeply takes place at
processes for learning home
Understanding Learn more At school
of social effectively
processes
Underlying On the job
learning
Among peers
13. ECC2012-13
ECC2012
¤ Everyone designs who devises courses of action aimed at
changing existing situations into preferred ones. The
intellectual activity that produces material artifacts is no
different fundamentally from the one that prescribes
remedies for a sick patient… The natural sciences are
concerned with how things are …. Design, on the other
hand, is concerned with how things ought to be, with
devising artifacts to attain goals.
¤ (Simon 1988, p. 67)
14. ECC2012-13
ECC2012
Reasons
1. Learning as a
natural,
pervasive
cognitive
function.
15. ECC2012-13
ECC2012
¤ (Bransford et al 2000)
¤ Learning is a basic, adaptive function of humans.
¤ More than any other species, people are designed to be
flexible learners and active agents in acquiring knowledge
and skills.
¤ Much of what people learn occurs without formal instruction,
but highly systematic and organized information systems—
reading, mathematics, the sciences, literature, and the
history of a society—require formal training, usually in
schools.
16. ECC2012-13
Reasons
Learning and
teaching as both
natural and cultural
Humans have
created a special
technology for
promoting learning
when learning does
not come naturally
17. ECC2012-13
ECC2012
¤ (Pinker 2002, p. 222)
¤ Education is neither writing on a blank slate nor allowing a child's
nobility to flower.
¤ Rather education is a technology that tries to make up for what the
human mind is innately bad at.
¤ Children don't have to go to school to learn to walk, talk, recognize
objects, or remember the personalities of their friends even though
these tasks are much harder than reading, adding, or remembering
dates in history...
¤ Because much of the content of education is not cognitively
natural, the process of mastering it may not always be easy or
pleasant, notwithstanding the mantra that learning is fun... they are
not necessarily motivated in their cognitive faculties to unnatural
tasks like formal mathematics.
18. ECC2012-13
ECC2012
Reasons
2. Societal
transformations
have occurred
that pose new
problems to
education
e.g. information
revolution
19. ECC2012-13
ECC2012
Reasons
• Preoccupation about
international competition to
standards
• Crisis of ideologies
• From standards to
“what works policies”
• and Evidence-Based
Education approaches
20. ECC2012-13
ECC2012
¤ (US Department of Education 1983)
¤ sIf an unfriendly foreign power had attempted to impose on
America the mediocre educational performance that exists
today, we might well have viewed it as an act of war. As it
stands, we have allowed this to happen to ourselves. We
have even squandered the gains in student achievement
made in the wake of the Sputnik challenge. Moreover, we
have dismantled essential support systems which helped
make those gains possible. We have, in effect, been
committing an act of unthinking, unilateral educational
disarmament.
21. ECC2012-13
ECC2012
¤ (US Department of Education 2008)
¤ If we were “at risk” in 1983, we are at even greater risk now.
The rising demands of our global economy, together with
demographic shifts, require that we educate more students
to higher levels than ever before. Yet, our education system
is not keeping pace with these growing demands”… “The
pace of change in the global economy poses an already
enormous and growing challenge for educators. As
Microsoft founder Bill Gates has said, “You need to
understand things in order to invent beyond them.
22. ECC2012-13
¤ (Sawyer 2006, p. 1-2)
¤ In the knowledge economy memorization of facts and
procedures is not enough for success.
¤ Educated graduates need a deep conceptual
understanding of complex concepts, and the ability to work
with them creatively to generate new ideas, new products,
and new knowledge.
¤ They need to be able to critically evaluate what they read,
to be capable of express themselves clearly, … to learn
integrated and usable knowledge, … to take responsibility
for their continuing, lifelong learning.
23. ECC2012-13
ECC2012
Reasons
3. Bounded rationality,
cognitive biases and the
fallacies of intuition
26. ECC2012-13
¤ (Simon 1988, p. 116)
¤ We have new top-down research techniques that enable us
to observe and model the step-by-step progress of thinking
and learning with shorter and shorter steps, even on the
scale of seconds and fractions of a second.
¤ We have new bottom-up research techniques, such as
magnetic resonance imaging and single-cell recording, that
enable us to study the localization of the neural processes
that occur during thought and learning and to study the
chemistry of neurons.
¤ With the help of these new tools, we are even beginning to
forge links between bottom-up and top-down advances,
gaining glimpses of the neurologic bases for human symbolic
processes.
27. ECC2012-13
¤ Just as the revolution in molecular biology changed the
whole face of medicine by providing both new
understanding of physiological processes and new means of
intervention when the processes are out of kilter, so the
revolution in the study of the mind, usually called the
cognitive revolution, is allowing us to enter a new era of
human learning and teaching.
¤ This era does not reject the practical knowledge that has
built up over millennia but greatly improves and enriches it.
Good teachers and good learners may be born, but they
cannot reach their potential, or anything close to it, without
a deep understanding of the learning processes and how to
enhance them. We are becoming more and more able to
provide that understanding.
29. ECC2012-13
Cognitive studies meet education
A NEW FIELD OF APPLIED RESEARCH TO EDUCATION AND ITS
PERIMETER
Its meaning
Its reasons
A BIT OF HISTORY
What’s new
30. ECC2012-13
William James’ mild optimism
William James 1899: Talks
to teachers on psychology
Philosopher - pragmatism
Psychology – scientific vs
introspection
31. ECC2012-13
¤ Psychology ought certainly to give the teacher radical help. And
yet I confess that, acquainted as I am with the height of some of
your expectations, I feel a little anxious lest, at the end of these
simple talks of mine, not a few of you may experience some
disappointment at the net results. In other words, I am not sure
that you may not be indulging fancies that are just a shade
exaggerated.
¤ That would not be altogether astonishing, for we have been
having something like a 'boom' in psychology in this country.
Laboratories and professorships have been founded, and reviews
established. The air has been full of rumors. The editors of
educational journals and the arrangers of conventions have had
to show themselves enterprising and on a level with the novelties
of the day. Some of the professors have not been unwilling to co-
operate, and I am not sure even that the publishers have been
entirely inert. 'The new psychology' has thus become a term to
conjure up portentous ideas withal; and you teachers, docile and
receptive and aspiring as many of you are, have been plunged in
an atmosphere of vague talk about our science, which to a great
extent has been more mystifying than enlightening.
32. ECC 2012
¤ There is nothing but the old psychology, which began in
Locke’s time, plus a little physiology of the brain and senses
and the theory of evolution
¤ I say moreover that you make a great, a very great mistake, if
you think that psychology, being the science of the mind's
laws, is something from which you can deduce definite
programs and schemes and methods of instruction for
immediate schoolroom use. Psychology is a science, and
teaching is an art; and sciences never generate arts directly
out of themselves. An intermediary inventive mind must make
the application, by using its originality.
33. ECC 2012
¤ … the use of psychological principles certainly narrows the
path for experiments and trials. We know in advance, if we
are psychologists, that certain methods will be wrong, so our
psychology saves us from mistakes.
¤ It makes us, moreover, more clear as to what we are about.
We gain confidence in respect to any method which we are
using as soon as we believe that it has theory as well as
practice at its back.
¤ it fructifies our independence, and it reanimates our interest,
to see our subject at two different angles,—to get a
stereoscopic view, … to be able, at the same time, to
represent to ourselves the curious inner elements of his
mental machine
34. ECC 2012
Thorndike’s optimism
Edward Thorndike 1910: The
contribution of psychology to
education
The first to apply principles of
psychology to learning, and to
education.
His theories have been very
influential in education in the
USA
Laws of learning: readiness,
exercise, effect (positive)
35. ECC2012-13
¤ Psychology is the science that backs education, like agriculture
depends on botany
¤ Just as the science and art of agriculture depend upon
chemistry and botany, so the art of education depends upon
physiology and psychology.
¤ The foundation upon which education builds is the equipment
of instincts and capacity given by nature apart from training.
¤ Just as knowledge of the peculiar inheritance characteristic of
any individual is necessary to efficient treatment of him, so
knowledge of the unlearned tendencies of man as a species is
necessary to efficient planning for education in general.
36. ECC2012-13
¤ Psychology contributes to a better understanding of the aims
of education by defining them, making them clearer; by
limiting them, showing us what can be done and what can
not; and by suggesting new features that should be made
parts of them.
¤ …in all cases psychology, by its methods of measuring
knowledge and skill, may suggest means to test and verify or
refute the claims of any method.
¤ Experts in education studying the responses to school
situations for the sake of practical control will advance
knowledge not only of the mind as a learner under school
conditions but also of the mind for every point of view.
37. ECC2012-13
¤ I hope that it is obvious and needless, and that the relation
between psychology and education is not, in the mind of any
competent thinker, in any way an exception to the general
case that action in the world should be guided by the truth
about the world; and that any truth about it will directly or
indirectly, soon or late, benefit action.
38. ECC2012-13
Watson’s plan
J.B. Watson 1913:
Psychology as the
behaviorist views it
Full-fledged behaviorism is a
reaction to the use of introspection,
to the absence of controlled
experiments, and to the focus on
consciousness that characterized
psychology at the turn of the XX
century
39. ECC2012-13
¤ Behaviorism had the aim of making of psychology a science
that can be applied
¤ If psychology would follow the plan I suggest, the educator, the
physician, the jurist and the business man could utilize our data
in a practical way, as soon as we are able, experimentally, to
obtain them.
¤ Those who have occasion to apply psychological principles
practically would find no need to complain as they do at the
present time. Ask any physician or jurist today whether scientific
psychology plays a practical part in his daily routine and you
will hear him deny that the psychology of the laboratories finds
a place in his scheme of work. I think the criticism is extremely
just. One of the earliest conditions which made me dissatisfied
with psychology was the feeling that there was no realm of
application for the principles which were being worked out in
content terms.
40. ECC2012-13
¤ The psychology which I should attempt to build up would take
as a starting point, first, the observable fact that organisms,
man and animal alike, do adjust themselves to their
environment by means of hereditary and habit equipments.
These adjustments may be very adequate or they may be so
inadequate that the organism barely maintains its existence;
secondly, that certain stimuli lead the organisms to make the
responses. In a system of psychology completely worked out,
given the response the stimuli can be predicted; given the
stimuli the response can be predicted.
¤ In experimental pedagogy especially one can see the
desirability of keeping all of the results on a purely objective
plane. If this is done, work there on the human being will be
comparable directly with the work upon animals. … We need
to have similar experiments made upon man…
41. ECC2012-13
Skinner’s teaching machines
J.B. Watson 1913:
Psychology as the
behaviorist views it
Centrality of learning in radical
behaviorism
Theory of operant conditioning,
Reinforcement
Behaviorism allows to control learning,
not just describing it
42. ECC2012-13
¤ The learning process is now much better understood.
¤ Much of what we know has come from studying the behavior of
lower organisms, but the results hold surprisingly well for human
subjects.
¤ The emphasis in this research has not been on proving or
disproving theories but on discovering and controlling the
variables of which learning is a function. This practical
orientation has paid off, for a surprising degree of control has
been achieved.
44. ECC2012-13
¤ By arranging appropriate “contingencies of reinforcement,”
specific forms of behavior can be set up and brought under the
control of specific classes of stimuli.
¤ The resulting behavior can be maintained in strength for long
periods of time. A technology based on this work has already
been put to use in neurology, pharmacology, nutrition,
psychophysics, psychiatry, and elsewhere. The analysis is also
relevant to education. A student can be “taught” in the sense
that he is induced to engage in new forms of behavior and in
specific forms upon specific occasions.
46. ECC2012-13
Behaviorist’s assumptions & limits
¤ implicit assumption:
¤ nothing interesting is going on “inside” (mind is like a blank
slate)
¤ in theory, and as a matter of exaggeration, virtually anything
can be taught
¤ As a matter of fact even radical behaviorism recognizes that
the animal is not a blank slate: only behaviors that are
possible, that are spontaneously realized by the animal can
be reinforced. Skinner considers that anything the child is
ready to learn given her development stage can be taught,
not anything in general.
47. ECC2012-13
¤ (Bruer 1993 p. 3)
¤ In the mid 1950s, behaviorism was the prevailing orthodoxy in
American psychological science.
¤ In education, behaviorist learning theory emphasized
arranging the student’s environment so that stimuli occurred in
a way that would instill the desired stimulus‐response chains.
Teachers would present lessons in small, manageable pieces
(stimuli), ask students to give answers (responses), and then
dispense reinforcement (preferably positive rather than
negative) until their students became conditioned to give the
right answers.
¤ (Bransford et al. 2000 p. 6‐8)
¤ A limitation of early behaviorism stemmed from its focus on
observable stimulus conditions and the behaviors associated
with those conditions. This orientation made it difficult to study
such phenomena as understanding, reasoning, and thinking—
phenomena that are of paramount importance for
education...
48. ECC2012-13
The cognitive revolution
1956 Cambridge MIT
Miller: The magic number 7
Chomsky: A review of B.F.
Skinner Verbal Behavior
Bruner: A study of thinking
1958 Herbert, Shaw, Simon: Elements of
a theory of human problem solving
1960 Harvard Center for Cognitive
Studies (Bruner & Miller)
49. ECC2012-13
¤ Noam Chomsky: A review of BF Skinner Verbal Language
¤ One would naturally expect that prediction of the behavior of a
complex organism (or machine) would require, in addition to
information about external stimulation, knowledge of the internal
structure of the organism, the ways in which it processes input
information and organizes its own behavior.
¤ … Every time an adult reads a newspaper, he undoubtedly comes
upon countless new sentences which are not at all similar, in a
simple, physical sense, to any that he has heard before, and
which he will recognize as sentences and understand; he will also
be able to detect slight distortions or misprints.
¤ Talk of "stimulus generalization" in such a case simply perpetuates
the mystery under a new title.
¤ These abilities indicate that there must be fundamental processes
at work quite independently of "feedback" from the environment.
50. ECC2012-13
¤ Insofar as independent neurophysiological evidence is
not available, it is obvious that inferences concerning the
structure of the organism are based on observation of
behavior and outside events.
¤ The differences that arise between those who affirm and
those who deny the importance of the specific
"contribution of the organism" to learning and
performance concern the particular character and
complexity of this function, and the kinds of observations
and research necessary for arriving at a precise
specification of it.
¤ If the contribution of the organism is complex, the only
hope of predicting behavior even in a gross way will be
through a very indirect program of research that begins
by studying the detailed character of the behavior itself
and the particular capacities of the organism involved.
51. ECC2012-13
¤ (Simon 2000 p. 115)
¤ Exciting research in cognition today combines computer
modeling with neuropsychological studies of the functioning of
the brain and with the experimental study of human learning
and problem solving.
¤ This research is helping to test and improve detailed theories of
the human symbolic processes used in learning and thinking
and to build theories of how skills and knowledge can be taught
effectively and efficiently.
52. ECC2012-13
Constructivism
The cognitive revolution
• inherits the interest for learning
manifested by behaviorism,
• broadens the view (innate capacities,
a larger number of learning processes)
• states the necessity of developing new
methods for peeping into the black box
• Looks at constructivism
53. ECC2012-13
Vygotsky
Lev Vygotsky
• Role of social interaction in
cognitive development
• Zone of proximal development
• Link between development of
language and thinking
54. ECC2012-13
Piaget
Jean Piaget
• Children are like scientists
• Children explain the world on the basis of
their innate structures
• Equilibration between cognitive structures
and environment: Accomodation and
assimilation mechanisms
• Stages of development: Qualitatively
different ways of making sense of the
world
56. ECC2012-13
Situated cognition and the Learning
Sciences
Very soon after the cognitive
revolution, many cognitivists became
dissatisfied with the computational,
representational view of cognition put
forward by the classical cognitive
sciences.
Dissatisfaction concerned the vision of
learning, as well
The first conference of the Learning
Sciences Institute, 1987 (stems from the
Artificial Intelligence and education
previous series of conferences)
57. ECC2012-13
Related visions of cognition:
• Embodied Situated Cognition
(Brooks 1991)
• Distributed cognition (Hutchins
1995)
• …Criticism towards GOFAI,
representationalism,
computationalism ….
58. ECC2012-13
¤ (Anderson Reder Simon 1996)
¤ Situated learning … emphasizes the idea that much of what
is learned is specific to the situation in which it is learned.
learning takes places in concrete situations and it is there
that must be studied
59. ECC2012-13
¤ Artificial intelligence research has foundered on the issue of
representation. When intelligence is approached in an
incremental manner, with strict reliance on interfacing to the
real world through perception and action, reliance on
representation disappears. In this paper we outline our
approach to incrementally building complete intelligent
Creatures. The fundamental decomposition of the intelligent
system is not into independent information processing units
which must interface with each other via representations.
Instead, the intelligent system is decomposed into
independent and parallel activity producers which all
interface directly to the world through perception and action,
rather than interface to each other particularly much. The
notions of central and peripheral systems evaporate
everything is both central and peripheral. Based on these
principles we have built a very successful series of mobile
robots which operate without supervision as Creatures in
standard office environments. (Brooks 1991)
60. ECC2012-13
¤ (Hutchins 1995)
¤ I will attempt to show that the classical cognitive science
approach can be applied with little modification to a unit of
analysis that is larger than an individual person.
¤ One can still ask the same questions of a larger socio-technical
system that one would ask of the individual. That is, we wish to
characterize the behavioral properties of the unit of analysis in
terms of the structure and processing of representations that are
internal to the system. With the new unit of analysis, many of the
representations can be observed directly, so in some respects, this
may be a much easier task than trying to determine the
processes internal to the individual that account for the
individual's behavior.
¤ Posing these questions in this way reveals how systems that are
larger than an individual may have cognitive properties in their
own right that cannot be reduced to the cognitive properties of
individual persons (Hutchins, 1995). Many of the outcomes that
concern us on a daily basis are produced by cognitive systems of
this sort.
61. ECC2012-13
Learning and the brain
1990 Decade of the brain
1994 Cognitive neuroscience
society
1990s Brain-based education
End 1990s Neuroeducation/Mind,
Brain and Education
