Keynote on the cognitive demand of the multimodal media object of videogames. Presented at "DigiLitEY Training School 2016", in the Institute of Education, University of Minho. 7 june 2016.
HỌC TỐT TIẾNG ANH 11 THEO CHƯƠNG TRÌNH GLOBAL SUCCESS ĐÁP ÁN CHI TIẾT - CẢ NĂ...
Videogames and Multimodal Literacy
1. Videogames and Multimodal Literacy
Nelson Zagalo, University of Minho
http://nelsonzagalo.googlepages.com
http://virtual-illusion.blogspot.com
Institute of Education, University of Minho 7 June 2016
DigiLitEY Training School 2016
2. …been doing
Authored and edited books on:
Storytelling, Aesthetic Emotions, Virtual Worlds, Videogames and Creativity
3. Research in the domain of Human–Computer Interaction, within a Communication
Cognitive approach.
In specific, Interaction Design, and in concrete, Engagement.
…domains
4. a) Syntax
Code, symbolic objects of expression.
b) Semantic
Meaning, interpretation of the symbols and its interrelations.
c) Pragmatic
Context, understanding the relation between participants.
Multimodal literacy: communication
For the multimodal literacy purpose, problems arise when
different objects of expression (modes) are used to support
the syntax, affecting semantics but mainly pragmatics.
Levels of the communication process (Watzlawick, 1967).
5. Communication can occur between humans, between humans and
machines, between humans mediated by technologies, and even between
humans mediated by artifacts (like videogames).
Defining pragmatics
6. The human communication occurs through body and can be
externalized to objects of expression (media), which add new modes to
human expression, each making use of different codes.
Modes
verbal (voice expression),
non verbal (body language),
sound (object produced),
picture (drawing, photo…),
text (visual symbols),
animation (moving pictures),
audiovisual (moving pictures with sound),
interaction (action simulation).
Syntax
Each mode develops its own conventions, languages, creating what we
call Media Arts (painting, sculpture, literature…), which we use to
express ourselves and communicate.
Multimodal literacy: modes
8. Multimodal arts: theater, film, and videogames
Videogames are an integration of previous media arts, like film,
and even theater.
Videogame: Cinema + Computer Processing
new expression: Interactivity (user’s performance)
Cinema: Theater + Photography
new expression: Montage (sequence of photos)
Theater: Sculpture + Painting + Literature + Music + Dance + Fashion
new expression: Drama (actor’s performance)
9. Multimodal path: from text to videogame
Through literature the
main goal of the writer
is to tell the narrative
as detailed as
possible in order to
develop a strong
“fabula”, or mental
story, in the receptor’s
mind.
Film doesn’t need to spend time telling details
because they are shown. The story world is
given already built to the receptor, purposing
direct perception of the visual world that
enhances sensorial emotion and learning.
Coppola, The Godfather, 1972Puzo, The Godfather, 1969
10. Story is no more an act of “telling”, neither “showing” but an integrated set
of active participation, of “doing”. Interactive media open a new space
(virtual) for the mediation of knowledge. The story containing learning
messages will evolve only through the actions and decisions of the
participator.
Videogames go beyond representation to become simulation.
EA, The Godfather: The Game, 2006
Do Something Do Something
Multimodal path: from text to videogame
11. Representation is made of conventions of symbols, different from reality,
involve strong need of interpretation. More demanding in abstract
reasoning, cultural background, and imagination activation.
Simulations are illusions of the real, purposing a sort of direct access to the
real without its insecurities. Permits learning through infinite repetition, with
little change, modelling for a fixed mental model.
Representation vs. Simulation
12. Acting upon the real, upon objects, simulations, makes learning become a
task memorized through somatic actions, using body perception instead of
mere mental activity, configuring an optimized situated cognition
environment.
One of the greatest examples of the gains of simulations have been flight
simulators, which were able to lower drastically accidents.
The simulation gain
13. In the 1960’s, Seymour Pappert (MIT) following his studies with children,
purposed a sort of visual programming language, LOGO, which he defined
as "body-syntonic reasoning”, because students could write code putting
themselves in the place of the turtle, mentally simulating, doing the
reasoning and predicting based in the turtle’s position and movement.
The computing evolution: becoming visual
14. Then in the 1970’s, Alan Kay (Xerox) purposed moving from text-based to
image-based interfaces. The idea arose from Pappert studies, the
evaluation done for the interaction with objects and written words,
understanding that people would be more able to act upon objects than on
written words.
The computing evolution: becoming visual
15. In the 2000’s, Mitchel Resnick (MIT) developed a real visual language for
kids, Scratch, which in 2016 was brought into the physical world by Osmo.
The written symbols became visual elements, and finally physical objects.
Programming was born in abstract modelling but is now made of concrete
objects, sort of “programming simulator”.
The computing evolution: becoming physical
16. The computing evolution: in the future
“Holodeck”, machine presented in 1998 in the Star Trek
series, previously dreamed by André Bazin in the paper “The
Myth of Total Cinema” (1948).
18. Why multimodal literacy evolved for
simulation?
Transforming the abstract topics into realistic
objects, and adding to them a fully interactive layer,
to guarantee a virtual learning by doing.
19. What can you “do” in videogames: 3 levels
1 - Manipulation
Act upon the representation, change and
transform, express yourself trough virtual
objects.
3 - Participation
Take decisions, choose, decide, experiment
with moral values, and check your real beliefs.
2 - Navigation
Follow conventions or define your own
approach to the virtual world. You’re what you
do in the world.
20. Doing to Learn
Brenda Romero, Gaming for understanding, at TEDxPhoenix (2011)
Learnign about Middle Passage at schools
21. Families in Africa Boat to take people to the US
New World , board game improvised
by Brenda Romero
Doing to Learn
Rules
. Takes 10 turns to cross the ocean.
. Can not take complete color sets of
pawns.
. We have 30 units of food.
. Each turn roll the dice.
. Dices determine the food units to be
consumed.
23. Choice / Power
Autonomy means, to empower the receptor to the limit of experimenting
with moral values, testing beliefs, trying in “practice” your own theories.
To Do: autonomy
When your friend Chloe asks you to put an end to her life,
you need to decide, and choose if you let her die or not.
“Life is Strange” (2015)
24. Skills / Progress
Competence means, to develop the skills requested by the simulation
to keep progressing. Progress is the main fuel of interest.
No text manuals to read, you’re taken by the hand, each action done,
you go up another step in comprehending what’s expected from
you. A proper scaffolding process.
To Do: competence
25. Others / Recognition
Relatedness means, to be part of a community, be as capable as the
others, which can be easily translated by Leaderboards, or school
friends recognition.
To Do: relatedness
26. Videogames are interactive audiovisual
experiences, everything we take into account in
movies must be taken into account here, but the
center of the cognitive effort, the real challenge for
the players, goes beyond all the interpretation you
need to understand the audiovisual medium, you
need to become part of the experience, and in order
to fully understand it you need to perform, to do.
The videogame’s core of the multimodal literacy
27. Videogames and Multimodal Literacy
Nelson Zagalo, Universidade do Minho
http://nelsonzagalo.googlepages.com
http://virtual-illusion.blogspot.com
nzagalo@ics.uminho.pt
Editor's Notes
I’m a professor and researcher of interaction design, I’ve no affiliation with Education. Why am I here?
Interaction Design is the specificity of the design of artefacts concerned with the communication with humans.
Education, is the relation built within the transfer of knowledge and skills from generation to generation.
Education as Interaction Design, has Communication as a central motive. The question of Education is not only related about what to transmit, but also How to transmit. And that’s about that I want to talk here.
Communication is not a simple act of transmission, but an act of putting in common, so to create understanding. In this sense, what I want to purpose here, is, how to design better the communication in education, a sort of “learning design model”.
Education as Interaction Design, have communication in their core. The question of Education, more than what to transmit, is about how to transmit, is about how to build relations, which is the central question of Interaction Design.