Successfully reported this slideshow.
Videogames and Multimodal Literacy
Nelson Zagalo, University of Minho
http://nelsonzagalo.googlepages.com
http://virtual-i...
…been doing
Authored and edited books on:
Storytelling, Aesthetic Emotions, Virtual Worlds, Videogames and Creativity
Research in the domain of Human–Computer Interaction, within a Communication
Cognitive approach.
In specific, Interaction ...
a) Syntax
Code, symbolic objects of expression.
b) Semantic
Meaning, interpretation of the symbols and its interrelations....
Communication can occur between humans, between humans and
machines, between humans mediated by technologies, and even bet...
The human communication occurs through body and can be
externalized to objects of expression (media), which add new modes ...
Analogue
Sculpture
Painting
Literature
Music
Theatre
Dance
Fashion
Industrial
Cinema
Photography
Animation
Television
Vide...
Multimodal arts: theater, film, and videogames
Videogames are an integration of previous media arts, like film,
and even t...
Multimodal path: from text to videogame
Through literature the
main goal of the writer
is to tell the narrative
as detaile...
Story is no more an act of “telling”, neither “showing” but an integrated set
of active participation, of “doing”. Interac...
Representation is made of conventions of symbols, different from reality,
involve strong need of interpretation. More dema...
Acting upon the real, upon objects, simulations, makes learning become a
task memorized through somatic actions, using bod...
In the 1960’s, Seymour Pappert (MIT) following his studies with children,
purposed a sort of visual programming language, ...
Then in the 1970’s, Alan Kay (Xerox) purposed moving from text-based to
image-based interfaces. The idea arose from Papper...
In the 2000’s, Mitchel Resnick (MIT) developed a real visual language for
kids, Scratch, which in 2016 was brought into th...
The computing evolution: in the future
“Holodeck”, machine presented in 1998 in the Star Trek
series, previously dreamed b...
Multimodal expression evolution
Symbolic, abstraction (text)
Audiovisual, realism (image/sound)
Simulation, surrogate (int...
Why multimodal literacy evolved for
simulation?
Transforming the abstract topics into realistic
objects, and adding to the...
What can you “do” in videogames: 3 levels
1 - Manipulation
Act upon the representation, change and
transform, express your...
Doing to Learn
Brenda Romero, Gaming for understanding, at TEDxPhoenix (2011)
Learnign about Middle Passage at schools
Families in Africa Boat to take people to the US
New World , board game improvised
by Brenda Romero
Doing to Learn
Rules
....
Motivation
CompetenceAutonomy Relatedness
Motivation model from the “Self-determination theory” (Deci e Ryan, 1985).
What ...
Choice / Power
Autonomy means, to empower the receptor to the limit of experimenting
with moral values, testing beliefs, t...
Skills / Progress
Competence means, to develop the skills requested by the simulation
to keep progressing. Progress is the...
Others / Recognition
Relatedness means, to be part of a community, be as capable as the
others, which can be easily transl...
Videogames are interactive audiovisual
experiences, everything we take into account in
movies must be taken into account h...
Videogames and Multimodal Literacy
Nelson Zagalo, Universidade do Minho
http://nelsonzagalo.googlepages.com
http://virtual...
Prochain SlideShare
Chargement dans…5
×

Videogames and Multimodal Literacy

406 vues

Publié le

Keynote on the cognitive demand of the multimodal media object of videogames. Presented at "DigiLitEY Training School 2016", in the Institute of Education, University of Minho. 7 june 2016.

Publié dans : Formation
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Videogames and Multimodal Literacy

  1. 1. Videogames and Multimodal Literacy Nelson Zagalo, University of Minho http://nelsonzagalo.googlepages.com http://virtual-illusion.blogspot.com Institute of Education, University of Minho 7 June 2016 DigiLitEY Training School 2016
  2. 2. …been doing Authored and edited books on: Storytelling, Aesthetic Emotions, Virtual Worlds, Videogames and Creativity
  3. 3. Research in the domain of Human–Computer Interaction, within a Communication Cognitive approach. In specific, Interaction Design, and in concrete, Engagement. …domains
  4. 4. a) Syntax Code, symbolic objects of expression. b) Semantic Meaning, interpretation of the symbols and its interrelations. c) Pragmatic Context, understanding the relation between participants. Multimodal literacy: communication For the multimodal literacy purpose, problems arise when different objects of expression (modes) are used to support the syntax, affecting semantics but mainly pragmatics. Levels of the communication process (Watzlawick, 1967).
  5. 5. Communication can occur between humans, between humans and machines, between humans mediated by technologies, and even between humans mediated by artifacts (like videogames). Defining pragmatics
  6. 6. The human communication occurs through body and can be externalized to objects of expression (media), which add new modes to human expression, each making use of different codes. Modes verbal (voice expression), non verbal (body language), sound (object produced), picture (drawing, photo…), text (visual symbols), animation (moving pictures), audiovisual (moving pictures with sound), interaction (action simulation). Syntax Each mode develops its own conventions, languages, creating what we call Media Arts (painting, sculpture, literature…), which we use to express ourselves and communicate. Multimodal literacy: modes
  7. 7. Analogue Sculpture Painting Literature Music Theatre Dance Fashion Industrial Cinema Photography Animation Television Video Comics Graphic design Digital Videogame Virtual Reality 3D Animation Generative Music Physic Computation Programming Interaction Design Multimodal literacy: the media arts Integration of modes
  8. 8. Multimodal arts: theater, film, and videogames Videogames are an integration of previous media arts, like film, and even theater. Videogame: Cinema + Computer Processing new expression: Interactivity (user’s performance) Cinema: Theater + Photography new expression: Montage (sequence of photos) Theater: Sculpture + Painting + Literature + Music + Dance + Fashion new expression: Drama (actor’s performance)
  9. 9. Multimodal path: from text to videogame Through literature the main goal of the writer is to tell the narrative as detailed as possible in order to develop a strong “fabula”, or mental story, in the receptor’s mind. Film doesn’t need to spend time telling details because they are shown. The story world is given already built to the receptor, purposing direct perception of the visual world that enhances sensorial emotion and learning. Coppola, The Godfather, 1972Puzo, The Godfather, 1969
  10. 10. Story is no more an act of “telling”, neither “showing” but an integrated set of active participation, of “doing”. Interactive media open a new space (virtual) for the mediation of knowledge. The story containing learning messages will evolve only through the actions and decisions of the participator. Videogames go beyond representation to become simulation. EA, The Godfather: The Game, 2006 Do Something Do Something Multimodal path: from text to videogame
  11. 11. Representation is made of conventions of symbols, different from reality, involve strong need of interpretation. More demanding in abstract reasoning, cultural background, and imagination activation. Simulations are illusions of the real, purposing a sort of direct access to the real without its insecurities. Permits learning through infinite repetition, with little change, modelling for a fixed mental model. Representation vs. Simulation
  12. 12. Acting upon the real, upon objects, simulations, makes learning become a task memorized through somatic actions, using body perception instead of mere mental activity, configuring an optimized situated cognition environment. One of the greatest examples of the gains of simulations have been flight simulators, which were able to lower drastically accidents. The simulation gain
  13. 13. In the 1960’s, Seymour Pappert (MIT) following his studies with children, purposed a sort of visual programming language, LOGO, which he defined as "body-syntonic reasoning”, because students could write code putting themselves in the place of the turtle, mentally simulating, doing the reasoning and predicting based in the turtle’s position and movement. The computing evolution: becoming visual
  14. 14. Then in the 1970’s, Alan Kay (Xerox) purposed moving from text-based to image-based interfaces. The idea arose from Pappert studies, the evaluation done for the interaction with objects and written words, understanding that people would be more able to act upon objects than on written words. The computing evolution: becoming visual
  15. 15. In the 2000’s, Mitchel Resnick (MIT) developed a real visual language for kids, Scratch, which in 2016 was brought into the physical world by Osmo. The written symbols became visual elements, and finally physical objects. Programming was born in abstract modelling but is now made of concrete objects, sort of “programming simulator”. The computing evolution: becoming physical
  16. 16. The computing evolution: in the future “Holodeck”, machine presented in 1998 in the Star Trek series, previously dreamed by André Bazin in the paper “The Myth of Total Cinema” (1948).
  17. 17. Multimodal expression evolution Symbolic, abstraction (text) Audiovisual, realism (image/sound) Simulation, surrogate (interactive)
  18. 18. Why multimodal literacy evolved for simulation? Transforming the abstract topics into realistic objects, and adding to them a fully interactive layer, to guarantee a virtual learning by doing.
  19. 19. What can you “do” in videogames: 3 levels 1 - Manipulation Act upon the representation, change and transform, express yourself trough virtual objects. 3 - Participation Take decisions, choose, decide, experiment with moral values, and check your real beliefs. 2 - Navigation Follow conventions or define your own approach to the virtual world. You’re what you do in the world.
  20. 20. Doing to Learn Brenda Romero, Gaming for understanding, at TEDxPhoenix (2011) Learnign about Middle Passage at schools
  21. 21. Families in Africa Boat to take people to the US New World , board game improvised by Brenda Romero Doing to Learn Rules . Takes 10 turns to cross the ocean. . Can not take complete color sets of pawns. . We have 30 units of food. . Each turn roll the dice. . Dices determine the food units to be consumed.
  22. 22. Motivation CompetenceAutonomy Relatedness Motivation model from the “Self-determination theory” (Deci e Ryan, 1985). What you do and why?
  23. 23. Choice / Power Autonomy means, to empower the receptor to the limit of experimenting with moral values, testing beliefs, trying in “practice” your own theories. To Do: autonomy When your friend Chloe asks you to put an end to her life, you need to decide, and choose if you let her die or not. “Life is Strange” (2015)
  24. 24. Skills / Progress Competence means, to develop the skills requested by the simulation to keep progressing. Progress is the main fuel of interest. No text manuals to read, you’re taken by the hand, each action done, you go up another step in comprehending what’s expected from you. A proper scaffolding process. To Do: competence
  25. 25. Others / Recognition Relatedness means, to be part of a community, be as capable as the others, which can be easily translated by Leaderboards, or school friends recognition. To Do: relatedness
  26. 26. Videogames are interactive audiovisual experiences, everything we take into account in movies must be taken into account here, but the center of the cognitive effort, the real challenge for the players, goes beyond all the interpretation you need to understand the audiovisual medium, you need to become part of the experience, and in order to fully understand it you need to perform, to do. The videogame’s core of the multimodal literacy
  27. 27. Videogames and Multimodal Literacy Nelson Zagalo, Universidade do Minho http://nelsonzagalo.googlepages.com http://virtual-illusion.blogspot.com nzagalo@ics.uminho.pt

×