“From Video Game to Digital Playground” Virtual Reality and Augmented Reality Siggraph Asia 2018 courses

“From Video Game to Digital Playground”
Hirofumi Motoyama
Eiji Iwata
BANDAI NAMCO Studios Inc.

“From Video Game to Digital Playground”
Virtual Reality and Augmented Reality

Introduction:
Hirofumi Motoyama (Future Lab)
Creative Director at BANDAI NAMCO Studios Inc.
20yrs+ experience as a game designer in the video game/entertainment industry.
Previously worked for NAMCO, UBISOFT, and SEGA.
2

Involvement in Video Game Development until 2010
©NBGI © Ubisoft Entertainment. All Rights Reserved.
3

Involvement in Digital Playground Development from 2010 onwards
©BANDAI NAMCO Amusement Inc.©NBGI
4

Presentation Timeline
Chapter one: The history of video game and XR – 20min
1. History of Video Games & VR/AR/MR
2. Link Between Digital & Real World
3. Digital Content Moving Out of the TV Screen
Chapter two: Digital Playground - The projection based VR – 20min
4. Children & Virtual Reality
5. Design Process - Analog Play & Digital Technology
6. Game Design Which Embraces Human Tendencies
Chapter three: Social Communicator - The HMD based MR – 20min
7. Mixed Reality is “Social Communicator”
8. Social Play = Collaborative Play
9. Amazing Findings
Chapter four: Technical details of MR – Coordinate Engineer – 30min
10. Coordinate system in Mixed Reality attraction.
11. Spatial Scan & Simulation on Editor
12. Light Up The Reality
Chapter five: Our vision of the “Playful World.” – 10min
13. The latest Digital Playground Projects
14. The Magical Moment
15. Conclusion: The world we live in is a “Playful world”
Q&A – 5 min
TOTAL: 105 min 5

1. History of Video Games & VR/AR/MR
8

Video game console
https://en.wikipedia.org/wiki/Video_game_console
10

PAC-MANTM&©BANDAI NAMCO Entertainment Inc.
12

14
Avatar Controller Player
Screen Interface Human

15
Player is manipulator

16
Manipulating Avatar is really fun!

Video game console
https://en.wikipedia.org/wiki/Video_game_console
17

19
2007
https://en.wikipedia.org/wiki/Multi-touch#/media/File:Aldea_Digital_2013_01.jpg

20
Avatar Multi-touch Player

22
Project Natal (2009) - Kinect
Microsoft, Kinect is a trademark of Microsoft Corporation.

23
Avatar
Player
➡ Controller
Screen Interface
Player is controller

24
But…Player is controller is fun?
Avatar
Player
➡ Controller
Screen Interface

25
Project Natal (2009) - Kinect
Microsoft, Kinect is a trademark of Microsoft Corporation.

29
Player is Avatar in screen
©BANDAI NAMCO Amusement Inc.

30
Player becomes Avatar
Screen Screen
Player = Avatar
Sensing
Player = Avatar

31
Sensing technology worked well for avatar sensing
Screen
Player = Avatar
Sensing

32
Controller
as sensing
Player
➡ Avatar
Interface
Screen

33
Explore the play experience by observation
REAL Product

34
Kid’s Beach

37
VR MRAR
https://www.apple.com/jp/newsroom/2018/06/apple-unveils-arkit-2/?imgid=c02b48c234d25efa8cb364f3ff4ee86a
https://news.microsoft.com/uploads/2018/02/MSHoloLens_CommercialPillar_Education_01602-5a8c80979ab97.jpg
https://www.oculus.com/go/

38
Controller
as sensing
Player
➡ Avatar
Interface
Screen

2. Link Between Digital & Real World
39

40

Joy Stick
41

Game controller
https://en.wikipedia.org/wiki/Game_controller
Imitation
42

https://en.wikipedia.org/wiki/Multi-touch#/media/File:Aldea_Digital_2013_01.jpg
Touch
43

44
Hand
https://newsroom.fb.com/news/2016/12/oculus-touch-now-available/

Wearable
Sensing
45
Ⓒ BANDAI NAMCO Studios Inc.

3. Digital Content Moving Out of the TV Screen
46

47

TV Screen
48
Limited screen
Larger,
wider

https://www.oculus.com/go/
outside of the monitor” experience TV Screen
49

50
Future of outside of the monitor experience?
Looking Glass Factory, Inc
Looking Glass Factory, Inc
Illustration credit is Lui Antonio, Jr and Hannah Liongoren

4. Children & Virtual Reality
52

53
Sandbox
4.5m×6.5m
Sand Volume
6.5 ton
10 Projectors
4 Kinect V2
8 Speakers
80 fishes real-
time recognition
6 months to
develop
Kid’s Beach

“Kid’s Beach”: Play Content Overview
• catch a variety of realistic fish
• Playing with sea or tide pool
• Transition between shallows – tide pools – Big waves – whale jump
• Interaction with sea creatures such as sea anemone and fish
55

In JAPAN
56

In UK
(Image: Handout, Publicity Picture)
https://www.birminghammail.co.uk/whats-on/family-kids-news/indoor-beach-launched-tamworth-just-12330438
57

58
Virtual Reality without HMD
Run!
Its a big wave!
splash around in
shallow water

“EASY” to Enjoy with “MANY” People
Family oriented amusement Even if you are too close
no problem
59

60
60
“First Step” into the content is crucial point
This feels good!
It’s fun!My feet
feels cold!
Linking Immersive Experience to the Physical Body

61
Reality Further Enhanced by “Actual Things”
Reminiscing childhood
memoriesfeeling the sand

62
Acquire reality in “whole body” by surround sound
©BANDAI NAMCO Amusement Inc.Ⓒ BANDAI NAMCO Studios Inc.

5. Design Process - Analog Play & Digital Technology
63

64
Playground Equipment as an “Universal Language” Pt. 1
• Video game is a form of universal language
• If the game is fun, people in any country will actively play.
• Although there are differences in language, culture, and environment,
through gaming, players descend their soul down in the same world.
• Worldwide products consists of elements that can be enjoyed globally.

65
Playground Equipment as an “Universal Language” Pt. 2
https://commons.wikimedia.org/wiki/File:Pieter_Bruegel_the_Elder_-_Children%E2%80%99s_Games_-_Google_Art_Project.jpg
Bruegel the Elder - Children’s Games

66
“Analog” & “Digital” Entertainment
Interactive Indoor Beach
Analog Analog & Digital
Indoor Sand box

67
“Primitive Interaction” Brings Back Fun & Happy Memories Pt. 1
Recalls back
to fun and
happy
memories
experienced
in the real
world
Having fun in the real world
No Tutorial, No Instruction Manual
Children Starts Playing Immediately Reality & Virtual Reality Spiral of Fun

②
①
The fish cannot be
caught
approximately 30 cm
above virtual sea
surface.
68
× Fish
escapes/can
not be
caught
○ Can
catch fish
“Primitive Interaction” Brings Back Fun & Happy Memories Pt. 2
51

5. Game Design which Embraces Human Tendencies
69

70
Game design allowing "getting tired"
There is no problem children get tired one content because they
will interest to another content.
migratory behavior.

71
Feeling of “Presence” Solidified Through Limiting Digital Content

72
• “Combining analog amusement with digital technology”
• “Bringing back playful memories through primitive senses”
• “Understanding & accepting human nature of losing interest”
• “Removing competitive game mechanics to add focus on the
physical “presence” of playing on the real space”
Digital Playground design

73
“Digital Playground”

7. Mixed Reality is “Social Communicator”
75

76
iPhone unveiled 9th
January 2007
Source：
UPDATEJANUARY 8, 2017
iPhone at ten: the revolution continues
https://www.apple.com/newsroom/2017/01/iphone-at-ten-the-
revolution-continues.html
Image:
https://www.apple.com/newsroom/2017/01/iphone-at-ten-the-revolution-
continues/?imgid=1483642528978

77
Smartphone is
Internet Communicator

78
Internet communicator with Multi-Touch UI
：https://en.wikipedia.org/wiki/Multi-touch#/media/File:Aldea_Digital_2013_01.jpg

79
“Breakthrough Internet Communicator”
Person ⇒ Display ⇒ Person
Display with multi-touch

80
Everyone is gathering, but everyone is watching the display.
“Breakthrough Internet Communicator”

81
Mixed Reality is,
Internet Communicator
Social Communicator

82
Smartphone VS Mixed Reality Glass
Image：de:code 2017 JAPAN keynote http://www.atmarkit.co.jp/ait/articles/1706/08/news018_3.html
Eye Contact through glass
Social communicatorInternet Communicator
Communication between people
Person ⇒ Display

83
Social Communicator becomes “Social Play”
Social Communicator
Social Play = “PAC in Town” in real world

Three Core Pillars
１）Real world:
8 square meters playfield at ARS Electronica Center Lobby
２）Centering theme - Real People & their Interaction:
3 Players equips a HoloLens and interact with each other by high-fiving
to change role (to PAC-MAN).
３）Diversity:
Each player is assigned to a specific role (PAC-MAN, CHERRY or a
POWER PELLET).
84

8. Social Play = Collaborative Play
85

86
PAC IN TOWN
Playground 8m×8m
3 players
3 HoloLens
1 Kinect V2
8,000 people played
3 developers
1 months to develop
Available Tomorrow !
SIGGRAPH ASIA 2018!

87
Namjatown x MR Project Pt. 1
Real PAC-MAN Attraction “PAC IN TOWN”
[Limited Exhibition: Jan 15th (Mon) 2018 – May 6th (Sun) 2018 ]

88
Namjatown x MR Project Pt. 2
©BANDAI NAMCO Amusement Inc. http://panora.tokyo/53151/©BANDAI NAMCO Amusement Inc.
Real RIDE Attraction Explosion! Mosquito Battle
[Limited Exhibition: Feb 10 (Sat) 2018 – March 31th (Sat) 2018 ]

The rule is different compared to the classic PAC-MAN game
3 players will transform into PAC-MAN
to consume 100 PAC-DOTS within a set limited time.
① ② ③
91

A frequent question that’s asked about “PAC IN TOWN” is…
"Why wasn’t this game designed as single player
just like the original PAC-MAN?”
“Why is there a need to support three players?”
92

93
Collaborative Play
There are three key benefits of Collaborative Play .
Mixed Reality works best with

Benefits of MR 1: Naturally starts communication
Everyone will sharing
an experience
“Our human instinct” when given a common goal is to communicate and work together.
94

95
Benefits of MR 1: Naturally starts communication

Benefits of MR 2: Players start teaching / advising one another
As teamwork is crucial to completing the game.
the player who are more into playing game starts teaching and giving advices for those who do not.
96

97
Benefits of MR 2: Players start teaching / advising one another

Benefits of MR 3: Sports factor
It’s safe & accident-free because players see the real people and real world.
players are given the freedom to move around just like playing a sport.
98

99
Benefits of MR 3: Sports factor

Collaborative Play is FUN!
Mixed Reality provides a unique platform which
enhances human interaction.
100

Mixed Reality has potential in
“encouraging sociability” through entertainment
This message alone sums up Mixed Reality’s uniqueness and what it has to
offer to the players.
“An innovative take on original PAC-MAN. The
social factor was fantastic. Great way to
experience playfulness with strangers.”
102

how the “PAC IN TOWN” alone triggered interaction between people who
were complete strangers before the game.
103

Mixed Reality has potential in
“encouraging sociability” through entertainment
by collaborative play
104
Ⓒ BANDAI NAMCO Studios Inc. ©BANDAI NAMCO Amusement Inc.

Introduction:
Eiji Iwata
Research Engineer at BANDAI NAMCO Studios Inc.
106

Eiji Iwata (Engineer)
2011: Joined Bandai Namco Games.
2013: Started development of AR/MR games on AR smart eyeglass Moverio.
CEDEC 2014: Session lecturer ”Unearthing New Entertainment with Binocular
AR Glass.“
2015: Participated in tech demo for “Summer Lesson.” PlayStation VR
Current: Developing MR games on HoloLens
Profile
107

What Engineers Should Do in MR?
1. Maintain orderly relations between virtual and reality.
108
2. Bring reality into development as resources.
3. Light up the reality!

Technological Topics
109
2. Scanning of environment and installation in development process.
3. Design of audience view focused on engagement.

110

About Project Mosquito
111
Explosion! Mosquito Battle

Play Flow of attraction
113
Free roaming
• Show guide UI
• Show guide characters
• Visualize spatial map
Riding
• Show characters
• Interact with spatial map
• Play

Seed of Complication 1 : Environmental Condition
114
Too Dark for
Position Tracking!
Real Time Tracking
and
Pre-Recorded Tracking

Seed of Chaos 2 : Development Style
115
Global Coordinate
Scene Local
Coordinate
Wave 1 Scene Wave 2 Scene Wave 3 Scene

Seed of Chaos 3 : Feedback from Test Play
116
Adjust my position,
or adjust your position?

Seed of Chaos 4 : Miss Understanding
Nothing appears on HoloLens
Integration
Position Correction Marker
Z Direction = Front
Programmer
C# C#
117
Z Direction = Back

Everything Messes Up Your Coordinate System
Environmental
Condition
Development
Style
Feedback from
Test Play
Miss
Understanding

You need wrapper!
Adopted Strategies
119
Coordination Wrapper API
C#
Coordination Engineer
Other Engineer Scene Designer Artist
C#

PAC IN TOWN – Spatial Sharing
• Unity World Coordinate
• View Camera Coordinate
• Location Venue Coordinate
• Kinect Coordinate
• Coordinates of two players
- System of Coordinates of the amusement
venue is determined by marker recognition.
- The coordinates of 3 players are shared
through the network per frame.
120

Operation Mosquito – Complex Coordinate System
• Unity World Coordinate
• View Camera Coordinate
• Amusement Venue Coordinate
• Amusement Ride Coordinate
• Estimation of simulation and tracking *
• Head of the person on the ride
• World coordinate of subscene (special sequence)
* We realized SLAM may not track consumers
riding the attraction due to darkness
- Worked around the issue by ignoring position
estimation via tracking during the ride and replace
the system by position estimation simulation.
121

11. Spatial Scan & Simulation on Editor
122

Creating On-site Spatial Environment with Unity
Recreating on-site environment at the office
123

Comparison of Simulation on Editor & Actual Location Attraction
124

Example 1: Level Design of Actual On-Site Location Created on Unity
Placement of characters in a specific position
Unity screen HoloLens field of view
125

Example 2: Level Design of Actual On-Site Location Created on Unity
Place events at specific points of the track course
Unity screen HoloLens field of view
126

Track of the ride
using HoloLens
Marker indicating
position of attraction
ride
Wall and placed
objects scanned by
HoloLens
Complete Scan of On-site Environment
127

Integration of Scanned Data
Mesh: Lower half
portion
Mesh: Upper half
portion
Track line of the amusement ride
c
Integration
Optimization
Integrated data of the real
environment
128

Optimization of Scanned Data
1. Deleting floor
and ceiling
2. Delete
unnecessary
mesh deriving
from the ride
and human
body
Mesh before
processing
Mesh after
processing
129

Two Coordinate System Modes: Internal Development Mode & On-site Mode
In-Office ModeOn-site Mode
Directional view is dependent
on HoloLens’ feature but is
fixed on the editor
The directional view follows
the direction of the ride.
130

Methodology for Optimizing Scanned Data
1. Retrieve maximum height and index from MeshFilter acquired from Spatial Map
2. Data optimization is conducted under fixed condition
• Delete mesh outside of a defined area from the course track to secure optimal field of
vision
• Trim surrounding mesh to avoid objects placed by Unity from being hidden
• Mesh acquired from the floor and ceiling will be deleted as it may obstruct the view
131

Topic: You’ll Take Pain in Networking
132

- Pixel-by-pixel of depth and color obtained by Kinect V2
are spatially placed, which enable correct occlusion.
Audience View of “PAC IN TOWN”
134

- Initial Aim:
Entertain audience.
- Unexpected Merit:
Engage everyone around the player to make it better experience.
Unexpected Merit of Audience View
135

What happened by Audience View
1. Players gather attention of an audience.
136
2. Operators and audience support players.
The Real World was Empowered!

What Players see on MR Glasses?
137
Real World
Holographic

Support Tool Developed for On-Site Operation
• Simple : Push 1 button to run games.
• Supportive: trouble shooting is shown.
139

What is the Synonym of Mixed Reality
140
Mixed Reality
Augmented Reality
Fake Reality

What Can You Do in MR Development?
1. Maintain orderly relations between virtual and reality.
141
2. Bring reality into development as resources.
3. Light up the reality!

13. The latest Digital Playground projects
143

144
Powered By
Tachikawa, Tokyo
TAKASHIMAYA S.C. 8F
Digital Playground - From Beach to Adventure Island

147
PAC IN TOWN x Mobility
＋ RIDE Experience
From fixed location to Playable City
Available Tomorrow !
SIGGRAPH ASIA 2018

150

151

How one ex-game developer helped grant Batkid's
wish to save Gotham City
Polygon
https://www.polygon.com/2013/12/14/5208764/batman-eric-johnston-batkid-make-a-wish-san-francisco
"The Batman to Miles' Batkid was a man
named Eric Johnston, a veteran video
game programmer who helped create some
of the games you know and, in some cases,
deeply love and fondly remember.
Mike Jutan, another Lucasfilm programmer
who played the part of The Penguin.
152
Polygon
https://www.polygon.com/2013/12/14/5208764/batman-eric-johnston-batkid-make-a-wish-san-francisco

153
People made collaborative effort to recreate San Francisco into Gotham City.
the future of playful world for the fusion of the real and the digital world

15. Conclusion:
The world we live in is a “Playful world”
154

Summery
The world we live in is a “Playful world”
155
“The player is the star” in playful world.

Thank you
FOR YOUR TIME!
Questions?
Hirofumi Motoyama
Eiji Iwata
BANDAI NAMCO Studios Inc.

“From Video Game to Digital Playground” Virtual Reality and Augmented Reality Siggraph Asia 2018 courses

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (19)

Similaire à “From Video Game to Digital Playground” Virtual Reality and Augmented Reality Siggraph Asia 2018 courses

Similaire à “From Video Game to Digital Playground” Virtual Reality and Augmented Reality Siggraph Asia 2018 courses (20)

Dernier

Dernier (20)

“From Video Game to Digital Playground” Virtual Reality and Augmented Reality Siggraph Asia 2018 courses