Extended Reality for Virtual Social Learning

Extended Reality for Virtual Social Learning
Zi Siang See
SMARTlab, School of Creative Industries, University of Newcastle
Email: zisiang.see@newcastle.edu.au
Research: https://www.newcastle.edu.au/profile/zisiang-see
FB: www.facebook.com/zisiang
Linkedin: https://au.linkedin.com/in/zisiangsee
Demo: www.zisiangsee/demo
Keynote Workshop: Zi Siang See, Xia Sheng Lee (Bruce)
OpenLearning 2018
November 2018
Presentation: Zi Siang See, Xia Sheng Lee
The transition of Emerging Technologies for Education Research
Xia Sheng Lee (Bruce)
Yalong Intelligent Equipment Group Co.,LTD.
Email: leexiasheng@gmail.com
Linkedin: https://www.linkedin.com/in/brucexslee
FB: https://www.facebook.com/bruce.l.sheng
Research: https://scholar.google.com/citations?user=5OdbSv4AAAAJ&hl=en
Research &
collaboration

User Experience: Quiver
Print, Color, Play, try it at http://www.quivervision.com
Presentation: Zi Siang See et al.

Is User Experience Important to You?

Pedagogy and Learning Approach

Neil D. Fleming David A. Kolb Maria Montessori Albert Bandura

Neil D. Fleming
VARK Learning Style
• Visual
• Auditory
• Reading
• Kinesthetic
David A. Kolb
Experiential Learning
• Experience
• Observation
• Conceptualization
• Experimentation
Maria Montessori
Montessori Learning
• Freedom
• Mixed-age classrooms
• Uninterrupted choice
• Exploration
Albert Bandura
Social Learning
• Observation
• Imitation
• Modeling

Research Presentation: Zi Siang See, Lizbeth Goodman
A UON Project led by:
Donovan Jones, Shanna Fealy (University of Newcastle)
School of Nursing & Midwifery, Faculty of Health & Medicine

Is social learning = you are who you help?

Researcher: Zi Siang See
Academic Researcher (Interaction Design), University of Newcastle
Zi Siang See: Industry and Academic Research
Presentation: Zi Siang See

Research and Specialization
Subject Matter Expertise
• Creative Multimedia
• Interaction Design
• Virtual Reality (VR)
• Augmented Reality (AR)
• Inclusive Design

Digital Technology Management Specialization
Formerly at University of Reading
Xia Sheng Lee (Bruce): Industry and Academic Research

Research and Specialization
Subject Matter Expertise
• Digital Technology Management
• Virtual Design in Construction
• Building Information Modeling
• 3D Terrestrial Laser Scanning
• Virtual Learning

Preview

Introduction
Multimedia (Combination of Elements)
• Text
• Image
• Sound
• Video
• Animation
Industries
• Creative and advertising
• Corporate communications visuals
• Architectural imaging
• Cultural heritage preservation
• Scientific achieve

Introduction
Multimedia (Combination of Elements)
• Text
• Image
• Sound
• Video
• Animation
• Touch
• Scent
• Taste
• Sight
Industries
• Creative and advertising
• Corporate communications visuals
• Architectural imaging
• Cultural heritage preservation
• Scientific achieve
Development /Trend
• Augmented and Virtual Reality
• Human Computer Interaction

Figure: a vision about 2000 imagined in 1910, through 24 illustrations by Villemard.

Desk /Table Pocket /Palm /Hand Head /Eye Ubiquitous
Major changes in computing platforms

Virtual Reality
Figure: NASA, 1988 Virtual Environment System

Figure: Super cockpit project, Tom Furness.

Near-eye-display and fully augmented interactive display experience
How far can we go?

Augmented Reality

Augmented Reality: Glass
Exploration in modern wearable interface
• Fashionable, daily life, UX practicality?

Google Glass
• Google Glass, developer version, 2013
• Can be configured on user’s customized or personal spectacles.

Recon Jet
• Sporty design
• Cost “friendly”
• Smartphone compatible
• External Connectivity
Presentation: Zi Siang See et al.https://www.youtube.com/watch?v=BhBtuNe_8is

• Displays, tracers, and AR systems in general need to become more accurate, lighter,
cheaper, and less power consuming (Azuma, 2001).
Figure: wearable computing, Steve Mann.

Ideas worth spreading:
How can we develop extended
reality (XR) content or systems
for good purposes and inclusive?

Mixed Reality brings effective collaborative outcome?
• Effects of Sharing Physiological States of Players in a Collaborative Virtual Reality
Gameplay. (Dey et al., 2017)
• Digital Taste Interface project, Electronic taste stimulation (Ranasinghe et al., 2011)
• Billinghurst, M. (2011). The Future of Augmented Reality in Our Everyday Life.

Google Cardboard

Google Cardboard
Programmable NFC
Tag
Low cost resin
/plastic lens
Mobile device as
computing module
Endless configurable size and
shapes (and materials)
Magnetometer

Google Cardboard v2
Presses the
screen

User Experience = full senses, realism, contextualization
https://www.youtube.com/watch?v=sfowhrpS-6M

User Experience

What do I see?
http://www.zisiangsee.com/vtour/vtour_hdr_sphericalpanorama/index.html Presentation: Zi Siang See et al.https://www.youtube.com/watch?v=hjKd24UCPYY

Well, in a Social Learning Way
Do you (do we):
• Do what you can’t?
• Do what you can?
• Lead others by example?
• Follow others?
• No idea…

Oculus Rift
Figure: Oculus, by Facebook

Android VR
Figure: Android VR collaborating with other developers
Google has announced eight hardware partners that will make Daydream-ready phones,
including Samsung, HTC, LG, Xiaomi, Huawei, ZTE, Asus, and Alcatel.

Apple + Metaio?
Metaio was one of the most technologically-ambitious AR organization, acquired by Apple in 2015

Apple’s Initiatives
SteamVR Coming To Mac (unveiled in WWDC 2017), AR Kit will be available too.

Steam VR
Presentation: Zi Siang SeeVinay Narayan - HTC Vive (https://visionsummit2017.com)

Definitions
Virtual Reality
• Virtual Reality (VR) which can be referred to as immersive multimedia or computer-
simulated life, replicates an environment that simulates physical presence in places in the
real world or imagined worlds and lets the user interact in that world. Virtual reality
artificially creates sensory experiences, which can even include sight, hearing, touch, and
smell.
Augmented Reality
• Augmented Reality (AR) mixes a live real-world view with virtual interactive content on a
mobile or wearable device. One of the key enablers for this is tracking technology, such
as computer vision techniques for tracking off pre-defined markers or markerless images.

Milgram’s Reality-Virtuality continuum
Mixed Reality
Reality - Virtuality (RV) Continuum
Real
Environment
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Environment
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays, IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

AR + VR + MR = XR?

Gartner’s Hype Cycle

Gartner’s Hype Cycle (Emerging Technologies)

Platforms
VR AR
Sourced and reinterpreted from
http://www.augment.com/blog/which-headset-is-right-for-you/

Case study: How can we combine
inclusive and interaction design?

Hair Raising Message
Presentation: Zi Siang See et al.https://www.youtube.com/watch?v=064ipuBiWDg

Practical consideration
• Ubiquitous computing – the installation of hardware are invisible to users
• Focus on user experience
• Has strong storyline
• Brings social good and awareness

Penguin Navi
Presentation: Zi Siang See et al.https://www.youtube.com/watch?v=IK4-zPD_25U

• Mix computing and entertainment
• High level of usability design
• Focus on bringing the intended outcome – going to the aquarium
• Make use of existing mobile devices and capability
• Practical, practical, practical!

How One Dad Is Using VR To Help His Daughter
https://uploadvr.com/vr-good-one-dad-williams-syndrome
Figure: Craig and Imogen Chaytor's Battle With Williams Syndrome
Presentation: Zi Siang See et al.https://www.youtube.com/watch?v=LOHvriN7Xk4

Empathy Glasses
Katsutoshi Masai, Kai Kunze, Maki sugimoto, Mark Billinghurst (2016) Empathy Glasses.
Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems
Presentation: Zi Siang See et al.https://www.youtube.com/watch?v=CdgWVDbMwp4

• Emphatic computing
• Shared experience, emotion, same viewpoint
• Real-time
• “Put myself in their shoes”
• Achieve and complete tasks together

Mobile Audible AR
Zi Siang See, Mark Billinghurst, Vimalan Rengganaten, Shelly Soo (2016) Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality.
Siggraph Asia 2016 Mobile Graphics and Interactive Application
Presentation: Zi Siang See et al.https://www.youtube.com/watch?v=VGsaTDKC2qY

Virtual Reality in Medicine Leadning
University of Basel, Switzerland
https://www.unibas.ch/en/News-Events/News/Uni-Research/Virtual-Reality-in-Medicine.html
Presentation: Zi Siang See et al.https://youtu.be/AttXbcLUyR0

Journey to Birth
Presentation: Zi Siang See et al.https://www.youtube.com/watch?v=7s3dP8QIyjc

• A key initiative by University of Newcastle
• Midwifery and health sciences education
• Can be used for educating mothers
• Reduce confusion and provides clear understanding
• Fit to the idea of portability in mixed reality context
• Designed by midwifes for midwifes
• Multi-disciplinary research

Work Smarter (Collaboration and Outcome-orientated)
Figure: Elevator technicians are using AR to improve safety and workflow
https://uploadvr.com/microsoft-hololens-elevator-technicians/

VRPhysio
Presentation: Zi Siang See et al.https://www.youtube.com/watch?v=WmSyWb6djDU

VRPhysio
• Customizable
• Clinic can track and help patients remotely
• Analytics
Presentation: Zi Siang See et al.https://www.youtube.com/watch?v=WmSyWb6djDU

54% Want To Use VR/AR In Physical Stores
• 85% -- Travel experiences
• 84% -- Museums and galleries
• 84% -- Education, training
• 83% -- For designing something
• 79% -- Live entertainment
• 77% -- Exercise or sport
• 75% -- Navigation
• 74% -- Retail and shopping
• 70% -- Business
• 69% -- Meditation
• 63% -- Fully immersive gaming
• 53% -- Gaming through mobile
• Comprised interviews of 16,000 consumers.
• Consumer's perspective: the main reason for not having tried VR or AR is the lack of
opportunity, based on a new study.
Presentation: Zi Siang See et al.https://www.mediapost.com/publications/article/301863/54-want-to-use-vrar-in-physical-stores.html

Implication
Further reading (by Prof Mark Billinghurst): https://www.slideshare.net/marknb00/comp-4026-lecture1-introduction
• Designing for “Everyone” pleases no one (Mark and Gun’s Advanced HCI slide, 2018)
• Is managing user experience important to your project?

Research Possibilities
Exploration
• Method and apparatus for augmented reality
• Method and apparatus for virtual reality
• Ubiquitous computing
• Obstacles and issues
• Technology management
• Education technology
• Engineering and concept design
• User interface (UI) design
• User experience (UX)

Can we use VR/AR technologies to help more people?
Effort
• Empathic computing
• Inclusive AR and VR
• Experience synchronization
• Improve well-being
• Solving conflicts?
• Medical-based and healthcare benefits
• Archeology, cultural heritage: see the whole story in 360 (digital preservation)
• Sharing community and social stories
• Combine interest of different professional disciplines

MOOC?
Zi Siang See
Research: Zi Siang See, Xia Sheng Lee, Adam Brimo, Harold Thwaites, Lizbeth Goodman

Focus of this work
Focus of this Study
• Case study of massive open online course (MOOC) for AR VR.
• Obstacles, challenges and usability issues (based on the MOOCs we developed).
• Investigation of how MOOC addresses best practices and the fundamental
requirements of AR and VR training, and direction of future AR/VR-enabled MOOC.
MOOC for AR VR Training: Obstacles, Challenges and Usability
Zi Siang See, Xia Sheng Lee, Adam Brimo, Harold Thwaites, Lizbeth Goodman (2018) MOOC for AR VR Training: Obstacles, Challenges and Usability.
IEEE Games, Entertainment & Media Conference, Galway, Ireland 15th-17th August 2018
Research &
collaboration

Our MOOCS
AR VR The Promise of Sci-Fi
• http://www.openlearning.com/sunway-university
BIM101
• https://www.openlearning.com/courses/revit101basicbim
BIM101
• https://www.openlearning.com/courses/revit101basicbim

Lizbeth Goodman
Professor of Inclusive Design
SMARTlab, University College Dublin
Zi Siang See
MOOC: AR VR The Promise of Sci-fi
SMARTlab, University of Newcastle
Xia Sheng Lee
MOOC: BIM101, BIM202
University of Reading
Adam Brimo
Founder & CEO
OpenLearning.com
Harold Thwaites
Professor, Digital Media
CRCDM, Sunway University

• MOOC brings a number of benefits (Chauhan et al., 2015). With right design/workflow of
MOOC, instructors can support the learning process rather than take control of it.
Background
Contribution & intention - to provide an investigation on how some of fundamental criteria of AR and VR can be catered in MOOC setting.

Augmented Reality
• Handheld AR display
• AR Browser (HTML5 + WebVR)
• Unity Vuforia SDK
• Visual Programming
• Virtual Reality
• Mobile VR display
• Samsung Gear VR + phone
• Unity SteamVR SDK
• Visual Programming
• Building Information Modelling
• Authoring Development
• Revit
• May link to AR + VR
Technologies involved in MOOC(s)

Case Study and Approach
• Classroom is vital for hands-on scenarios, however it may not be convenient for everyone
• MOOC is accessible 24/7, but it requires effort (students & instructor) to solve non-
standardized technical questions. (e.g. AR/VR, BIM, other tech-based development pace)
• Unique characteristics of MOOC includes the option of paid or free certification, on-
demand customized content dissimilar to conventional university syllabus.
• Question: formal education vs social way of learning?

Basic Goals
• Image-based AR tracking
• Way Finding Experience
• Authoring Conventional Multimedia
• Interaction and GUI
High-end Goals
• 3D object tracking
• Markerless Tracking (e.g. SLAM)
• Authoring AR-optimized multimedia
• Customizing mechanics in AR
Augmented Reality: Recommended Aspects to Cover in MOOC

Basic Goals
Image-based tracking allows AR content to be
overlayed on top of a 2D surface. This can be a
magazine cover, book or a billboard. Some
portal-based AR service providers provide
features of overlaying text, image, video etc.

Basic Goals
Way-finding AR uses sensors (GPS, compass,
gyro, wifi, mixed-mode) tracking approaches for
including enhanced location-based experience.
Such scenarios can be seen in the popular
Pokemon Go, Shopping AR Alert, Museum AR.

Basic Goals
Production processes: Image and video editing,
creative design process, sound montage, color
management, file compressions, web-based
content development, server hosting.

Basic Goals
Hand gesture input is essential in mobile-based
AR experience. It allows simple basic button to
be pushed (for linking social media, Youtube) or
scaling or rotating an object.

Basic Goals
High-end Goals
So far, we have no major problems
introducing basic goals in MOOC

Basic Goals
High-end Goals
What about these high-end goals
which may have special requirements?

Ideally, tracking real-world object is an
essential elements for AR experience.
However, it has been difficult to
implement in actual project.
High-end Goals

Markerless tracking is vital in the current
or next stage or AR experience. However, it
is only functionable in high-end mobile
devices. Lower range devices may suffer
from lack of computing power for
rendering such content and environment
tracking (Android ARCore vs Apple ARKIT)
High-end Goals

FACT – currently there are no standardized
ways to develop AR-enabled content.
Versions of authoring platforms keep
evolving (Unity 5 vs Unity 2018), and
Android and Apple has very different
development SDK requirements.
High-end Goals

Some AR features require customized
hardware and software aspects. These
emerging media experience may only be
suitable for resourceful organizations or
creative projects.
High-end Goals

Q: can we still introduce high-end goals of
AR authoring in a MOOC?

Basic Goals
• Environment simulation
• VR360 (Spherical Panorama)
• Locomotion, interactions
High-end Goals
• Presence
• Hands or Avatar (self realisation)
• Ambisonic audio
• Authoring hyper-realistic VR scenes
Virtual Reality: Recommended Aspects to Cover in MOOC

Basic Goals
It is a visual programming creation process.
Sometimes, also known as level design from the
game development perspective, it involves a
number of skills and technical knowledge during
the creation process (3D modeling,
photogrammetry, shader, texture, BIM)

Basic Goals
The idea being expressed during the early 19
centuries of a panorama painter’s goal was to
reproduce the real world so realistically that the
onlookers could believe that what they were
seeing was real (Benosman and Kang, 2001).
Creating spherical panorama and HDR are core
areas in computational photography.

Basic Goals
Lomotion, means how users are allowed to
“walked around” in virtual reality (3dof or 6dof).
SteamVR allows users to have room-scale
locomotion experience, this is an evolving aspect
in VR tech development. Interactions such as
working with objects in virtual scenes are
essential.

Basic Goals
Work with text, image, animation, video and
sound. These process may be alright for
multimedia-ready students or professionals,
however it involves a range of know-how
technical understanding for creating creative
multimedia content.

Basic Goals
High-end Goals
• Presence
• Ambisonic audio
Some students find it difficult to learn
these in MOOC vs classroom

Basic Goals
High-end Goals
• Presence
• Ambisonic audio
It may require specialized hardware and
software, and non-standardized knowledge

High-end Goals
• Presence
• Ambisonic audio
This is top priority in VR experience
development. However, it can be
extremely challenging to port in various
real-world elements into VR world. One of
the current and important topics in HCI is
“extended reality (XR)”.

High-end Goals
• Presence
• Ambisonic audio
This is top priority in VR (and AR)
experience development. However, it can
be extremely challenging to port in various
real-world elements into VR world. One of
the current and important topics in HCI is
“extended reality (XR)”.

High-end Goals
• Presence
• Ambisonic audio
Improving sense of presence involves a
number of elements and this is a key area
of research discussion, Hands VR and
Avatar for self-realisation are some of the
features developers could introduce in VR
experience. However, some of these are
challenging elements to be included in
MOOC for VR training, in the situation of
limited hardware access by students.
Q: introduce shared learning facility?

High-end Goals
• Presence
• Ambisonic audio
Ambisonic audio, is a specialized area in
VR360 video or VR content development.
Unity 2018 supports ambisonic audio
decoder, but may only work with
specialized settings.
Note: we need Matthew Sansom and team
to discuss about this technical area.

High-end Goals
• Presence
• Ambisonic audio
One of the key areas in VR. This implies
bringing in almost any real-world elements
we could find, and attempt to implement it
into virtual environment. This includes
managing lighting, shadow, plants, terrain,
water, wind, texture, dynamic, physics.

Q: How can we mix AR and VR in a
creative project?

• Sustainable learning group.
• Resembles the behaviors of learners
similar to casual interaction within
social media platforms.
• For now, it was found that a number of
users are unable to make positive
progress in AR VR MOOC due to a
number of reasons.
• Can the problem of lack of
hardware/software be solved?
• Can learners and instructor easily solve
technical problems in MOOC? Interactions of learners in 3 months
Discussion

• Collaborative learning
• A near-linear and stable learning
community if the subject is still valid.
• New and old learners.
• Can MOOC platform evolve to support
native AR/VR functions?
• Can MOOC of AR, VR, MR, VDC, BIM
subjects being introduced as “non-
conventional special content?
• Will MOOC platforms support and
produce “Netflix original” content?
Interactions of learners in 5 years, VDC/BIM subject, involved about 5000 learners.
Discussion

• Livelihood of supporting learning
community that is sustainable and
scalable in MOOC platform.
• Example: this figure demonstrates
graphically a MOOC’s global diversity of
learner profiles from 152 countries,
involving 4903 learners.
• In the future, the case study will also
continue to facilitate and experiment
with hybrid facilities such as 3D
scanning and VR avatar.
• We are contactable and reachable for
INCLUSIVE collaborations. Diversity of students, VDC/BIM subject, involved about 5000 learners.
Implication

Questions in Mind for Future Studies
• How can we make MOOC with AR and VR inclusive (for everyone)?
• How can we introduce future learning center or facility for aiding AR/VR-enabled MOOC?

AR VR Demo
Zi Siang See

Demo (work-in-progress): Inclusive VR Museum VR
• Electronic Cultural Atlas Initiative (ECAI)
• Towards inclusive user experience and digital preservation of cultural artifacts

• Inclusive contributors (artists, creators, more)
• Customization

• Co-design with local contributors (Newcastle Library, Hunter Water)
• Brings value to community, engagement, long-term involvement

Demo: Museum VR: The Tomb of Sultan
• 3D Scanned /Photogrammetry of artifacts
• User experience –revisiting an ancient site where the king rested.

Demo: Museum VR: The Tomb of Sultan
• Interaction: Can we see the hands? And can we gain some control? Physics?

Presentation: Zi Siang SeePresentation: Zi Siang See et al.

Creative Possibilities?

Example Setting
• Mobile Devices
• HMD compatibility
• Cost-effective
• Transmedia
• Creative Branding
• Public access
Zi Siang See, Mohd Shahrizal Sunar, Mark Billinghurst, Arindam Dey, Delas Santano, Human Esmaeili, Harold Thwaites (2017) An Augmented Reality and Virtual Reality Pillar for Exhibitions: A Subjective Exploration.
International Conference on Artificial Reality and Telexistence & Eurographics Symposium on Virtual Environments.
Research Presentation: Zi Siang See

Will OpenLearning be inclusive and XR-enabled?
Researchers: Zi Siang See, Xia Sheng Lee

Ideas for AR and VR? Start building yours :)

Thank you / Discussion
Zi Siang See

Extended Reality for Virtual Social Learning

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Extended Reality for Virtual Social Learning

Similaire à Extended Reality for Virtual Social Learning (20)

Dernier

Dernier (20)

Extended Reality for Virtual Social Learning