HMD shipments are forecast to grow rapidly over the next few years, reaching around 76 million units by 2020. Immersive computing technologies like virtual reality, augmented reality and mixed reality are poised for growth as they blend physical and digital worlds and allow for natural language and gesture-based interactions. Developers can create immersive applications for these platforms across entertainment, training, manufacturing and other areas using tools like Unity, Windows Mixed Reality and Azure cognitive services.
3. The future: Immersive
Computing
• Natural language and interactions between
people and technology
• Bots and agents
• VR, AR and MR
2000s: Mobile
• Social
• User download apps
from App Stores
1990s: Internet
• Search
• User “visits” websites
1980s: PC
• Desktop
4. These new experiences are poised for rapid growth
HMD shipments to reach ~76M units by 2020 growing at 65% CAGR
Source: IDC, Worldwide Quarterly Augmented and Virtual Reality Headset Tracker, 3Q16
196,722
1,937,954
10,252,907
22,672,192
37,674,548
54,003,645
75,972,265
0
10,000,000
20,000,000
30,000,000
40,000,000
50,000,000
60,000,000
70,000,000
80,000,000
2014 2015 2016 2017 2018 2019 2020
7xgrowth of market
forecasted through
2020
6. The state of things as they
“actually exist” through our human
senses without any technology
Artificially created sensory experiences of
people, environments and objects, which
can include sight, touch, hearing, and smell
PHYSICAL REALITY DIGITAL REALITY
13. Virtual Reality
Virtual Reality immerses the wearer of the
HMD within virtual worlds
Virtual Reality HMD’s typically require the use of
a PC or phone and in some cases, extensive
hardware within the room to provide the wearer
the illusion of presence within the virtual world
17. Augmented Reality
Where Virtual Reality (VR) replaces
the physical reality, Augmented
Reality (AR) augments or modifies
the physical reality
Often Augmented Reality is intended
to enhance the perception of the
physical reality, such as within an
aircrafts Heads Up Display (HUD)
20. Azure Computer Vision API
Tag images based on content
Categorize images
Identify the type and quality of images
Detect human faces and return their coordinates
Recognize domain-specific content.
Generate descriptions of the content
Use optical character recognition to identify text
found in images
Distinguish color schemes
Flag adult content
Crop photos to be used as thumbnails
Animal_dog
People
People_crowd
Outdoor_mountainFood_bread
Distill actionable information from images
23. Windows Mixed Reality headsets
Head mounted displays (HMDs) that provide mixed reality experiences by leveraging the
native Windows Mixed Reality experiences in Windows 10
26. Immersive headsets
So easy to set up: Works out-of-box
More freedom to place the system where you want
Wide range of affordable headsets and PCs
The largest apps library in the category 20k+
33. Developing Windows Mixed Reality Apps
Basic interaction model
Environmental understanding features
2D apps
Immersive and holographic apps
Windows Store
34. Areas of opportunity
Creation &
Design
Training &
Development
Assembly &
Manufacturing
Communication &
Understanding
Entertainment &
Engagement
37. Immersive Solution
Immersive App
(Unity UWP)
Immersive App
(Unity UWP)
2D App
(cross-platform)
Dashboards &
Visualizations
IntelligenceInformation
Management
Big Data Stores Machine
Learning
& Analytics
Power BI
Bot
Service
Data
Catalog
SQL Data
Warehouse
Data Lake
Analytics
Cognitive
Services
Data
Factory
Data Lake
Store
Machine
Learning
CortanaEvent Hubs HDInsight
Stream
Analytics
Cosmos DB
Cloud
(Azure)
This slide focus on HCI – human computer interaction, that is, how humans have interacted with computers throughout the last 3 decades.
This slide sets the stage for the presentation, illustrating the journey we have been on since the 1980’s with the introduction of the PC, the 1990’s with the introduction of the Internet and into the 2000’s with Mobile computing. The present and future is all about Immersive Computing and related technologies. In the following sections we’ll briefly explore VR and AR before exploring Mixed Reality and the opportunity it represents for our ISV partners.
Rick (22)
Classic chicken and egg problem e.g. no devices, no apps
More HMDs are being sold, experiences are getting better
Get in the game and get ahead of your competition
Reza
Before we jump into our session, I wanted to take a moment to give a high-level overview of this program.
The session were in today is an Envisioning session, where we talk about the latest trends and how to capitalize on these trends.
We highly encourage you all to follow through to the other two phases as well though.
The design sessions are led by Microsoft technical experts and dive into architectural considerations and design best practices for incorporating these technologies and
finally the Accelerate phase is where you have an opportunity to work side by side with Microsoft technical experts on building out your specific solution
And something unique about this program is that we have allocated a special offer for companies going through this program to help you kickstart your projects, which I’ll dive deeper into that at the end
“Immersive technology refers to technology that blurs the line between the physical world and digital or simulated world”
“Immersive technology refers to technology that blurs the line between the physical world and digital or simulated world”
Immersive experiences consist of 3 key pillars – perception, interaction, and environment.
Perception (human senses; but of course current technology doesn’t do much with smell and taste, yet; but there are some examples)
vision (3D display, holography, head-mounted display, full-dome)
auditory (3D audio, surround sound, etc.)
tactile (haptic)
olfaction (smell)
gustation (taste)
Environment – creation and management of the immersive space for the immersive experience, whether it is a single user, private experience, or one that is shared among many users. Just like mobile applications today, this often means integration with server-side resources and services.
This is dangerous for now
Transition statement: "That gets us today and lets talk about some of the areas we see immersive computing happening today and more specifically the devices we see those experiences on."
Ryan (10-21)
Virtual Reality – “An artificial, computer-generated world that can be experienced and interacted with”
Eye balls are covered up; completely immersed; you have been transported elsewhere; you are taking out of the physical world
Sega Master System 3D glasses
2012 Oculus with a kick start campaign; 4 years ; later we had a consumer ready device.
HTC Vive
Gear VR – head rotation
Playstation VR
Virtual Reality typically requires a PC, which is OK, but it also requires the use of external sensors (cameras and base stations) to track presence in a space
Track light and dark spots; highly details math
http://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?uimode=nextgeneration&eventid=1402899&sessionid=1&key=81802DAB649756CFCCB96A7067D4E07F&contenttype=A&eventuserid=305999&playerwidth=1000&playerheight=650&caller=previewLobby&text_language_id=en&format=fhaudio
Examples virtual reality
Discuss the wide range of devices and experiences they deliver. Make note that these VR HMD’s are all fully occluded, so are 100% digital reality
RICK
Why they are relevant… and perhaps industry antidotes
Use case – GIBLIB is company that specializes in surgical videos. They recently launched a VR service that allows medical students, physicians, etc to live stream a surgery in full 360 virtual reality
Video stream
Connect to video stream
Media Services
Next slide – why msft; lightweight non-commercial; linux is not; osx is not the go to;
You stay in your physical world but your physical world is augemented; it has data layed over the top of ypur physical reality;
A phone up ; a tablet up
magical window; data overlayed; they are still in their physical world;
Pokemon;
AR is more accessible because it doesn’t necessarily require dedicated specialized hardware – there are many experiences that can be built into a simply smartphone – but it can also go all the way up to providing real time information about the world around you through object recognition and computer vision
Cog services
RYAN -
Sprinkle Azure; IKEA inventory; graphics; RESTUL; database no-sql / SQL
Identify an object; layering cloud data
David
Beyond the case studies some other things you might want to do with computer vision include working out what’s in the photo right? That way you can write some code that does intelligent things based on the content of the image. This could be for moderation where you don’t want to see certain types of images, or perhaps you want to intelligently crop photos?
The Vision API is a great place to start solving these problems.
You stay in your physical world but your physical world is augemented; it has data layed over the top of ypur physical reality;
A phone up ; a tablet up
magical window; data overlayed; they are still in their physical world;
Pokemon;
It’s the magic behind the experiences available across our OEM and HoloLens devices. It’s the platform that allows for rich user experiences that merge our real world and our digital world
(FTE note: similar to Windows Hello – we could imagine Windows Mixed Reality running across a variety of mobile or desktop devices).
Mixed Reality is the future of computing and Windows makes jumping in easier than ever before
Head-Mounted Display Hardware: Microsoft HoloLens is the first mixed reality device to track user movement across six degrees of freedom
User Experience: Windows 10 is the first operating system designed to operate the way people do
Consumer Applications: The world’s largest catalog of apps grows with new franchise titles and experiences on Xbox and Windows
Commercial Applications: A brand new platform for global institutions and government to transform the way we work and achieve more
3D Images and 360 Video: Windows 10 is the world’s leading platform for beautiful image capture and viewing in three dimensions
Fixed color on header
Animate + gradient to show hardware mix and blend
RICK -
Sprinkle Azure; IKEA inventory; graphics; RESTUL; database no-sql / SQL
Identify an object; layering cloud data
Transition: "Ryan, no MSFT presentation would be complete with talking about the cloud, Azure."
Tethered; inside / outside tracking; no base stations and external cameras
3 DOF – head rotation; x,y,z
6 DOF – body movement – lateral movement; forward, back, left and right, up, down
Degrees of Freedom
Freedom of movement: Experience virtual reality in any room of your house. Experience complete freedom with 6DOF tracking.
Effortless Setup: Works right out of the box, no need to drill holes or setup extraneous hardware. Windows does all the work.
Single, Consistent User Interface: Windows provides a consistent interface across varied Mixed Reality devices.
One SDK for many devices: Build apps using the familiar Windows development tools you already know and love, and have them run across multiple HMDs.
More consumer choice: With HoloLens and multiple OEM partners, Windows Mixed Reality is available on a wide range of devices, giving the consumer more freedom.
More sales channels: Monetize and promote your app from day one in the Windows Store, merchandising in a Store optimized for VR and holographic apps.
More customers: By targeting a larger variety of PC specs, Windows Mixed Reality opens the door to reach more consumers than ever before.
Transition: "Rick, and we are seeing rapid growth in the market aren't we."
One of the advantages of developing apps for Windows Mixed Reality is that there is a spectrum of experiences that the platform can support. From fully immersive, virtual environments, to light information layering over a user’s current environment, Windows Mixed Reality provides a robust set of tools to bring any experience to life. It is important for an app maker to understand early in their development process as to where along this spectrum their experience lies. This decision will ultimately impact both the app design makeup and the technological path for development.
Enhanced environment apps (HoloLens only)
One of the most powerful ways that mixed reality can bring value to users is by facilitating the placement of digital information or content in a user’s current environment. This is an enhanced environment app. This approach is popular for apps where the contextual placement of digital content in the real world is paramount and/or keeping the user’s real world environment “present” during their experience is key. This approach also allows users to easily move from real world tasks to digital tasks and back easily, lending even more credence to promise that the mixed reality apps the user sees are truly a part of their environment.
Blended environment apps
Given Windows Mixed Reality’s ability to recognize and map the user's environment, it is capable of creating a digital layer that can be completely overlaid on the user’s space. Thin layer respects the shape and boundaries of the user’s environment, but the app may choose to transform certain elements best suited to immerse the user in the app. This is called a blended environment app. Unlike an enhanced environment app, blended environment apps may only care enough about the environment to best use its makeup for encouraging specific user behavior (like encouraging movement or exploration) or by replacing elements with changes (a kitchen counter is virtually skinned to show a different tile pattern). This type of experience may even transform an element into an entirely different object, but still retain the rough dimensions of the object as its base (a kitchen island is transformed into a dumpster for a crime thriller game).
Immersive environment apps
Immersive environment apps are centered around an environment that completely changes the user’s world and can place them in a different time and space. These environments can feel very real, creating immersive and thrilling experiences that are only limited by the app creator’s imagination. Unlike blended environment apps, once Windows Mixed Reality identifies the user’s space, an immersive environment app may totally disregard the user’s current environment and replace it whole stock with one of its own. These experiences may also completely separate time and space, meaning a user could walk the streets of Rome in an immersive experience, while remaining relatively still in their real world space. Context of the real world environment may not be important to an immersive environment app.
Spatial mapping provides a detailed representation of real-world surfaces in the environment around the HoloLens, allowing developers to create a convincing mixed reality experience. By merging the real world with the virtual world, an application can make holograms seem real. Applications can also more naturally align with user expectations by providing familiar real-world behaviors and interactions.
Holograms don't need to stay private to just one user. Holographic apps may share spatial anchors from one HoloLens to another, enabling users to render a hologram at the same place in the real world across multiple devices.
Six questions to define shared scenarios
Before you begin designing for shared experiences, it’s important to define the target scenarios. These scenarios help clarify what you’re designing and establish a common vocabulary to help compare and contrast features required in your experience. Understanding the core problem, and the different avenues for solutions, is key to uncovering opportunities inherent in this new medium.
Through internal prototypes and explorations from our HoloLens partner agencies, we created six questions to help you define shared scenarios. These questions form a framework, not intended to be exhaustive, to help distill the important attributes of your scenarios.
Mixed reality experiences are enabled by new Windows features for environmental understanding. These enable developers to place a hologram in the real world, and allow users to move through digital worlds by literally walking about.
The basic interaction model for HoloLens is gaze, gesture and voice, sometimes referred to as GGV. All mixed reality devices benefit from the input ecosystem available to Windows, including mouse, keyboard, gamepads, and more. With HoloLens, hardware accessories are connected via Bluetooth. With immersive devices, accessories connect to PCs via Bluetooth, USB, and other supported protocols.
The environmental understanding features like coordinates, spatial sound and spatial mapping provide the necessary capabilities for mixing reality. Spatial mapping enables holograms to interact with both the user and the world around them. Coordinate systems allow the user's movement to affect movement in the digital world.
Holograms are made of light and sound, which rely on rendering. Understanding the experience of placement and persistence, as demonstrated in the mixed reality shell is a great way ground yourself in the user experience.
From scientists and technicians to designers and clients, five areas of opportunity have emerged where Microsoft partners are find value with mixed reality. These areas are already providing massive insight into the future needs of platforms like Windows Mixed Reality and can help you understand how these new experiences will impact the ways we learn, collaborate, communicate, and create.