Cognitive Assistant
for The Real-world Accessibility
Chieko Asakawa
IBM Fellow, IBM Research
IBM Distinguished Service Professor,
Carnegie Mellon University

© 2011 IBM Corporation2
Cognitive Ability of Visually Impaired People
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.8
1.0
1.2
1.4
1.6
1.8
2.0
2.2
2.4
2.6
2.8
3.0
3.2
3.4
3.6
音声速度(倍速)
再生率(%) A
B
C
D
E
F
G
RecallRate(%)
Speech speed
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X

3 © 2016 IBM Corporation

4 © 2016 IBM Corporation
Information accessibility Mobility

Braille Digitalization
Paper Braille Typewriter
Comparison:
Braille and Printed
Japanese Dictionary
Digital Braille System

Home Page Reader
6 © 2016 IBM Corporation
nThe first practical voice browsers in the
world
nBecame a product in 1997 and translated
into 11 languages
IBM Home Page Reader (1999)
Japanese, Italian, French, German, Spanish, English (U.S.,
U.K.)
“Home Page Reader is my small window
to the world.
I can read, write, and access information.
I can do everything to participate in
society.”
— comment from a user, 1997

© 2016 IBM Corporation

Smart phone demo
© 2016 IBM Corporation

Exponential Increase of Accessible Information
(Conceptual Chart)
© 2016 IBM Corporation
Paper Braille
Digital Braille
Voice Web
access
1980 20001990 2010
Amountof
accessibleinformation
9

https://www.youtube.com/watch?v=cTHa_5Y5IA4
10
https://theses.lib.vt.edu/theses/available/etd-
05072012-
161514/unrestricted/DAngio_PC_D_2012_1.pdf
http://www.ted.com/talks/dennis_hong_m
aking_a_car_for_blind_drivers
Blind Driver Challenge at the Rolex 24 At Daytona with Audio Description

Uber Self-driving Car Pilot
11 © 2016 IBM Corporation

Walk by myself Finding discount signsFinding restaurantsShopping
The Second Challenge: Real World Accessibility
© 2016 IBM Corporation12

Cognitive Assistant
13
Augment missing or weakened abilities
by the power of cognitive computing.
© 2016 IBM Corporation

“甦るヒーローライブラリー第3集 光速エスパー”, TCエンタテインメントより
Chika
14 © 2016 IBM Corporation

© 2016 IBM Corporation
https://www.ted.com/talks/chieko_asakawa_how_new_technology_helps_blind_people_explore_the_world

Foundation of Cognitive Assistant
© 2016 IBM Corporation16
Localization
Radio-wave
Vision
Dead reckoning
Recognition
Objects
People
Environment
Knowledge
Personal
Social media
Encyclopedic
Interaction
Speech
Gesture
Sonification

Localization
Quasi-zenith satellites
for Japan
2018
http://qzss.go.jp/technical/technology/tech01_orbit.html
Accurate indoor localization
BLE Beacons + Dead reckoning
WIFI

Recognition of Environment
Vision-based Environment
Recognition
Infrared Depth Sensor on smartphones
Google Tango
https://youtu.be/Qe10ExwzCqk

Recognition of Things
Watson Vision API
coffee
espresso
wine
glass
glasses pottery
dish

Oil filter
Chicken soup
Pen sharpener Spice bottle
Cristmas socks
Cheetos
Beer bottle Diet coke
Photos for learning taken by blind
subjects
Incorrect results
Before leaning
Correct results
After learning
Personal Object Recognition
Honorable mention at CHI2017

© 2016 IBM Corporation
Knowledge – Importance of Open data
Detailed pedestrian maps
Crowd prediction
Restaurant
Shops
Items
Temporal changes
21
Accessibility information

Interaction
© 2016 IBM Corporation
Approaching to human accuracy
http://blogs.microsoft.com/next/2016/10/18/historic-achievement-
microsoft-researchers-reach-human-parity-conversational-speech-
recognition/#sm.0019gj85a12umd89ys32kew39p8d6
Autonomous Bus with Voice Dialog
https://www-03.ibm.com/press/jp/ja/pressrelease/50250.wss

デモ
© 2016 IBM Corporation

Cognitive Assisting Robot
Manuela Veloso is leading the CoBot project.
On November 18, 2014, the CoBot robots jointly reached
1,000km of autonomous localization and navigation.

© 2016 IBM Corporation
Sep 2016 - August 2019
Awarded Amount: $1million
Investigator:
• Kris Kitani, Computer Vision, Robotics Institute, CMU
• Manuela Veloso, Robotics, Robotics Institute, CMU
Advisor:
• Chieko Asakawa, IBM Fellow, Robotics Institute, CMU
National Robotics Initiative
A Cognitive Navigation Assistant for the Blind
25
Implementation of a navigation assistant that uses a collection of
sensing modalities and algorithms to guide a blind person through the
knowledge landscape of an unfamiliar environment.

World Robot Summit
© 2016 IBM Corporation
Industry robots Services robots Disaster response robots
26
Japanese government is planning a robot competition event in 2020, the same year,
Olympics and Paralympics will be held in Tokyo.
http://www.nedo.go.jp/robot/about.html http://www.miraikan.jst.go.jp/exhibition/future/robot/robotworld.html#sl
ide1
https://www-
robotics.jpl.nasa.gov/tasks/taskImage.cfm?TaskID=236&t
daID=700043&Image=501

18th century, the mathematician Euler authored a
leading textbook on mathematics after he became
blind.
Captions for TV programs were
invented for deaf people, but
are now widely used in sports
bars and to help immigrants,
etc.
Based on Self-standing
wheelchair.
2011, voice dialog
technologies for
smartphones
TelephoneKeyboard
Voice recognition
Voice dialog
TV Captions
Voice
synthesis
and OCR
Modern Mathematics
Autonomic vehicles
Segway
“Blind driving challenges”
Early use of keyboards was
a to help people with
hand-and-motor disabilities
in writing
First consumer product was a
reading assistant for the blind
in the 1970s.
1876, Alexander Graham Bell invented the
phone after teaching hearing impaired
people.
Innovations Flourish From Accessibility Needs
Smart
Machines
27
17th century,
Dom Perignon. Champagne
© 2016 IBM Corporation

1960s
Talking Typewriter
1960s Talking Typewriter
1975 1403 Braille Printer
1984 Talking 3270 Terminal
1990 VoiceType™
2008 Social Accessibility
1998 ViaVoice®
2000 Accessibility Center
1997 Home Page Reader
1988 ScreenReader/DOS
2004 aDesigner
2007 Eclipse Accessibility Tools Framework
2009 Rich Internet Accessibility Technologies
1914 First IBMer with disabilities
IBM History of Accessibility
28 © 2016 IBM Corporation
2012 Brazil Accessible Vocational Training Project
2011 Senior Workforce Project 1984
Talking 3270 Terminal

© 2016 IBM Corporation
2010 20302020
Amountof
Accessiblerealworldinformation
(conceptualnumber)
Future: Real-world Accessibility – Exponential Growth
Cognitive Assistant Lab @ CMU
Smartphone accessibility
Standardization?Core technology
development
Mobile Cognitive ?
Apple Siri
Microsoft Seeing AI project
Google Tango Phone
Tokyo Nihonbashi
Demo
Watson Vision API
Adoption?Early days
2016
Dec.
Tokyo Olympics
Paralympics