Kris Kitani from Carnegie Mellon University and Chieko Asakawa, IBM Fellow, presented a “Cognitive Assistant for the Blind” as part of the Cognitive Systems Institute Speaker Series.

1. Cognitive Assistant for the Blind
Chieko Asakawa,
IBM Fellow,
IBM Research
Kris Kitani,
Robotics Institute,
Carnegie Mellon University
1

2. © 2016 IBM Corporation
18th century, the mathematician Euler authored a
leading textbook on mathematics after he became
blind.
Captions for TV programs were
invented for deaf people, but
are now widely used in sports
bars and to help immigrants,
etc.
Based on Self-‐‑‒standing
wheelchair.
2011, voice dialog
technologies for
smartphones
TelephoneKeyboard
Voice recognition
Voice dialogTV Captions
Voice
synthesis
and OCR
Modern Mathematics
Automatic vehicles
Segway
“Blind driving challenges”
Early use of keyboards was
a to help people with hand-‐‑‒
and-‐‑‒motor disabilities in
writing
First consumer product was a
reading assistant for the blind
in the 1970s.
1876, Alexander Graham Bell invented the
phone after teaching hearing impaired
people.
Smart
Machines
2
17th century,
Dom Perignon. Champagne
Accessibility Needs Ignite Innovation

3. 1960s
Talking
Typewriter
1960s Talking Typewriter
1975 1403 Braille Printer
1984 Talking 3270 Terminal
1990 VoiceType™
2008 Social Accessibility
1998 ViaVoice®
2000 Accessibility Center
1997 Home Page Reader
1988 ScreenReader/DOS
2004 aDesigner
2007 Eclipse Accessibility Tools Framework
2009 Rich Internet Accessibility Technologies
1914 First IBMer with disabilities
2012 Brazil Accessible Vocational Training Project
2011 Senior Workforce Project
1975
1403
Braille
Printer
IBM History of Accessibility
3 © 2016 IBM Corporation

4. Progress in Information Accessibility
Exponential increase of accessible information
(Conceptual chart)
1980 1990 2000 2010
Paper Braille
Digital Braille
Voice Web
access
4
Amountofaccessibleinformation
© 2016 IBM Corporation

5. 5
Walk by myself
Finding discount posters
Finding restaurants
Shopping
Many things are difficult…
Challenges of Real World Accessibility
© 2016 IBM Corporation

6. Chika
6
Chika from “Esper the Light Speed”
© 2016 IBM Corporation

7. Natural
Language
Processing
Infer from
Knowledge
Predict the
Future
Natural
Interaction
Learning
Senses
Watson 1.0, 2.0, …
Apply Watson
Modeling
and
Simulation
Neural Chip
SyNAPSE
Image
Processing
Speech
Recognition
Technology
Traffic Simulation
Dialog System
Question and Answering System
7
Challenges of “Cognitive Computer”
© 2016 IBM Corporation

8. 8
Demonstration Video
© 2016 IBM Corporation

9. Localization
Radio-wave
Vision
Dead reckoning
Recognition
Objects
People
Environment
Knowledge
Personal
Social media
Encyclopedic
Interaction
Speech
Gesture
Sonification
Building Blocks of Cognitive Assistant
9 © 2016 IBM Corporation

10. Understanding the Functionality of the Environment
Nick Rhinehart and Kris Kitani, CVPR 2016

11. © 2016 IBM Corporation
When we observe a scene…
… we know how we can act in that environment

12. © 2016 IBM Corporation
When we observe a scene…
… we know how we can act in that environment
Walk
Walk

13. © 2016 IBM Corporation
When we observe a scene…
… we know how we can act in that environment
Sit Sit
Sit

14. © 2016 IBM Corporation
When we observe a scene…
… we know how we can act in that environment
Open
Open
Open

15. © 2016 IBM Corporation
Can we teach a computer
to
understand scene functionality
by
observing people?

16. © 2016 IBM Corporation
Setup:
One wearable camera
Detect actions using computer vision
Learn the function of an environment

17. © 2016 IBM Corporation

18. © 2016 IBM Corporation

19. © 2016 IBM Corporation

20. © 2016 IBM Corporation

21. 21
Information Accessibility Progress (revisited)
1980 1990 2000 2010
Paper Braille
Digital Braille
Voice Web
access
Amountofaccessibleinformation
Exponential increase of accessible information
(Conceptual chart)
2020 2030
2016
© 2016 IBM Corporation
“Real-world” information access