Astute symposium 10/10/2013 - Keynote Kay Stanney

1. Sirris Symposium: Human Factors and
Technologies for Pro-active, Contextaware and Data-intensive Applications
Human-­‐Systems
Integra0on
in
Adap0ve
Mission
Cri0cal
Systems
Kay
Stanney,
Ph.D.,
C.H.F.P.
Design
Interac7ve,
Inc.,
President
&
Founder
University
of
Central
Florida,
Courtesy
Appt.
October
10,
2013

2. Agenda
Ø A
bit
about
me
and
Design
Interac7ve,
Inc.
Ø Human
Augmenta7on:
Essen7al
Emerging
Transforma7onal
Technology
Ø Past
Approaches
to
Human
Augmenta7on
§
§
Adap7ve
Automa7on
Augmented
Cogni7on
Ø The
Future
of
Human
Systems
Integra7on
Ø Conclusions

3. A
bit
about
me
and
Design
Interac9ve,
Inc.
17
October
2013
Design
Interac-ve,
founded
in
1998,
is
a
human
factors
engineering
firm
that
helps
clients
overcome
their
most
pressing
human
performance
challenges.
Unlike
most
firms,
we
use
deep
behavioral
and
physiological
diagnos-cs
to
design
adap-ve,
engaging
solu-ons
that
op-mize
performance
and
profoundly
enhance
the
user
experience.

4. DI
Divisions
DIVISION'
MARKET'POSITIONING'STATEMENT'
Defense&Solutions&
Our&Defense&Solutions&Division&provides&operational&analysis,&performance&assessment,&and&
advanced&technology&solutions&to&Department&of&Defense&clients&who&aim&to&enhance&training&
effectiveness&and&efficiency.&&We&use&deep&behavioral&and&physiological&diagnostics&to&deliver&
adaptive,&meaningful,&and&intuitive&learning&experiences&for&the&Warfighter.&&
Medical&Innovations& Our&Medical&Innovations&Division&provides&innovative&personal&health&solutions&for&medical&care&
providers&and&consumers.&We&combine&unobtrusive&biomonitoring&technology&with&adaptive&
assessment&solutions&that&continuously&analyze&collected&data&to&offer&preventative&and&corrective&
measures&in&any&setting.&&
Emerging&Markets&
and&Technologies&
&
Our&Emerging&Markets&Division&specializes&in&userDcentered&design&and&usability.&We&leverage&our&
cuttingDedge&military&R&D&to&develop&innovative&design&and&evaluation&tools,&human/machine&
interfaces,&and&smart&mobile&solutions&that&empower&users&and&enhance&the&user&experience.&&
Across
our
Divisions,
DI’s
solu0ons
save
lives,
reduce
cost,
enhance
the
user
experience,
and
op0mize
human
performance
-­‐
while
defining
the
future
of
human-­‐systems
integra0on.
17
October
2013

5. Emerging
DI
Products
–
Low
Cost
EEG

6.
Emerging
DI
Products
-­‐
New
Training
Solu9ons

7. Emerging
DI
Products
-­‐
Lessons
Learned
Tool
Playbook is a rapid authoring tool that can be
used to capture, publish, and share operational
observations, insights, and lessons (OIL).
Playbook provides an easy-to-use platform to
record and share personal experiences quickly
and effectively.

8. DI
Products
–
easyGaze
and
GazeWare

9. Emerging
DI
Products
–
STRAP
STRAP vest communicates a haptic
language based on military hand signals
Demonstrated rapid
retention rates
learning and high

10. Emerging
DI
Products
–
New
Evalua9on
Tools

11. Emerging
DI
Products
–
SIMI
Sensor
Suite

12. Rela9ng
DI’s
SIMI
to
the
ASTUTE
Project
Build
an
EEG-­‐based
Measure
of
Situa9on
Awareness
Sensor:
EEG
MeasureIT
Measure:
EEG
Alpha
&
Theta
Diagnose:
High
Theta
&
Low
Alpha
=
Low
SA

13. Human
Augmenta9on:
Essen9al
Emerging
Transforma9onal
Technology

14. Human
Augmenta9on:
Essen9al
Emerging
Transforma9onal
Technology
As
natural
human
capaci-es
become
increasingly
mismatched
to
data
volumes,
processing
capabili-es,
and
decision
speeds,
augmen-ng
human
performance
will
become
essen-al
for
gaining
the
benefits
that
other
technology
advances
can
offer.
Technology
Horizons:
A
Vision
for
Air
Force
Science
&
Technology
During
2010-­‐2030
Dr.
Werner
J.A.
Dahm
United
States
Air
Force
Chief
Scien7st
May
15,
2010
(p.
58)

15. Human
Augmenta9on
Essen9al
for
Gaining
Benefits
of
Emerging
Technology
Advances
Robo7cs
&
UAVs
VR
&
Virtual
Assistants
Predic7ve
&
Content
Analy7cs
Human
Augmenta7on
Biochips;
Health
Monitoring
Augmented
Reality;
Wearable
UIs

16. Emerging
Technologies
Human Augmentation
Emerging Technology

17. Emerging
Technologies
Priority
Mix
2013
Human
Augmenta7on
Transforma7onal

18. ASTUTE
Focused
on
Human
Augmenta9on
Ø ASTUTE
is
focused
on
the
transforma7onal
emerging
technology
of
human
augmenta7on
Ø Pro-­‐ac7ve
systems
are
one
approach
to
human
augmenta7on
Ø ASTUTE’s
goals
for
proac7ve
systems
are
to:
§
§
§
Measure
user
state
and
relate
it
to
context
Provide
pro-­‐ac7ve
sugges7ons
based
on
these
user
state
and
context
data
Thereby
realizing
adap7ve
HMIs
that
increase
situa7onal
awareness,
improve
decision
making,
and
augment
other
aspects
of
human
performance

19. Past
Approaches
to
Human
Augmenta9on
Adap9ve
Automa9on
&
Augmented
Cogni9on

20. Past
Approaches
to
Human
Augmenta9on
Ø Mission
cri7cal
systems
put
immense
pressure
on
human
cogni7on
Ø These
context
demand
swi`er,
highly
accurate,
and
ever
more
resilient
capabili7es
§
§
§
NASA
proposed
adap7ve
automa7on
as
a
means
to
address
such
demands
DARPA
proposed
augmented
cogni7on
as
new
HSI
paradigm
through
which
to
achieve
gains
in
mission
cri7cal
performance
These
efforts
can
inform
ASTUTE’s
proac7ve
systems
efforts

21. Past
Approaches
to
Human
Augmenta9on:
Adap9ve
Automa9on
Ø Rouse’s
(1998)
adap7ve
automa7on
theory
suggested
that
both
user
and
system
should
be
able
to
ini7ate
changes
in
the
level
of
system
automa7on
in
response
to
situa7onal
demands
Ø Systems
implemented
based
on
these
early
theories
generally
followed
a
binary
(on/off)
approach
to
adap7ve
automa7on
§
§
Some
relied
on
physiological
measures
of
operator
state
to
trigger
automa7on
Others
relied
on
task
or
context
based
measures
(e.g.,
cri7cal
events;
operator
performance;
task
models)
to
trigger
automa7on
ü ASTUTE’s
proac7ve
systems
plan
to
leverage
both
operator
state
&
context

22. Workload
Matched
Adap9ve
Automa9on
Source:
hep://www.docstoc.com/docs/99925451/Adap7ve-­‐Automa7on-­‐Matched-­‐to-­‐Human-­‐Mental-­‐Workload

23. Past
Approaches
to
Human
Augmenta9on:
Adap9ve
Automa9on
Ø Several
closed-­‐loop
solu7ons
evolved,
many
of
which
controlled
the
levels
of
task
automa7on
based
on
physiological
indices:
§
§
§
EEG
measures
(e.g.,
theta,
alpha,
beta,
and
gamma
band
ac7vity
to
develop
an
engagement
index)
Cardio-­‐circulatory
measures
(e.g.,
HR,
HRV)
Combina7on
of
physiological
indicators
(e.g.,
EEG,
ERPs,
and
HRV;
EEG,
HR,
respira7on
interval,
and
eye
blinks/
interblink
intervals)

24. Past
Approaches
to
Human
Augmenta9on:
Adap9ve
Automa9on
Ø Adap7ve
automa7on
o`en
7mes
substan7ally
improves
human
performance:
§
§
§
44%
reduc7on
in
tracking
task
errors
and
a
33%
reduc7on
in
error
rates
on
resource
management
tasks
(Wilson
&
Russell,
2003)
50%
improvement
on
UAV
opera7ons
(Wilson
&
Russell,
2007)
300%
improvement
in
throughput
and
detec7on
in
image
analysis
tasks
(Bloom,
et
al.,
2009)

25. Past
Approaches
to
Human
Augmenta9on:
Adap9ve
Automa9on
Ø Some
issues:
§
§
§
§
§
§
§
Unbalanced
mental
workload
Mistrust
Overreliance
Complacency
Insensi7ve
physiological
measures
Reduced
situa9on
awareness
Decision
biases
u ASTUTE’s
proac7ve
systems
seek
increased
SA
and
enhanced
decision
making

26. Past
Approaches
to
Human
Augmenta9on:
Adap9ve
Automa9on
è
Augmented
Cogni9on
Ø Notwithstanding
challenges
and
barriers
presented
by
real-­‐7me
psychophysiological
monitoring,
early
work
in
adap7ve
automa7on
demonstrated
immense
poten7al
derived
through
systema7c
integra7on
of
operator
state
and
system
state
Ø Further,
possibili7es
of
leveraging
these
synergies
for
more
than
adap7ve
automa7on
soon
became
evident,
as
they
provided
a
means
by
which
to
augment
cogni7on
and
thus
extend
the
human
poten7al

27. Past
Approaches
to
Human
Augmenta9on:
Augmented
Cogni9on
Ø Augmented
cogni7on
has
provided
many
of
the
theories,
principles,
and
prac7ces
needed
to
realize
proac7ve
systems
Ø Augmented
cogni7on
R&D
has
primarily
focused
on
mission
cri7cal
systems
that
put
immense
pressure
on
human
cogni7on
§
§
Such
context
have
common
ground
with
Emergency
Dispatching
domain
Thus,
lessons-­‐learned
via
Augmented
Cogni7on
R&D
can
inform
the
ASTUTE
Project
ü ASTUTE’s
proac7ve
systems
can
benefit
from
AugCog
lessons-­‐learned

28. Past
Approaches
to
Human
Augmenta9on:
Augmented
Cogni9on
–
The
Objec9ve
Problem:
The
one-­‐two
punch
of
informa7on
overload
and
mul7tasking
• Increased
volume
of
informa7on
available
in
command
centers,
while
staffing
levels
remain
constant
-­‐
must
to
try
to
do
more,
and
do
it
faster
Augmented
Cogni9on
Objec9ve:
Develop
technologies
capable
of
extending,
by
an
order
of
magnitude
or
more,
the
informa7on
management
capacity
of
individuals
working
with
21st
century
compu7ng
technologies.

29. Past
Approaches
to
Human
Augmenta9on:
Augmented
Cogni9on
–
The
Challenge
Ø Augmented
cogni7on
presents
a
pro-­‐ac7ve
paradigm
through
which
to
achieve
human
performance
gains
in
mission
cri7cal
context
such
as
Emergency
Dispatching
domain
– Challenge
is
to
real-­‐7me
sensor,
measure
and
diagnose
the
collec7ve
human
state
(cogni7ve,
physical,
affec7ve)
and
then
use
theory-­‐based
algorithms
to
proac7vely
adapt
to
and
augment
innate
human
abili7es
– Making
the
unobservable
–
observable

30. Augmented
Cogni0on:
Some
History

31. When
we
started
the
Augmented
Cogni9on
Program:
We
asked
-­‐
do
we
have
it
wrong?
Where
has
it
been
established
that
such
WIMPs
(windows,
icons,
menus,
pointers)
are
the
ul-mate
HSI
design?
What
is
appealing
about
the
WIMP
and
internet
browser
as
the
interface
for
the
human
to
the
computer?
Are
we
falling
into
a
least
common
denominator
trap?
Can’t
we
do
beYer?
ADM
(ret.)
Lee
Kollmorgen

32. All these designs rely on WIMP interfaces –
Windows, Icons, Menus, Pointing Devices

33. Is
it
9me
for
new
HSI
paradigms?
While
current
HSI
paradigms
have
empowered
computer
users
of
varying
ability,
they
alone
cannot
handle
many
of
the
challenges
of
today’s
opera9onal
environments:
• Mul9-­‐tasking
• Mul9ple
informa9on
streams
• Varying
contexts
We
need
systems
that
ac9vely
integrate
the
human
and
provide
proac9ve
support
based
on
individual
capabili9es
and
limita9ons:
• Alleviate
cogni9ve
bo^lenecks
• Support
situa9on
awareness
• Enhance
decision
making
• Account
for
individual
differences

34. Evolution of HSI Paradigms
Command-­‐Line
• Pure
text
interface
• Individual
or
batched
commands
• Predetermined
sequence
à
User
feeds
system
GUI
AugCog
• Graphical
interface
• System
awareness
• Metaphorical
representa7ons
• Sta7c
implementa7on
• Object-­‐oriented
• “Direct
manipula7on”
• Reac7ve
(event-­‐based)
• To
user
input
(behavioral)
• To
system
event
• Standardized
reac7ons
• Baby-­‐steps
towards
dynamic
interac7on
• Personaliza7on
• Smart
menus
à
System
acts
on
user
behavior
• User
state
• User
behavior
• System
state
• Task
context
• Proac7ve
-­‐
adapts
to
changes
in:
• User
performance
• Task-­‐context
requirements
• Provides:
• Dynamic
and
adaptable
representa7ons
• Individualized
response
• Act
on
operator
intent
à
System
proac7vely
adapts
to
user,
task,
and
context

35. Augmented Cognition
Ø A
new
HSI
paradigm
§ Intui7ve
coupling
between
human
and
machine
§ Providing
the
right
informa7on
-­‐
at
precisely
the
right
7me
-­‐
in
the
right
format
to
amplify
human
capabili7es
Ø An
augmented
cogni7on
system
has
three
main
components:
§
§
§
Sensors:
Neurophysiological,
physiological,
and
behavioral
sensors
Measures:
Cogni7ve,
physical,
and
affec7ve
user
state
measures
Adapta7on
Strategies:
Proac7ve
techniques
to
alleviate
situa7ons
of
overload,
inaeen7on,
stress...
and
improve
human
performance

36. AugCog Sensors & Measures
Source:
hep://www.spawar.navy.mil/s7/publica7ons/pubs/tr/tr1940vicond.pdf

37. AugCog Sensors & Measures
Source:
hep://www.spawar.navy.mil/s7/publica7ons/pubs/tr/tr1940vicond.pdf

38. AugCog Sensors & Measures
ü ASTUTE’s
proac7ve
systems
could
design
future
sensors
and
measures
focused
on
increased
SA
and
enhance
decision
making
Source:
hep://www.spawar.navy.mil/s7/publica7ons/pubs/tr/tr1940vicond.pdf
ü ASTUTE’s
proac7ve
systems
could
learn
from
AugCog
sensors,
measures,
and
classifica7on
methods

39. AugCog Adaptive Strategies

40. AugCog Adaptive Strategies
Adapt presentation of information
Modality augmentation
(redundancy, switching)
Transposition
Add or change mode of information
presentation
Change information type from
verbal to spatial or vice versa
Cueing
Augment display to capture
attention of user
Decluttering
Reduce amount/complexity of
information displayed
Context-sensitive help
Provide information specific to
system state at time help is needed

41. AugCog Adaptive Strategies
Adapt scheduling of information
Pacing
Sequencing
Hold low priority information until
current high priority tasks
completed
Simultaneous events converted into
sequential form
Decompose tasks into smaller
portions and re-arrange subtasks

42. AugCog Adaptive Strategies
Adapt system autonomy
Delegate
Mixed Initiative
Transfer tasks to fully-automated
system
Provide operator with most
appropriate level of control for
situation; both operator and system
can adjust system autonomy
ü ASTUTE’s
proac7ve
systems
could
use
AugCog
Adap7ve
Strategies

43. Potential Future Adaptive Strategies
Innova9ve
Adap9ve
Techniques
New
Adapta9on
Objec9ves
-­‐
today:
Cogni7ve
boelenecks
-­‐
today:
Reduce
distrac7ons
-­‐
tomorrow:
Increase
SA,
reduce
confusion…
-­‐
tomorrow:
Reduce
inaeen7veness,
fear…
-­‐
Use
ambient
environment
-­‐
Music
tempo/genre
change
-­‐
Empathe7c
proac7ve
system
-­‐
Use
predic7ve
cogni7ve,
physical,
and
affec7ve
state
ü ASTUTE’s
proac7ve
systems
could
design
future
adap7ve
strategies
focused
on
enhanced
SA
and
DM
Individual
Cogni9ve
Profiles
-­‐
Aiding
novice
dispatchers
-­‐
Suppor7ng
seasoned
dispatchers

45.
Results
of
Augmented
Cogni9on
Program
Summarized
in
AugCog
Prac99oner's
Guide
ü ASTUTE’s
proac7ve
systems
can
benefit
from
AugCog
lessons-­‐learned

46.
Where
we
were
at
end
of
Augmented
Cogni9on
Program:
Need
for
broadening
scope
of
human
state
assessment

47. Next Steps:
Develop
plethora
of
sensors
&
measures
that
drive
adap9ve
systems
Sensors
to
Measure
Cogni7ve
State
Workload,
Uncertainty,
Confusion,
etc.
Adap7ve
Smart
Glasses
Adap7ve
Smart
Phone
Sensors
to
Measure
Physical
State
Sensors
to
Measure
Affec7ve
State
Body
Temp.,
O2
Level,
Physical
Fa7gue
Adap7ve
Smart
Home
Adap7ve
Smart
Tablets
Anxiety,
Fear,
Stress,
Confidence
Adap7ve
Smart
Cars

48. Extensions
aber
Augmented
Cogni9on
Where
DI
has
taken
AugCog
R&D

49. Appling
SIMI
to
Image
Analysis

50. SIMI
for
Image
Analysis
DI
used
EEG
and
eye-­‐tracking
to
develop
real-­‐7me
neurophysiological
indicators
of
‘interest’
during
image
analysis
Sensor:
Eye
tracking
Measure:
Ø Capture
parameters
to
assist
in
determining
‘interest’
(e.g.,
fixa7on
dura7on,
pupilometry)
Sensor:
Electroencephalography
(EEG)
Measures:
Ø Previously
used
to
indicate
individual’s
workload,
arousal,
aeen7on,
drowsiness,
percep7on
of
events
Ø Image
level
analysis:
iden7fy
images
that
contain
one
or
more
points
of
interest
Ø FLERPs:
Fixa7on
level
analysis:
iden7fy
‘interest’
at
specific
fixa7on
points
within
an
image
ü ASTUTE’s
proac7ve
systems
should
consider
use
of
FLERP’s

51. SIMI
for
Image
Analysis
Depic7on
of
all
fixa7on
points
on
the
image
drawn
on
an
Area
Of
Interest
(AOI)
layer

52. SIMI
for
Image
Analysis
• Fixa7on
dura7on
(ms)
in
addi7on
to
the
loca7on
Fixa7ons
greater
than
700ms
• Define
Hits,
Misses,
Correct
Rejec7ons,
False
Alarms
Behavior-­‐based
classifica7ons

53. Appling
SIMI
to
Instrument
Flight
Panel
Training

54. Sensors:
Eye
Tracking,
EEG,
&
Performance
Measures:
Visual
Aeen7on
Alloca7on,
Controls
Engaged,
Errors,
Cogni7ve
Measures
Diagnosis:
Skill
Deficiencies
–
Not
looking
at
Al7meter;
Workload
High
ERROR
Altitude out of range
HIGH WORKLOAD

55. Performance
Feedback

56. Look
at
Error
Details!
ü ASTUTE’s
proac7ve
systems
should
consider
linking
errors
to
physiological
data
for
more
in
depth
diagnos7cs

57. Error
Detail
Screen
Cont.

58. Error
Detail
Screen
Cont.

59. Error
Detail
Screen

60. Error
Detail
Screen

61. Diagnosis:
Looking
at
Al7meter;
Skill
Aeained…
Now:
Arousal
Issue
–
Time
to
move
on
to
next
training
objec7ve
ERROR DETECTED!
Altitude out of range
LOW WORKLOAD/
LOW AROUSAL

62. Appling
SIMI
to
Baggage
Screening

63. SIMI
for
Baggage
Screening:
ScreenADAPT
Individualized
training
for
visual
search:
– Sensors:
U7lizes
eye
and
EEG-­‐based
sensor
technology
– Measures:
Real-­‐7me
cogni7ve
state
and
performance
evalua7on
– Diagnosis:
Non-­‐op7mal
cogni7ve
state,
exper7se
level,
and
deficiencies/inefficiencies
in
screening
performance
– Adapts
in
real-­‐7me
to
op7mize
training:
– Tailored
feedback
and
training
» Exposure
training
» Discrimina7on
training
– Image
generator:
» Allows
instructor
upload
of
new
threats,
distractors,
bags
» Produces
endless
combina7ons
of
image
components
to
avoid
image
repe77on
Provides
individualized
training
for
visual
search
skills

64. Sensors
&
Measures
Performance:
Error
Classifica7on
– Hit,
Miss,
Correct
Rejec7on,
False
Alarm
– Recogni7on
error
-­‐
looked
at
threat,
didn’t
flag
threat
– Scanning
error
-­‐
didn’t
even
look
at
threat
Cogni7ve
State
via
EEG
– Readiness
to
Learn
• Workload
and
drowsiness
Eye
Tracking
– Gaze
paeerns
– Recogni7on
Error
ü ASTUTE’s
proac7ve
systems
should
consider
use
of
recogni7on
and
scanning
error
diagnos7cs

65. Two
Adap9ve
Training
Techniques
Exposure
training
used
to
strengthen
object
detec7on
ability
when
trend
of
False
Alarms
is
iden7fied
– Includes
both
immediate
and
delayed
feedback
to
support
prac7ce
and
training
Discrimina7on
training
used
to
strengthen
object
recogni7on
when
trend
of
Misses
is
iden7fied
ü ASTUTE’s
proac7ve
systems
should
consider
use
of
adap7ve
feedback
based
on
performance
trends

66. SIMI
Applied
to
Emergency
Dispatch
Which
sensors,
measures,
diagnoses,
and
adap9ve
strategies?

67. SensorIT
–
MeasureIT
-­‐
DiagnoseIT
then
Proac9vely
ADAPT!
•
ASTUTE’s
objec7ves
are
to
increase
situa7onal
awareness
and
improve
decision
making
• Situa7onal
Awareness:
•
•
•
•
•
Decision
Making:
•
•
•
•
How
are
you
sensing
SA?
How
are
you
measuring
SA?
How
are
you
diagnosing
SA?
How
are
you
adap7ng
to
SA?
How
are
you
sensing
decision
making?
How
are
you
measuring
decision
making?
How
are
you
diagnosing
decision
making?
How
are
you
adap7ng
to
decision
making?
proac7ve
ü ASTUTE’s
systems
should
consider
how
to
sensor,
measure,
diagnose,
and
adapt
to
SA
level
and
DM
performance

68. SIMI
Applied
to
Emergency
Dispatch

69. SIMI
Applied
to
Emergency
Dispatch
Gamma
waves
reveal
high
engagement
Beta
waves
reveal
over-­‐
arousal
EEG
Captures
Data
Link
to
Context
–
Find
Decision
Error
–
over
arousal
and
high
engagement
were
due
to
CONFUSION
Add
clarifying
info
to
address
decision
error

70. Augmented
Cogni9on:
Remaining
Hard
Problems
Ø Sensors
§
Real-­‐7me,
noninvasive,
highly
sensi7ve
and
reliable
sensors
to
gather
neuro/physiological
data
rela7ng
to
human
cogni7ve,
physical,
and
affec7ve
state
Ø Building
generic
human
state
classifiers
§
§
Determining
which
technology
yields
the
best
results
(e.g.
AI
vs
neural
networks
vs
machine
learning)
Using
the
specific
and
extrapola7ng
to
the
'generic’
Ø Measures
–
measures
–
measures
§
Developing
valid,
reliable
measures
of
a
plethora
of
cogni7ve,
physical,
and
affec7ve
state

71. Augmented
Cogni9on:
Remaining
Hard
Problems
Ø Designing
seamless
adapta7on
techniques
Ø When
to
adapt:
Ø
Ø
Ø
Valid
and
reliable
classifiers
used
to
gauge
when
to
adapt
Ø Classifiers
available
today
are
isolated
single
measures
of
1
state
(e.g.,
arousal,
workload)
-­‐
live
tutors
take
in
the
user
experience
as
a
whole
–
classifiers
need
to
be
mul7dimensional
Threshold
that
triggers
when
to
adapt
Ø Range
of
performance
within
which
to
adapt
at
vs.
above/
below
a
given
threshold
Ø ‘Generalized
theory’
[if
there
is
one]
that
can
drive
adapta7on
triggers
-­‐
want
to
avoid
the
yo-­‐yo
effect
of
'mi7ga7on
on',
'mi7ga7on
off'
paeern
Level
of
granularity
where
adapta7on
takes
place
-­‐
target
'paeerns
of
error'
or
individual
instances
of
error?

72. Augmented
Cogni9on:
Remaining
Hard
Problems
Ø Designing
seamless
adapta7on
techniques
Ø How
to
adapt:
Ø
Ø
Ø
How
to
transi7on
between
mi7ga7ons
-­‐
when
one
supersedes
another
and/or
mul7ple
mi7ga7ons
may
be
used
-­‐
how
to
effec7vely
insert
mi7ga7on
without
distrac7ng
the
user
Validated
mi7ga7on
strategies
proven
to
improve
opera7ons
or
training
for
a
given
diagnosis
(e.g.,
is
mi7ga7on
different
if
error
was
found
due
to
frustra7on
vs.
boredom?)
Determining
how
to
leverage
different
interfaces
to
achieve
the
same
results
(text
versus
speech
versus
earcons
versus
hap7c
language)
ü ASTUTE
could
tackle
many
of
these
challenges
as
they
develop
their
proac7ve
system
concept

73. The
Future
of
HSI
An
R&D
agenda
to
direct
the
HSI
field
through
2050

74. Where
to
from
here?
Augmented
Cogni7on
–
field
started
~2000
– Paradigm
shi`
from
‘dumbing
down
interac7ons’
via
WIMP
interfaces
to
dissolving
the
user
interface
through
direct
brain-­‐computer
interfaces
– Brought
about
interdisciplinary
teams
focused
on
monitoring
and
mi7ga7ng
human
processing
limita7ons
within
opera7onal
environments
– Focused
on
revolu7onizing
human-­‐system
integra7on
a`er
decades
of
being
“locked”
in
the
WIMP
paradigm
Aber
a
decade
of
AugCog,
it’s
9me
to
ask
again…
Where
should
HSI
go
from
here?

75. First:
Iden9fied
HSI
“Enablers”
Other
HSI
Enablers
Beyond
Human
Augmenta9on

76. Enhancing
Human
Performance
HSI
Enabler:
Augmenta9on
Augment
Eliminate
Increase
dynamic
range
and
number
of
couplings
through
next
genera7on
neuroadap7ve
systems
that
achieve
synergis7c
coopera7on
among
human
physical,
cogni7ve,
and
affec7ve
states
ü ASTUTE’s
proac7ve
systems
are
incorpora7ng
human
augmenta7on

77. Enhancing
Human
Performance
HSI
Enabler:
Big
Data
Iden7fy
how
best
to
leverage
big
data
to
simplify
user
decision
making
without
overwhelming
a
user’s
analy7cal
capabili7es
Big
Data
Simplify
Augment
Eliminate
ü Is
ASTUTE
considering
use
of
big
data?

78. Enhancing
Human
Performance
HSI
Enabler:
Autonomy
Iden7fy
how
best
to
design
autonomy
into
a
mission
such
that
performance
is
op7mized
and
unintended
opera7onal
consequences
are
avoided
Combine
Autonomy
Big
Data
Simplify
Augment
Eliminate
ü Is
ASTUTE
considering
use
of
autonomy?

79. Enhancing
Human
Performance
HSI
Enabler:
Transgenics
Gene7cally
alter
the
human
via
gene
therapy,
gene7c
breeding,
and
gene7c
engineering
Resequence
Transgenics
Combine
Autonomy
Simplify
Big
Data
Eliminate
Augment
ü Is
ASTUTE
considering
use
of
transgenics
(e.g.,
gene7c
engineering
to
enhance
heat
acclima7on)?

80. Second:
Iden9fied
HSI
“Eras”
What’s
beyond
the
digital
era?

81. Enhancing
Human
Performance
HSI
Eras
Ø To
map
out
an
HSI
R&D
agenda
through
2050,
the
current
and
future
eras
that
human-­‐systems
interac7on
will
traverse
through
were
considered
81

82. Enhancing
Human
Performance
HSI
Eras
Ø Individualism
Op7miza7on:
– Brain
era
seeks
to
achieve
“super-­‐intelligence”
– Physical-­‐feat
era
seeks
to
achieve
“super-­‐humans”
through
symbio7c
coupling
of
human
and
machines
to
overcome
universal
human
limita7ons
Ø Collec7vism
Op7miza7on:
– Human
quantum-­‐entanglement
era,
which
will
support
human-­‐human
communica7on,
where
crea7on
of
human
“superorganisms”
is
the
end
goal
– Ecological
quantum-­‐entanglement
era,
which
will
support
synergy
between
humans
and
their
environment,
where
“super-­‐symbiosis”
is
the
end
goal

83. Enhancing
Human
Performance
HSI
Eras:
Individualism
Op9miza9on
–
Brain
Era
Seeks
to
achieve
super-­‐intelligence
– Ar7ficial
intelligence
becomes
an
exocortex
to
eliminate
need
for
brainpower
– Big
data
algorithms
transform
data
into
intelligence
– Extend
innate
intelligence
with
cogni7ve
prostheses
– Resequence
our
brains
to
fundamentally
improve
intellectual
capacity
ü ASTUTE
is
considering
use
of
tablet-­‐based
and
PDA
cogni7ve
prostheses

84. Enhancing
Human
Performance
HSI
Eras:
Individualism
Op9miza9on
–
Physical
Feat
Seeks
to
achieve
super-­‐humans
– Exoskeletons
and
psychos7mulants
used
to
enhance
human
physical
ability
– Cloud
used
to
monitor
human
ac7vi7es
to
fuel
big
data
algorithms
that
can
realize
vast
expansion
of
human
physical
poten7al
and
op7mize
health
– Cyborgs
become
reality
–
implants,
prosthe7cs,
psycho-­‐pharmacological
agents
– Gene7c
muta7ons
op7mize
human
physical
capacity
to
protect
against
disease
ü Is
ASTUTE
considering
means
to
enhance
physical
ability
and
resilience?

85. Enhancing
Human
Performance
HSI
Eras:
Collec9vism
Op9miza9on
–
Human
Quantum-­‐Entanglement
Era
Seeks
to
achieve
superorganisms
– Robots
synergis7cally
augment
human
collec7ve
– Big
data
algorithms
combine
carbon
and
silicon-­‐
based
intelligence
into
a
single
collec7ve
consciousness
– Human
to
human
quantum
entanglement
for
collec7ve
intelligence
– Gene7cally
alter
humans
to
allow
for
chemical
communica7on,
telepathy,
and
other
means
to
support
communica7on
ü Is
ASTUTE
considering
means
of
achieving
collec7ve
intelligence?

86. Enhancing
Human
Performance
HSI
Eras:
Collec9vism
Op9miza9on
–
Ecological
Quantum-­‐Entanglement
Era
Seeks
to
achieve
super-­‐symbiosis
– Biosphere
morphs
to
op7mize
human
environment
collec7ve
– Directly
sense,
measure,
and
understand
molecular
processes
in
collec7ve
environment
– Sensor-­‐enabled
ecological
scavengers
to
predict
and
adapt
the
environment
to
op7mize
symbiosis
– Gene7cally
alter
human
such
that
they
are
beeer
suited
for
their
environment,
thereby
elimina7ng
heat
sensi7vity
and
other
maladapta7ons
ü ASTUTE
is
collec7ng
and
leveraging
environmental
data;
are
they
considering
use
of
adaptable
biospheres?

87. Emerging
HSI
Eras:
Big
Data
HSI
Enabler
Extends
Info
Highway
Brain
Era
populates
the
Neurosphere
Physical
Feat
Era
populates
the
Physiosphere
Human
Quantum-­‐Entanglement
Era
populates
the
Noosphere
Ecological
Quantum-­‐Entanglement
Era
populates
the
Biosphere
ü Is
ASTUTE
collec7ng
and
coordina7ng
data
from
neurosphere,
physiosphere,
noosphere,
and
biosphere?

88. HSI
Emerging
Concepts:
HSI
Eras
&
Enablers

89. HSI
R&D
Agenda
through
2050
ü ASTUTE
can
enhance
human
performance
by
considering
4
HSI
Eras
and
4
HSI
Enablers

90.
The
future
of
HSI…

91. Conclusions
Ø ASTUTE
aims
to
implement
proac7ve
systems
that
increase
SA
and
enhance
decision
making
Ø Lessons-­‐learned
from
adap7ve
automa7on
and
augmented
cogni7on
R&D
can
inform
the
design
of
ASTUTE’s
proac7ve
systems
Ø ASTUTE
should
also
look
to
future
HSI
emerging
concepts:
Ø Consider
HSI
enablers
beyond
human
augmenta7on,
to
include
autonomy,
big
data,
and
transgenics
Ø Consider
advances
in
other
HSI
eras
beyond
the
brain
era,
to
include
physical-­‐feat,
human
quantum-­‐entanglement,
and
ecological
quantum-­‐
entanglement
eras

92. Acknowledgments
Special
thanks
goes
to
Innoviris
for
their
support
of
this
presenta-on.
Any
opinions,
findings
and
conclusions
or
recommenda-ons
expressed
in
this
material
are
those
of
the
author
and
do
not
necessarily
reflect
the
views
or
the
endorsement
of
Innoviris.

93. Acknowledgments
This
material
is
based
upon
work
supported
in
part
by
the
Office
of
Naval
Research
(ONR)
under
contracts
N0001413M0047
and
N00014-­‐09-­‐M-­‐0385,
Department
of
Homeland
Security
(DHS)
under
contracts
N10POC20028
and
D11PC20053,
Defense
Advanced
Research
Projects
Agency
(DARPA)
under
contracts
W31P4Q-­‐06-­‐C-­‐0041
and
W31P4Q-­‐07-­‐C-­‐0214,
and
the
Air
Force
Research
Laboratory
(AFRL)
under
contracts
FA8550-­‐06-­‐C-­‐0151
and
FA8550-­‐06-­‐C-­‐0151.
Any
opinions,
findings
and
conclusions
or
recommenda-ons
expressed
in
this
material
are
those
of
the
author
and
do
not
necessarily
reflect
the
views
or
the
endorsement
of
ONR,
DHS,
DARPA,
and
AFRL.