2014 Karles Invitational Conference

The difference between “Winning the War”
and “Winning the Peace.”
Mr. John S. Canning
Combat Systems Engineer, G82
Naval Surface Warfare Center
Dahlgren Division
(540)653-5275
John.S.Canning@navy.mil
14 January 2014
2014 Karles Invitational Conference at NRL
Washington, D.C.
Distribution Statement A, Approved for Public Release, Distribution is Unlimited
Autonomy: Legal, Ethical, & Policy Issues
Drive a New Paradigm in Warfare

Distribution Statement A
Brief Structure
• Background
• Legal issues
• Ethical issues
• Policy issues
• The New
Paradigm

Background

Prelude
• The Dahlgren Division of the Naval Surface Warfare
Center (NSWC DD) is the Navy’s premier laboratory for
the development of weapon systems and shipboard combat
systems.
• In late 2002, NSWC DD began to look at the use of
weapons by unmanned systems, specifically, at the
autonomous use of weapons.
• We wanted to determine how these machines could be
allowed to figure-out for themselves when to pull the
trigger.
With apologies to BASF:
We don’t make your robot.
We make your robot better by hanging weapons on it!

Meet the Lawyers
• Meeting at Dahlgren with both the Navy JAG, and
the OSD Office of General Counsel.
– 23 September 2003
• Discussed the arming of robotic systems.
• Working relationship with Navy JAG has been
maintained.
• Key insight:
“The ultimate goal in warfare is not to kill the enemy,
but to bring hostilities to a complete and lasting close
as quickly, and as humanely, as possible.”
Hayes Parks – OSD Office of General Counsel (Ret)

Meet the Philosopher
• Dr. Rob Sparrow, Monash University
(Australia) School of Philosophical,
Historical and International Studies
– Dahlgren visit, February 2008
• Sparrow’s robotic interests:
– Building safe systems
– Designing for the Law of Armed Conflict
– The machine and the mission
– Not making ourselves redundant
• Widely-published in this area.

Meet the Computer Scientists
• Dr. Ron Arkin
– Regents' Professor, Associate Dean for Research and Space Planning
School of Interactive Computing, College of Computing, Georgia Tech
• Dahlgren visit, September2007
• Key insight:
– Develop an “ethics module” & autonomouslytarget people
• Prof Noel Sharkey
– University of Sheffield (UK) Professor of Artificial Intelligence and
Robotics
– Judge from the TV program “Robot Wars”
• AUVSI North American Denver event, August 2010
• Key insight:
• Shouldnot be weaponized at all
Note that they are on opposite ends of the spectrum.

Meet the Ethicist
• Dr. George Lucas
– Professor of Ethics & Public Policy (NPS)
– Distinguished Chair in Ethics, VADM James StockdaleCenter for
Ethics, U.S. Naval Academy.
• September 2009 meeting at the Naval Academy
– Asked Mr. Canning to come brief his then-current crop
of Stockdale Fellows on our approach to arming robots.
• Key insight:
– SupportsDahlgren’s approach to weaponizing robots

Meet the “Policy Wonk”
• Paul Scharre
– OSD (Policy)
• April 2010, OSD Working Group
• Spearheaded development of DoD Directive
3000.09, “Autonomy in Weapon Systems.”
• Key insight:
– “No matter how autonomous, people are
ultimately in-charge.”

International Committee for Robot Arms
Control (ICRAC)
• Stood-up December 2011
• Desires ban on armed robots
– Both Sparrow and Sharkey are ICRAC founding
members (Sharkey is Chair)
• Closely coupled with Christof Heyns, UN Special
Rapporteur on extrajudicial, summary or arbitrary
executions (Human Rights Council)
• Key insight:
– Stop “Killer Robots”
Note that their focus is on “Killer Robots.” They don’t differentiate
between semi-autonomous, and autonomous, systems.

Why Do We Mention These People and
Organizations?
• Each brings their own perspective to the autonomy issue
for weapons use.
• An effective design for an autonomous system must
consider ALL of them.
– We may choose to disagree with, or ignore, some of them, but we
betterhave a good reason when we do.
• Without doing this, we would be wasting our time,
designing something that would likely never get used.
Arming robots is not just a narrow
engineering problem.

Legal Issues

Hey, they’re lighting their arrows…can they do that?
This is all about what is, and isn’t, allowed under
the Law of Armed Conflict (LOAC).

From the beginning of human history, man has been
targeting his enemies with his weapons
How many millions have
died, or been injured?
Civil War dead
WWII Battle of the Bulge
Remembering the dead from Iraqi Freedom

Under the Napoleonic Theory of War (everything is fair
game), we have opted for the “bigger bang,” causing
potential for incidental injury to civilians and collateral
damage to civilian property to increase.
An atomic blast
The atomic dome in Hiroshima,
located directly under Ground Zero.
Safety of innocent civilians
wasn’t the greatest concern.

Lessons from WWII: Destruction beyond that necessary
to accomplish the military objective can prolong the war,
and can make securing a lasting peace more difficult.
WWII bomb damage in the
German city of Dresden
German civilians in Halberstad
following 8 APR 1945 bombing

TV brought the Vietnam war to the nation’s living rooms,
put a human “face” on the war and contributed to civil and
political unrest at home
Vietnam War protest in Washington, D.C.
Siege at Khe Sanh – 500lb
bombs falling on NVA trenches
“The Wall”

Despite man’s history of violence, there have long been
restrictions on the use of force during war. Today, treaties
as well as the Law of Armed Conflict or LOAC regulate the
use of force during armed conflict.
• Now, all weapons and weapon systems, from small arms and
ammunition to cruise missiles are subjected to a legal review to ensure
compliancewith the Law of Armed Conflict (LOAC) and applicable
treaties.
• Additionally, once declared legal, the employment of these weapons
may be further controlledby Rules of Engagementand the
DiscriminateUse of Force

Legal Review of Weapons
• DoD policy* requires that a legal review be conductedof all weapons and
weapon systems acquired to meet a military requirement of the US.
• Primarily this review requires an analysis of three factors:
(1) whether the weapon causes suffering that is needless, superfluous, or
disproportionate to the military advantage reasonably expected from the
use of the weapon. It cannot be declared unlawful merely because it may
cause severe suffering or injury;
(2) whether the weapon is capable of being controlled so as to be directed
against a lawful target, (i.e., it can discriminate between lawful and unlawful
targets);
(3) whether there is a specific treaty provision or domestic law prohibiting the
weapon’s acquisition or use.
• These three factors are analyzed in relation to the weapon’s intended
method of employment, not in relation to any possible use, as any lawful
weapon can be used illegally.
*See DODD 5000.1, subparagraph E1.15, and SECNAVINST 5000.2C, paragraph 2.6.
With regard to Armed AutonomousSystems, the critical issue
is the ability for the weapon to discriminate a legal target

Rules of Engagement Defined
• Directives issued by competent
authoritywhich delineatethe
circumstances and limitations under
which U.S. forces will initiate and/or
continuecombat engagement with
otherforces encountered.
JointPub 1-02
• ROE are based on the LOAC as well as
political and military factors and can be
utilized to guide the military use of
force during a particularoperation.
ROE can restrict the employment
of certain weapons dependingon
the tactical, strategic or political
situation.

Discriminate Use of Force (DUF)
• “Our concept of DUF strongly aligns with much of the current thinking about
effects-based operations (EBO). The coming of age of these concepts is
influenced both by opportunity and need.
• DUF brings new concepts for collaboration and massing of effects, which are
joint in character and integrated among joint force echelons and components.
It is enabled by new weapons; improved intelligence, surveillance, and
reconnaissance; shared situation understanding; improved individual and
collaborative training; greater agility; smaller footprints; and other emerging
capabilities of the U.S. military that allow more timely and precise use of force
than heretofore possible.
• The need is driven by the nature of current military campaigns. A striking
feature of these campaigns is the tension among multiple strategic and
operational objectives: cause regime change, destroy a terrorist organization,
decapitate leadership, but preserve infrastructure, don’t wage war on a people,
do hold an international coalition together, etc.”
“Reportof the Defense Science Board Task Forceon Discriminate Use of Force,”JUL 2003
Driven by new technology yielding better discrimination,
which leads to demand for even better technology

The Issue
• Using today’s paradigm of warfare, there is a requirement to maintain
an operatorin the “weaponsrelease”-loopto avoid the possibility of
accidentallykilling someone.
• An operatoris effectively “welded” to each armed unmanned system
for this purpose.
• This is a “performance- and cost-killer”when considering the
employment of large numbers of armed unmanned systems
How can we effectively employ autonomous armed
unmanned systems, while avoiding this problem?

Ethical Issues

Asimov’s Three Laws of Robotics:
1. A robot may not injure a human being or, through
inaction, allow a human being to come to harm.
2. A robot must obey orders given it by human beings
except where such orders would conflict with the First
Law.
3. A robot must protect its own existence as long as such
protection does not conflict with the First or Second Law.
Originally from Asimov’s 1942 short story “Runaround”
While labeled as “laws,” the real thrust behind
these are the ethics of protecting human life

“The Terminator”
Hollywood went to the far ethical extreme from Asimov with these films

Where’s Reality?
• For a long time, Asimov defined everyone’s
“reality” when it came to robo-ethics.
• “The Terminator” movie series took
everyone in a completely different ethical
direction.
• The issue is that the ethics for both of them
are rooted in fiction.
Neither represents reality

Policy Issues

New DoD Directive
• DoD has recently released a new Directive, 3000.09, “Autonomy in Weapon
Systems”
– Directive was the result of an 18-month effort that included the Office of the
Secretary of Defense (OSD), the Joint Staff, and the Military Services
– Mr. Canning was the sole representative on the Working Group from any of the
Service Laboratories (to include Air Force and Army).
• This Directive establishes policy for the development and use of autonomy in
weapon systems, including:
– Autonomous and semi-autonomous functions
– Manned and unmanned platforms
• The purpose is to establish guidelines to minimize the probability and
consequences of failures that could lead to unintended engagements without
unduly inhibiting progress toward the development of new capabilities.
This new DoD Directive closely mirrors Dahlgren’s approach.

The New Paradigm

Battlefield Economics
Today’ Soldier Today’s Robots
Tomorrow’s Desired Robot Force
Too expensive to keep complete “man-in-the-loop” fire control for very large numbers of robots. We must
move to autonomous robots that can decide for themselves when to pull the trigger.
There is a known cost associated with putting
the man on the battlefield with his weapon.
This is “man-in-the-loop” fire control.
We are placing an expensive machine between
the man and his weapon, raising cost..

Maybe we can!
This is your worst nightmare!
It is a safety issue concerning the innocents of war.

Target Discrimination:
How do you tell the difference?
Between a cruise ship…
…and a war ship?
Between peoplewho are just mad at you…
…and a determined enemy?

Today’s Valid Targets from a Legal Standpoint
Not a Military
Objective
Valid Military
Objective
Not a Military
Objective
Can’t Target Target People
Valid Military
Objective
Target Things Target All
People
Things
“We can target objects when they are military objectives and we can target people when they are military
objectives. If people or property isn't a military objective, we don't target it. It might be destroyed as collateral
damage, but we don't target it. Thus in many situations, we could target the individual holding the gun and/or the
gun and legally there's no difference.” – MAJ R. Craig Burton, USAF, Judge Advocate General's Legal Center
and School

Tomorrow’s Target Subset for Autonomous Systems
Not a Military
Objective
Valid Military
Objective
Not a Military
Objective
Can’t Target
People
Won’t Target
People
Valid Military
Objective
Target Things Target Things,
but Not People
People
Things
For autonomous systems, we are purposefully
restricting the target set to “things.”

A Proposed Paradigm for Autonomous Use
of Weapons
• Let the machines target other machines
– Specifically, let’s design our armed unmanned systems to automatically
ID, target, and neutralize or destroy the weapons used by our enemies –
not the people using the weapons.
– This gives us the possibility of disarming a threat force without the need
for killing them.
– We can equip our machines with non-lethal technologies for the purpose
of convincing the enemy to abandon their weapons prior to our machines
destroying the weapons, and lethal weapons to kill their weapons.
• Let men target men
– In those instances where we find it necessary to target the human (i.e. to
disable the command structure), the armed unmanned systems can be
remotely controllable by human operators who are “in-the-weapons-
control-loop”
• Providea “Dial-a-Level” of autonomy to switch from one to the
other mode.
This may overcome some of the political objectionsand legal
ramifications of the use ofArmed AutonomousSystems

Legal Precedence Established
• TOMAHAWK Anti-Ship Missile
– Passive Identification/Direction-Finding
Equipment
• CAPTOR Mine
– “Mousetrap that chases the mouse”
• AEGIS Ships
– “Auto-Special”Engagement Mode
• Close-In Weapon System
– Automatic Cruise Missile Defense
• Patriot Missile System
– Automated air defense
Each of these directly targets either the bow, or the arrow,
but not the archer. People may still die, but as a
secondary consequence of going after the weapon of war.

Tomahawk Anti-Ship Missile
1983 - 1995
The missile is launchedin the general direction of the target and at some distance
from the expected target position, it enters a serpentineflight pattern to search for
it using both active radar to lock on a detectedtarget, and passive identification
equipment to scan enemy emissions.
PI/DE Capability
From “The Commander’s Handbook on the Law of
Naval Operations,” NWP 1-14M
9.9 OVER-THE-HORIZON WEAPONS SYSTEMS
Missiles and projectiles with over-the-horizon or beyond-
visual-range capabilities are lawful, provided they are
equipped with sensors, or are employed in conjunction with
external sources of targeting data, that are sufficient to
ensure effective target discrimination.

CAPTOR Mine System
1979- 2000
The mousetrap that chases the mouse
CAPTOR acousticallydetects
submarines while ignoring
surface ships. Upon detection of a
target, the mine launchesan
acoustic homing Torpedo Mk 46
Mod 6.

AEGIS Auto-Special Doctrine
1973-Present
AEGIS Auto-SpecialDoctrine allows “hands-off” engagement of AAW
threatscompletelyfrom initial detection to kill assessment, and the decision
to re-engage, if necessary.

Close-In Weapon System
1980- Present
The MK 15 Phalanx Close-In Weapons System is a fast-reaction, rapid-fire 20-millimeter
gun system that provides US Navy ships with a terminal defense against anti-ship
missiles that have penetrated other fleet defenses. Designed to engage anti-ship cruise
missiles and fixed-wing aircraft at short range, Phalanx automatically engages functions
usually performed by separate, independent systems such as search, detection, threat
evaluation, acquisition, track, firing, target destruction, kill assessment and cease fire.

Patriot Missile System
1984- Present
“An incoming missile could be 50 miles
(80.5 kilometers) away when the Patriot's
radar locks onto it. At that distance, the
incoming missile would not even be visible
to a human being, much less identifiable. It is
even possible for the Patriot missile system
to operate in a completely automatic mode
with no human intervention at all. An
incoming missile flying at Mach 5 is
traveling approximately one mile every
second. There just isn't a lot of time to react
and respond oncethe missile is detected,
making automatic detection and launching an
important feature.”
http://science.howstuffworks.com/patriot-missile.htm

A Relevant Dichotomy
Anti-Tank LandminesAnti-Personnel Landmines
There is a huge international debate over the continuing use of Anti-Personnel Landmines,
with most of the world abandoning their use. The single essential of the problem is the fact
that conventional Anti-Personnel Landmines are designed to persist, remaining lethal for
decades after they are emplaced. This then becomes a long-term issue for civilian populations
living in the areas that were mined. There is not the same level of debate over the use of Anti-
Tank Landmines.
This highlights the issue of targeting the archer, as opposed to his bow, or arrow.

Enabling Technologies
• Sensors
• Artificial Intelligence
• Communications
• Protection
• Stabilized weapons
• Data recording

How Do You Tell a Lethal Machine from
One that Just Wants Your Rifle?
Answer: You can’t!
Your machine has to be able to present a credible lethal threat to the enemy.We
can keep the enemy guessing as to the lethality of the machine he is facing since
he won’t know if a human controller is on the other side of the interface,or just a
computer.(The Turing Test.)

Striking a Balance
Source: IEEE Technology and Society magazine, Spring 2009
Our approach to arming
autonomous unmanned systems
strikes a balance between the
legal, ethical, and policy issues
involved, while also considering
the financial costs of conducting
war.
It presents us with a truly new
paradigm for the conduct of
warfare in which we will be
trying to disarm a foe, as opposed
to killing him, but maintains the
ability to kill him, if necessary.

A Parting Shot

Backups

The Leading Worldwide Ethical & Philosophical
Positions on the Use of Armed Robots
• International ban on all armed unmanned systems. – Noel Sharkey,
University of Sheffield, founding member of the International Committee for Robotic Arms
Control (ICRAC)
• Develop an “ethics” module for armed robots that will allow
them to process the Rules Of Engagement more even-
handedly than human troops, and allow them to target and kill
humans. – Ron Arkin, Georgia Tech
• Design armed robots that target and engage a foe’s
implements of war – not the human enemy – and do this with
“weapons” that may not be traditional. – John Canning, NSWC DD
Which one is the most ethical?

0
20
40
60
80
100
120
Completely ban armed
unmanned systems
Autonomously target the
"bow," or the "arrow" -
not the human "archer"
Autonomously target all
valid military targets
Relative"Value"forEthics&Technology
Primary World-Wide Positions on Targeting of Autonomous, Armed Unmanned Systems
Comparison of Ethics and Technology for the Primary
International Positions on Autonomous, Armed
Unmanned Systems
Relative improvement over today's
battlefield ethics
Relative level of technology
Improvement needed

Why is Policy Important?
• Legal issues define what we must do to meet the requirements of both
domestic law and internationalagreements.
– That leaves a lot of “gray area” open to interpretation.
• Ethical issues define what we should do from a moral standpoint.
– They don’t necessarily state what we must do.
• According to Webster, policy is defined as “…a definite course or
method of action selected from among alternatives and in light of
given conditionsto guide and determine present and future decisions.”
– This tackles the gray areas not covered well by the legal issues, and provides more
firm direction than ethics do.
Policy provides more definitive direction on what
we can and cannot do in our robotic designs.

• This Directive was motivated by a desire to get ahead
of increasing autonomy in both unmanned and manned
systems and establish policy for an emerging
technology
– For a number of years, thought leaders inside and outsideDoD have
raised questionsabout the appropriaterole of autonomy in military
systems, including weapon systems
– Prior to this effort, DoD lacked formal policies on appropriatedegrees of
autonomy and human control in the use of force
• This Directive establishes a flexible and responsible
framework upon which to further refine and adapt
policy as technology continues to evolve
Policy for an Emerging Technology
Our efforts were part of a “forcing function”
that caused OSD to move on this policy.

Policy Established in Directive
• Codifies existing practices with respect to autonomous and
semi-autonomous functions:
– Semi-autonomoussystems using non-lethalor lethal force
• Systems that use autonomous functions to engage specific targets that have
been selected by a human operator
– Defensive, human supervised systems employing lethal force for local
defense of manned vehicles or installations
• For example: Patriot, Aegis, active protection systems
– Fully autonomoussystems using non-lethal,non-kinetic, anti-materiel
force, such as some forms of electronic attack
• Establishes a process (and criteria) for reviewing potential
future autonomous weapon systems
– Provides responsibleoversight for development of autonomous
systems, without tying the hands of developers
– Mandatesappropriatelevel of verification & validation (V&V) and test
& evaluation (T&E) to ensure safe and reliable functioning

Sensors
• “DC to Daylight”
– Broad spectrum coverage
– Detect the presence of weapons
• Radar
– Imaging
– Robust
– Enable target discrimination
• Distributed Imaging Radar Technology (DIRT)
• Optical
– IR
– Low Light Level
– “All-weather” capability
• Other
– ?
• No single “Silver Bullet” sensor
– Likely will need a combination of sensors
Imaging Radar
Night Vision
IR Image

Artificial Intelligence
• Situational Awareness
– Sensor fusion
• Efficient battlefield search for weapons
• ID weapons as weapons
– Automatic Target Recognition
– Share information about new weapons with others
• Communicate to enemy that his weapon is being targeted
– Give him the opportunity to abandon his weapon
• “Dial-a-Level” of autonomy
• Select correct weapon(s) for use
• Target/track enemy weapons
• Engage enemy weapons
• Swarm behavior
– Self-coordinating
A cartoon for AI
Linguistic Geometry

Communications
• Provide Common Relevant Operational Picture
(CROP) input to the Command Structure
• Local coordinating communications among other
unmanned systems
• “Skip echelon” capability
• Secure
– LPI/LPD
– Encryption
• High bandwidth
– HDTV
• Communicate with the enemy
Long-Range Acoustic Device
Navy Combat Information Center

Protection
• Expect to draw fire
– Remember, we will be using COTS gear
– Be prepared for it
• Armor
– Passive (i.e. Kevlar)
– Active (i.e explosive)
• Use redundant & dispersed components
• Active defenses
– Take out the source of incoming fire
• Hostile intent is already established
• Kill the source
– Take out the incoming fire itself
• Electronic Attack Systems
• Counter-RPG systems
– Self-repairing materials
Ye olde armor
Electronic Attack System
TROPHY counter-RPG System

Stabilized Weapons
• Shoot faster and straighter than a human
• Target the enemy’s weapons
• Stay inside the enemy’s OODA loop
• Non-lethals needed to separate human from his
weapons
– Active Denial technology
• Lethals needed to destroy weapons
– Lethal to weapons
– Traditional lethals
• Guns
• Missiles
– Unconventional lethals
• Directed Energy Weapons
Active Denial ACTDShip-mounted stabilized guns
The “weapon of
choice” may not be
a traditional gun or
missile.
It could be a
diamond-tipped
saw!

Data Recording
• What happensif the enemy spoofs our armed unmanned systems, and
causes them to kill when they shouldn’t?
– Political support can disappear virtually instantaneously
• Law enforcement departmentsequip today’s police cruisers with video
cameras and recorders to provide evidence of what happensduring
routinetraffic stops.
• Need to record, and download, sensor data from our unmanned
systems leading up to, and encompassing, engagements so that we
have a record of any attempts at spoofing.
• Suppliesdirect evidence of enemy guilt
From a police video of a traffic stop

2014 Karles Invitational Conference

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (14)

Similaire à 2014 Karles Invitational Conference

Similaire à 2014 Karles Invitational Conference (20)

2014 Karles Invitational Conference