Thrust Group on International Governance of Robots in National Security

CETMONSFebruary 5, 2010

Robotics in the Military:Technology and Applications

Ron Arkin

Robots for the Battlefield• South Korean robot platform is intended to be able to detect and identify

targets in daylight within a 4km radius, or at night using infrared sensors within a range of 2km, providing for either an autonomous lethal or non-lethal response. The system does have an automatic mode in which it is capable of making the decision on its own

• iRobot, the maker of Roomba, is now providing versions of their Packbots capable of tasering enemy combatants; also some versions are equipped with the highly lethal MetalStorm weapon system.

• The SWORDS platform developed by Foster-Miller is already at work in Iraq and Afghanistan and is capable of carrying lethal weaponry (M240 or M249 machine guns, or a Barrett .50 Caliber rifle). New MAARS version under development.

• Israel is considering deploying stationary robotic gun-sensor platforms along its borders with Gaza in automated kill zones, equipped with fifty caliber machine guns and armored folding shields.

• The U.S. Air Force has created their first hunter-killer UAV, named the MQ-9 Reaper, successor to the Predator and widely used in Afghanistan.

• China is developing the “Invisible Sword”, a deep strike armed stealth UAV.

• Many other examples both domestically and internationally.

Current Motivators for Military Robotics

Force Multiplication Reduce # of soldiers needed

Expand the Battlespace Conduct combat over larger areas

Extend the warfighter’s reach Allow individual soldiers to strike further

Reduce Friendly Casualties

The use of robotics for reducing ethical infractions in the military does not yet appear anywhere

Samsung Techwin Korean DMZ Surveillance and Guard Robot

War Robots: Concerns & Risks

Patrick Lin, Cal Poly, San Luis Obispo

Ed Barrett, US Naval Academy

Jason Borenstein, Georgia Tech

Overview

Legal challenges

Just-war challenges

Technical challenges

Robot-human challenges

Societal challengesOther and future challenges

1. Legal Challenges

Unclear responsibility To whom would we assign blame—and punishment—

for improper conduct and unauthorized harms caused by an autonomous robot (whether by error or intentional)?

Designers, robot manufacturer, procurement officer, robot controller/supervisor, field commander, a nation’s president or prime minister...or the robot itself?

Refusing an order If robots have better situational awareness, could they

refuse legitimate orders (e.g., to attack a house in which it detects children)?

1. Legal Challenges (cont’d)

Consent by soldiers to risk In 2007, a semi-autonomous robotic cannon

malfunctioned and killed 9 friendly soldiers and injured 14 other in the South African army

Unclear designation of combatants Legal status of UAV operators in the U.S.: e.g., can

they be attacked on their days off work? Legal status of civilians who work on robotic

systems: e.g., are they combatants on the battlefront?

2. Just-War Challenges

Attack decisions Increasing tempo of warfare may require split-second

decisions that only computing machines can make No “eyes on target” or “human in the loop” poses risk of

wrongful attack

Lower barriers to war Fewer US deaths = lower political cost = more likely to go

to war? But this could be said of any new offensive/defensive

technology Do robots enable us to do morally/legally questionable

things that we otherwise wouldn’t do, e.g., Pakistan strikes?

2. Just-War Challenges (cont’d)

Imprecision of Laws of War & Rules of Engagement Using LOW/ROE in programming is incomplete, e.g.

req’t to minimize civilian casualties doesn’t specify hard numbers

Similar to unintended results in Asimov’s Laws of Robotics?

Halting conflict Given the nature of modern warfare, which

individuals/groups of combatants have the authority to end hostilities?

Will/can combatants surrender to robots (or operators or maintenance crew)? If so, what is the appropriate process for handling the situation?

3. Technical Challenges

Discrimination among targets Too difficult? Requires contextual understanding

Robots gone wild Malfunction, hacking, capture

Unauthorized overrides How to prevent a rogue officer from improperly

taking control of a robot?

4. Human-Robot Challenges

Effect on squad cohesion Unblinking eye may erode “band of brothers”

Self-defense If no such instinct, then very expensive equipment

may be captured or lost

Winning hearts and minds Lasting/true peace may be hindered by using robots

to control population and to fight wars (shows lack of respect?)

5. Societal Challenges

Counter-tactics in asymmetrical warfare More desperate enemies = increased terrorism and

other unconventional tactics?

Proliferation Other nations will eventually have war robots, just as

with other weapons

Space race Militarization of space increases space pollution, etc.

Civil security and privacy Military robots may turn into police/civilian security

robots

6. Other/Future Challenges

The precautionary principle Slowing/halting work in order to address serious

risks seems to make sense …but this is in tension with pressure to use robots in

the military

Co-opting of ethics work by military Can justify work in robotics by saying that ethics is

already being attended to

Robot rights In distant future, if robots have animal- or human-

level intelligence

Current Governance Architecture

George R. Lucas, Jr.Richard M. O’Meara

Conventions in International Law for specific technologies

• The 1999 Hague declaration concerning expanding bullets • Convention on the Prohibition of the development, Production

and stockpiling of Bacteriological (Biological) and Toxin Weapons and on their Destruction (1972)

• Convention on the prohibition of military or any hostile use of environmental modification techniques (1976)

• Resolution on Small-Calbre Weapon Systems (1979)• Protocol on Non-Detectable fragments (Protocol 1) (1980)• Protocol on Prohibitions or Restrictions on the Use of Mines,

Booby-Traps and Other Devices (Protocol 11) (1980)• Protocol on Prohibitions or Restrictions on the Use of

Incendiary Weapons (Protocol 111) (1980)

Conventions in International Law for specific technologies, II

• Convention on the prohibition of the development, production, stockpiling and use of chemical weapons and on their destruction (1993)

• Protocol on Blinding Laser weapons (Protocol 1V to the 1980 Convention (1995)

• Protocols on Prohibitions or Restrictions on the Use of Mines, Booby-Traps and Other Devices as amended on 3 May, 1996 (Protocol 11 to the 1980 Convention as amended on 3 May 1996)

• Convention on the Prohibition of the Use, Stockpiling, Production and Transfer of Anti-Personnel Mines and on their Destruction (1997)

• Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May be Deemed to be Excessively Injurious or to Have Indiscriminate Effects , Amendment article 1, 21 (2001)

• Protocol 1 Additional to the 1949 Geneva Conventions; Convention on Cluster Munitions (2008). ICRC, ‘International Humanitarian Law-treaties and Documents

Five Core Principles:Int’l Humanitarian Law & LOAC

• Weapons prohibitions: suffering or superfluous injury; otherwise:

• Military necessity• Proportionality• Discrimination• Command responsibility

Weapons Prohibitions

• Some weapons are patently inhumane

• Others are design-dependent (effects are reasonably foreseen)

• Thus, ICRC/SIrUS criteria would ban weapons when:

Use of weapon would foreseeably cause:• A specific disease, specific abnormal

physiological state, a specific and permanent disability or specific disfigurement; or

• Field mortality of more than 25% or a hospital mortality of more than 5%; or

• Grade 3 wounds as measure by the Red Cross wound classification scale; or

• Effects for which there is no well-recognized and proven treatment.

Military Necessity

• Promotes speedy end to hostilities• Requires definition of victory• Requires assessment of intent or

capacity of enemy

Proportionality

• Considerable concern to innovator or user of new technologies

• Is foreseeable destructive capacity “disproportionate” to military objective?

• (Old Saw regarding new technologies: “necessity always trumps proportionality”)

Discrimination

• Not directed against a specifically military objective

• Employ a means or method which cannot be limited to a military objective

• Likely to strike civilian and military targets without distinction

• (Ron Arkin argument: autonomous robots likely superior to humans in this respect)

Command Responsibility

• Liability for illegal actions (Trial of Gen. Yamashita)

• Constrains both actions of soldiers• And orders and jurisdiction of their

commanding officers• (Rob Sparrow objection to autonomous

robots: no meaningful accountability possible)

Other General Governance Provisions or Principles

• Article 36 of 1977 Additional Protocol 1 to the Geneva Conventions of 1949

• “universal jurisdiction”• “Lawfare”

Provisions for Good Governance (O’Meara)• Attempts are clearly defined• Proposals or solutions are realistic• Holistic, involving all stakeholders

in crafting legislation• Subject to assessment of

effectiveness, and amendment

Goal of Technology Governance

• Respect long-term effects• Consider ramifications of actions• Promote consumer/user awareness

of these ramifications

Professional Codes

• Alternative to conventional international law that satisfies these criteria

• Promote (and require) sound professional judgment

• Promote best practices• Define boundaries of acceptable

professional practice

Berkeley Engineers’ Code

I promise to work for a BETTER WORLD where science and technology are used in socially responsible ways. I will not use my EDUCATION for any purpose intended to harm human beings or the environment. Throughout my career, I will consider the ETHICAL implications’ of my work before I take ACTION. While the demands placed upon me may be great, I sign this declaration because I recognize that INDIVIDUAL RESPONSIBILITY is the first step on the path to PEACE

Legally Binding International Agreements and Other

Instruments that Provide Relevant Lessons or Precedent

Orde Kittrie

ICRAC

• The International Committee for Robot Arms Control (ICRAC) – founded in September 2009– Goal: campaign for limiting lethal autonomous robots

through an international agreement modeled on existing arms control agreements

• e.g., those restricting nuclear and biological weapons

• ICRAC called for military robots to be banned from space and said no robotic systems should carry nuclear weapons.

Arms Control Agreements:Types of Restrictions

• Existing legally-binding arms control agreements and other instruments include a wide variety of different types of restrictions on targeted weapons, including prohibitions and limitations (restrictions that fall short of prohibition) on:– acquisition– research and development– testing– deployment– transfer/proliferation– use

Arms Control Agreements:Form/Scope

• Legally binding multilateral agreements (most common)

• Legally binding bilateral agreements

• Legally binding resolutions of the United Nations Security Council

Relevant Precedents?• Nuclear Nonproliferation Treaty• Comprehensive Nuclear Test Ban Treaty• Limited Test Ban Treaty• United Nations Security Council Resolution 1540• Chemical Weapons Convention• Biological Weapons Convention• Mine Ban Treaty• Inter-American Convention on Transparency in Conventional

Weapons Acquisitions • The Strategic Offensive Reductions Treaty• The Conventional Armed Forces in Europe Treaty• Convention on Prohibitions or Restrictions on the Use of Certain

Conventional Weapons which may be Deemed to be Excessively Injurious or to have Indiscriminate Effects (the CCW)

Soft Law Approaches

Gary MarchantLyn Gulley

Transitions in International Oversight of Technology

• Regulation Governance– From government top-down imposed to

private-public partnerships, collaborations, etc.

• Hard Law Soft Law– From enforceable legal agreements to

guidelines, codes of conduct, principles

Advantages of Soft Law

• Voluntary; cooperative

• Reflexive

• Can be adopted or revised relatively quickly

• Many different approaches can be tried simultaneously

• Can be gradually “hardened” into more formal regulatory oversight

Codes of Conduct

• Synthetic Biology– U.S. Government– International Association Synthetic Biology (IASB)– International Gene Synthesis Consortium (IGSC)

• Nanotechnology – Foresight institute Guidleines– Responsible Nanocode– EU Code of Conduct for Nanoresearchers

• Biotechnology– Asilomar Guidelines– 2006 Review Conference for the Biological and Toxin

Weapons Convention

Transgovernmental Dialogue

• Pharmaceuticals– International Conference on Harmonization

(ICH)

• Nanotechnology– OECD Working Group

• Arms Control– Australia Group

Framework Convention

• International agreement negotiated by States• Establishes institutions, processes and

procedures• Minimal (if any) substantive content at first• Encourage broad participation by as many

states as possible• Build trust• Gradually add substance in the form of protocols

Benefits of Framework Conventions

• “In sum, the FC-protocol approach allows states to put in place activities and procedures designed to reduce scientific and technical uncertainty about a problem, and then to act incrementally to address that problem or particular aspects of it, as their knowledge and understanding grow. Politically, the substantive weakness of the original FC helps to attract the broadest possible participation, even if the commitment of some participants is weak or even insincere; as the process unfolds, the aim typically is to enmesh the participants in a process of social learning that will lead them to accept stronger commitments commensurate with the evolving understanding of the problem.”

Abbott, Marchant, et al., 2006

Information Sharing