Supreme and Federal Court Judges’ Conference
Perth, 22-25 January 2017

Automated Weaponry and Artificial Intelligence: 
Implications for the Rule of Law
The Hon Justice Melissa Perry
Federal Court of Australia

1.	INTRODUCTION

On 28 July 2015, over a thousand experts in autonomous technologies wrote an open letter to the United Nations.  The letter warned of the dangers posed by lethal autonomous weapons (LAWS).  These weapons select and engage human targets independent of human intervention, replacing human soldiers with machines.  Artificial intelligence technology, we are told, “has reached a point where the deployment of such systems is – practically if not legally – feasible within years, not decades …”.  The message conveyed is that today’s science fiction may be tomorrow’s reality and that limits must be imposed on that progress to ensure that such technology is applied only so as to benefit humanity.
The growth and development in new technologies even in our daily lives bears out that message.  We are already living in a technological world beyond our imaginings when we were young.  Yet, as the letter warns, with such change comes responsibility.  The challenge for lawyers is to ensure that the development and use of new technologies, especially those designed to maim and kill, occurs lawfully and in a manner that ensures transparency and accountability.  No less is required for compliance with fundamental principles of the rule of law endorsed in the global Sustainable Development Goals (SDGs) by which member States of the UN resolved in September 2016 to “… promote the rule of law at the national and international levels …”.
Against that background, I will start by explaining what we mean by automated systems, and the contexts and manner in which automated weapons are currently employed or are in the process of development.   I will then outline existing principles governing the use and development of automated and potentially autonomous weapons systems, before identifying a number of key issues surrounding their use and development.  As Justice Logan has flagged, this provides a context in which the issue of accountability in particular for use of such systems under criminal and defence disciplinary law may arise in domestic courts, as well as accountability in international fora.

2.	WHAT IS AN AUTOMATED SYSTEM?
Automated weapons are so-described because they employ automated systems.  An automated system is, broadly speaking, a computerised process which uses coded logic or algorithms to make a decision or part of a decision, or to make recommendations.  But not all machines are created equal.  Automated decision-making systems can be used across a broad spectrum of government, civilian and military applications.  They also vary considerably both in their processing capacities and in the extent to which their operational capabilities are autonomous of or reliant upon human input.  Decisions can be wholly or partially automated as the process of automation is “characterized by a continuum of levels rather than as an all-or-none concept”.  Among other things, some of these systems have human involvement at the point of making a decision.  Other systems operate autonomously without further human involvement beyond the programming stage.  In other words, depending upon the system, humans may be “kept in the loop” of decision-making to varying degrees while, in others, humans are “outside the loop”, as it is colloquially known.  
Precursors to automated weapon systems which detect and react to incoming “threats” without the need for a human to “pull the trigger” have existed for generations.  One example is land mines.  These were first used widely in the First World War but are now prohibited under international law due among other things to their indiscriminate nature.  Analogous but more complex issues attend the potential development of lethal autonomous robotics with artificial intelligence.  Robotics of this kind would be programmed to make independent and complex decisions, to adapt to their environment and to act on those decisions – i.e. to independently select and attack targets.  
There is considerable interest in and funding of relevant research into this kind of weaponry.  This has prompted increasing international concern and debate.  For example, States party to the Convention on Certain Conventional Weapons (CCW) have met on a number of occasions in ongoing discussions concerning the issues that lethal autonomous weapons pose in terms of their legality and potential threat to the rule of law.  
So far as we know, lethal autonomous weapons do not yet exist.  Nonetheless weapons systems with significant autonomy are already in use.  Their use tends to be in defensive rather than offensive operations in simple, static and predictable environments.  For example, a number of defence systems are currently employed with human supervised autonomous modes which detect, track and guide weapons to intercept targets so as to protect civilian areas against short range rocket attacks.  However, Australian unmanned systems are not currently fitted with the capability to deliver lethal force.  An example of the use of unmanned platforms by the Australian Defence Force is the Heron-1 UAS – a remotely piloted aircraft deployed by the ADF in Afghanistan to provide commanders with real-time intelligence, surveillance and reconnaissance information.  Nonetheless, a Parliamentary Committee recommended in 2015 that the ADF procure armed unmanned platforms subject to the safeguards that they be used only by ADF personnel in accordance with international law and with appropriate transparency measures.  There are also reports of ADF personnel training in the US with the Reaper drone.  The Reaper is weaponised but retains a “human in the loop” meaning that ultimate control rests with a human operator. 
Technology utilizing autonomous systems is not limited to defence, but is widely used in civilian contexts.  Indeed this is a field where there may be a ready cross-over between technological advances initially developed for civilian use, on the one hand, and those for the military sphere or law enforcement applications, on the other hand.  As a simple example, one can readily envisage that technology currently in development for self-driving cars might be employed to improve the capabilities of remotely operated ground combat vehicles or other unmanned platforms.  In other words, the implications of developments in one sphere for the other cannot be left out of the equation.  
In common with automated weapons systems, automated decision-making systems in non-military contexts vary in their processing capacities and in the extent to which their operational capabilities are autonomous of, or reliant upon, human input beyond the software design stage.  This technology is not only employed in the private sector.  Governments in many countries now extensively employ automated systems to make decisions that impact daily on the rights of individuals.  These range, for example, from assessing a claimant’s entitlement to welfare benefits or assessing tax, to screening devices at an airport to decide whether a person should be permitted to enter the country or poses a security risk.  A timely reminder of the kinds of challenges which these automated systems may raise is afforded by the recent attention on Centrelink’s automated debt recovery system (described by one commentator somewhat colourfully as a “Weapon of Math Destruction”).  This example also reminds us that the implementation of automated processes has potentially serious human implications even outside the military sphere.  
Other significant questions which also arise include the risk that discretionary or evaluative decisions may be replaced with rigid criteria of the kind necessary to enable such automated systems to be utilised and the risk that algorithms which learn from patterns of behaviour (so called machine learning) may introduce gender, racial and other biases into decision-making.  The risks of machine based learning were all too vividly illustrated when Microsoft released a software program, Tay, on the internet to learn how humans talk in conversation and converse with them.  Within 24 hours, however, Tay was tweeting racial slurs and calling for genocide.
It follows that, while automated and autonomous weaponry systems raise most acutely the moral, ethical and legal issues associated with such technologies, they also provide a prism through which aspects of a broader debate as to the proper limits of computerised decision-making may be raised and considered.

3.	EXISTING LEGAL FRAMEWORK
Currently no international treaties deal specifically with automated or autonomous weapon systems.  Nonetheless, while weaponry of these kinds was conceived only after the rules of international humanitarian law had come into existence, those rules govern their development and use during armed conflicts.  Any other conclusion, as the International Court of Justice held in advising on the legality and use of nuclear weapons, “would be incompatible with the intrinsically humanitarian character of the legal principles in question which permeates the entire law of armed conflict and applies to all forms of warfare and to all kinds of weapons, those of the past, those of the present and those of the future.”  Importantly, these obligations include that in Art 36 of the Additional Protocol I to the Geneva Conventions on parties to determine whether a weapon or new method of warfare would be prohibited under the Protocol or any other rule of international law.  
Relevant also are international human rights.  While human rights obligations exist irrespective of the existence of armed conflict, during hostilities (and depending upon the nature of the right) their content may be impacted upon by international humanitarian law.  Consequently, a consideration of whether the taking of a life is arbitrary, for example, will fall to be determined by international humanitarian law as the applicable law during armed conflict.  If, in other words, the taking of a life during hostilities would not infringe international humanitarian law, it would not be arbitrary.  (I leave to one side for present purposes complex questions as to the extent of a State’s obligations to secure human rights beyond the territorial boundaries of that State.) 
I will turn now briefly to consider a number of key issues associated with the use of automated weapons systems. 

4.	KEY ISSUES
a.	The lawfulness of automated weapons systems in international humanitarian law
First, to what extent is the use of robotics in warfare lawful?
Subject to the setting of proper limits, technologies may and are employed to achieve legitimate military advantages and may even reduce casualties on both sides.  For example, it has been argued that “the greater intelligence, surveillance and reconnaissance persistence that can be provided by current unmanned systems can facilitate better target discrimination and lead to less incidental injury to civilians and damage to civilian property.”  It has also been suggested that the remote operators of unmanned systems may be less likely to resort to greater force to address threats, as opposed to troops in the battlefield who are personally at risk of harm.  The use of machines with autonomy short of full autonomy may also help to save lives in a manner preserving human dignity.  An example given is where the rifles of snipers undertaking a hostage rescue might be connected by a computer which releases force only when all of the snipers have a clear shot at the armed hostage takers.   
What then of lethal autonomous robotics – machines which identify and select human targets and implement their targeting decisions independent of human intervention?  Could machines of this character ever be developed and produced lawfully?  There is general acceptance that caution and some form of control are needed beyond that already afforded under international law.  Controls that have been suggested include to slow their proliferation, to limit such weapons to defensive applications, and to limit their firepower.  
But there is also a case that even these measures may not suffice and that by their nature lethal autonomous robots could not meet the requirements of international law.  Even if a decision by a human to take a life might be lawful under international humanitarian law, would such a decision be inherently arbitrary if entrusted to a machine?  Would it be acceptable under Additional Protocol 1 of the Geneva Conventions which provides that the acceptability of such systems should be examined according to the principles of humanity and the dictates of public conscience?  Such concerns led the European Parliament, for example, on 27 February 2014 to adopt a non-binding resolution on the use of armed drones which included support for a ban on “the development, production and use of fully autonomous weapons which enable strikes to be carried out without human intervention.” 
The twin rules of international humanitarian law of distinction and proportionality already present significant barriers to the use of lethal autonomous robotics.  The rule of distinction prohibiting the targeting of non-combatants or civilians requires complex assessments to be made of infinitely variable circumstances.  These include intentions, emotion and context which demand a complete “situational awareness”.  The complexity of modern armed conflict where, for example, combatants are not necessarily identified by uniform and civilians may be taking a direct role in hostilities, only adds to the potential ambiguity of determining a person’s role, if any, in hostilities at any given point in time. Equally, determining whether civilian losses would be proportionate to the military objectives sought to be achieved requires an abstract and qualitative assessment made on a uniquely case-by-case analysis.  Yet as one commentator has observed, “[t]hese are far from [the] algorithmic specifications for decision-making and action” which drive automated weaponry.
The use of automated systems where humans are left out of the loop would also impact on what might be termed situational innovation.  By this I mean the human capacity to respond by thinking “outside the square” in lateral and imaginative ways in response to particular circumstances.  In this field, this may mean achieving not merely a lawful result, but achieving a more compassionate one.  If I can return to the movie “Eye in the Sky” to illustrate the point.  The British and US authorities were advised that it was lawful to order an attack by a drone on a cell about to execute a suicide bombing.  The likely killing of a little girl selling bread from a stall proximate to the attack was considered proportionate to the harm that would be caused if the suicide bombing were permitted to proceed.  However, the agent on the ground took steps to avoid the child’s death by bribing a local child to buy the last of her bread in the hope that she would leave before the assault.  
b.	The “accountability gap”
In the second place, individual responsibility is key to ensuring accountability for violations of criminal law, both international and domestic, and of military law and is therefore critical to the rule of law.  Plainly a machine cannot be held responsible if humans are taken “out of the loop”.  Upon whom then would responsibility fall? Would it lie with the software programmers, those who design and produce the machines, military commanders, or political leaders?  The existing paradigm does not necessarily assist.  Responsibility depends upon the chain of command within military ranks.  Yet under Protocol I to the Geneva Conventions, a commander is not absolved from penal or disciplinary responsibility for a subordinate’s violation if she or he “knew or should have known that the individual planned to commit a crime” and failed to take all feasible measures to prevent the violation.  Among other things, this would require a sufficient level of understanding by the commander of the computer programming to attract liability under criminal law or military disciplinary law for its deployment in a given situation. 

c.	Humans “in the loop” and meaningful human control
Finally, is it a sufficient answer to these challenges to ensure human involvement in the final decision on whether a person is a lawful target?  While there is an emerging consensus in favour of a requirement that there must be “meaningful human control” over such decisions in order to comply with international law, there is lack of clarity as to precisely what this requires.  Does this mean, for example, that humans must remain in the loop of decision-making or is sufficient if a human is armed with the capacity to override the machine (colloquially called “on the loop”)?  
The indications are that it is generally accepted that humans must remain in the loop.  At the CCW meeting of experts on autonomous weapons systems in 2014 and 2015, a number of States indicated that human control was necessary.  The US Department of Defence has indicated that current policy is that “autonomous and semi-autonomous weapons systems shall be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force”.  The Australian Defence Force has taken the approach that it will “embrace semi-autonomous systems where that capacity can save lives or reduce expense … but where lethal force is involved a trained operator will remain responsible for the application of that force”.  Similarly the United Kingdom has taken the position that the operation of weapons systems will always be under human control, and that where a defensive system can operate in an automatic mode, there must be a person involved in setting the parameters of any such mode.
In determining what is sufficient or “meaningful”, it has been suggested that the capacity for human override or intervention to effectively reduce the risks of violations of international law would be largely illusory given that the processing speed of these machines is often measured in nanoseconds.  Further, even if delays were programmed into a system for “on the loop” intervention by humans, there is the added dimension that humans tend to trust the accuracy of automated systems in preference to their own judgment.  This phenomenon, sometimes called automation bias, may impact upon whether human operators decide to countermand machine recommendations or decisions. 
These and other concerns led the Special Rapporteur on extrajudicial, summary or arbitrary executions in 2013 to warn of the risks that the development of such robotics posed to the right to life, the rule of law and international security. He called upon “all involved – States, international organizations, and international and national civil societies – to consider the full implications of embarking on this road” and to do so while it is still possible.

5.	CONCLUSION
The imperative for the global community to define with precision the limits which constrain the development and deployment of automated and autonomous weapons is clear.  In line with the emphasis in the United Nations Sustainable Development Goals on the rule of law, this requires the erection of a more precise legal framework within which they may be developed, produced and used in a manner which will protect and preserve humanity.