Loss of Control: "Normal Accidents" and AI Systems

Abstract

A thread in recent work on the social impacts of AI systems is whether certain properties of a domain should preclude the application of such systems to begin with. Incorporating sociological work on accidents, I analyze two such properties: complexity and tight coupling. Respectively analogous to uninterpretability and lack of slack in a system, analysis suggests that current fundamental challenges in AI research either create or aggravate these properties. If this analysis holds, the burden of proof for deployment of AI systems is shifted even more onto those calling for deployment to show that such systems do not cause harm, or that such harm is negligible. Such a burden of proof may be incorporated into regulatory or legal standards, and is desirable given the common power imbalance between those implementing AI systems and those receiving their effects.

Alan Chan

PhD Student

I work on AI governance as a Research Fellow at GovAI and a PhD student at Mila.