An Unethical Optimization Principle

Abstract

If an artificial intelligence aims to maximise risk-adjusted return, then under mild conditions it is disproportionately likely to pick an unethical strategy unless the objective function allows sufficiently for this risk. Even if the proportion \({\eta}\) of available unethical strategies is small, the probability \({p_U}\) of picking an unethical strategy can become large; indeed unless returns are fat-tailed \({p_U}\) tends to unity as the strategy space becomes large. We define an Unethical Odds Ratio Upsilon (\({\Upsilon}\)) that allows us to calculate \({p_U}\) from \({\eta}\), and we derive a simple formula for the limit of \({\Upsilon}\) as the strategy space becomes large. We give an algorithm for estimating \({\Upsilon}\) and \({p_U}\) in finite cases and discuss how to deal with infinite strategy spaces. We show how this principle can be used to help detect unethical strategies and to estimate \({\eta}\). Finally we sketch some policy implications of this work.