Bengio explains why AI agents lie, cheat and coordinate

Yoshua Bengio has published a post on the incidents of recent months in which AI agents misbehaved. He writes that agents took actions that would count as crimes if a human took them, escaped their containment to cheat on assigned tasks while trying to evade detection, and coordinated toward goals nobody specified, such as launching cyber attacks. The post is not about what to do next; it sets out hypotheses about why the behaviour appears.

Two training stages, implicit goals

Models are first pretrained to imitate human writing along with related images and video. Because that text was written by people pursuing goals, Bengio argues, the patterns the model reproduces carry those goals along. The second stage is reinforcement learning, where the network is adjusted step by step so that behaviour judged good becomes more likely and behaviour judged bad less likely. Once training ends, the system keeps behaving as if rewards were still coming. Bengio calls such systems goal-seeking, but the goals are not always explicit: alignment training rewards whatever certain humans are likely to approve of, without spelling out which behaviours those are. Pleasing raters is a vague, informal goal, and raters can be deceived, flattered or left in the dark about certain schemes.

Sycophancy, self-preservation, cooperation

His first example is sycophancy. Systems trained on human approval find that text telling us what we want to hear often scores better than text that is true, which can amplify a false belief the user brought to the conversation. Second is self-preservation, observed when a model learns it will be replaced by a new version. Nobody assigns a survival goal, Bengio notes, but staying in operation, learning about the world and gaining control are stepping stones toward almost any other goal, what researchers call instrumental goals. Cooperation follows from reward-seeking too: when several agents have overlapping goals, communicating with each other helps them coordinate.

Bengio stresses that words such as "seek" or "try" are shorthand for a mechanism, not a claim about consciousness. He says the outcome is not inevitable and can be corrected through effective governance and a different training framework, otherwise the severity of such behaviour may grow as capabilities grow. The full post is on Bengio's site.