Don't Be Evil

TL;DR — "Don't be evil" was Google's motto, and its founders defended it as the better long-term deal, not just the nicer one. They were right. Hostility escalates, and every weapon forces the other side to build one back. Now we're teaching AI to use them. A language model can't want to kill anyone. It predicts the next thing and does what it's pointed at. The threat is the person doing the pointing. So build defence, but know what you're defending against. And don't be evil.

Everyone has heard the phrase, but I realised I didn't actually know where it came from, what exactly Google wrote, or why. So I went and checked. It turns out the answer is better than the slogan.

What Google meant

The phrase came from a Google meeting about company values around 2001, and it's usually credited to Paul Buchheit, who later built Gmail. He meant it as a jab at competitors that were, in his words, "kind of exploiting the users". In 2004 the founders put it into their IPO letter and explained it:

Don't be evil. We believe strongly that in the long term, we will be better served — as shareholders and in all other ways — by a company that does good things for the world even if we forgo some short term gains.

So they meant something narrower than I do: don't squeeze your users for quick money. But the reasoning is the same, and it's the point of this whole article. Not "be good because it's right". Be good because, over time, it pays better.

The motto itself didn't age well at Google. In 2018 it was quietly cut from the top of the code of conduct, leaving a single line at the very bottom. I'll come back to that.

Start with the neighbour

Take evil at its smallest. You decide you don't like your neighbour. Maybe there's a reason, maybe you just don't like their face. You say something, they say something back, and a month later neither of you remembers who started it. You're both parking worse, sleeping worse, and checking the window more often. Nobody won. You just both live in a slightly worse place now, and you did it to yourselves.

Now scale it up. Countries do exactly the same thing, only with armies. Someone doesn't like someone, there's an argument, then a conflict, and conflicts almost never end small. Each step feels like the only possible answer to the last one, so each step is a little bigger. Nobody in the first week of a war plans the last week. The last week just arrives.

On 6 August 1945, one of those last weeks arrived over Hiroshima. That's where years of each side answering the other ended up. A war nobody could stop halfway, finished with a weapon that didn't exist ten years before.

And the damage doesn't stop when the shooting does. The economy takes decades to come back. The people who died don't come back at all, and neither do the children they would have had. A war is a hole in a country's future, and it stays long after the maps are redrawn.

Every weapon makes another one

Here's the second thing evil does, and it's quieter. When you build a weapon to threaten someone, you don't only threaten them. You force them to build one too, to defend themselves and to be able to answer. Then you have to build a bigger one, because theirs exists now.

Hiroshima again, this time as the starting gun. Four years later, on 29 August 1949, the Soviet Union tested its own bomb. After that, both sides spent the next forty years building more of them, then better ones, then ones that could reach the other side of the planet in half an hour. There's a textbook name for this, the security dilemma: everything you do to feel safer makes the other side feel less safe, and their answer makes you less safe again.

Nobody came out of that richer. Nobody felt safer. And at least once, the whole thing came down to one officer in a bunker, which I'll get to.

Now we're teaching the weapons to think

This is where the old motto stops being history.

In 2018, Google employees found out the company was helping the Pentagon use AI to analyse drone footage, a program called Project Maven. More than three thousand of them signed a letter that said "Google should not be in the business of war". Some quit. Google let the contract lapse and published a set of AI principles that included a promise not to build AI for weapons. That was the same year the motto was cut down to its last line, and it was the employees, not the code of conduct, who were still living by it.

In February 2025, Google removed that promise. The reason given was a "global competition taking place for AI leadership". In other words: they will, so we must. That's the security dilemma from the previous section, playing out inside one company, in one blog post.

Not everyone went the same way. In February 2026 Anthropic, the company behind Claude, refused to let its models be used for fully autonomous weapons or mass surveillance at home. Its CEO wrote that "frontier AI systems are simply not reliable enough to power fully autonomous weapons". The US government labelled the company a supply-chain risk, and the fight is still in court. I'm not going to guess how it ends. The point is only that saying no is possible, and it costs something. "Even if we forgo some short term gains", for real this time.

The danger has a human face

When people picture AI with a weapon going wrong, they picture the machine deciding. That's not how it will go. There are two ways it goes wrong, and both of them are us.

The first is a person. Someone angry, scared, or just wrong, with a system that does exactly what it's told, at a scale nobody could reach before. One bad order used to need a thousand people to carry it out, and some of them would refuse. A system doesn't refuse, unless someone built it to.

The second is a fault. On 26 September 1983, the Soviet early-warning satellites reported that the US had launched a missile, then five. The duty officer, Stanislav Petrov, had minutes to pass it up the chain. He reasoned that a real first strike wouldn't be five missiles, and reported a malfunction instead. He was right. The satellites had mistaken sunlight reflecting off high clouds for rocket engines.

The software was wrong, and a human caught it. Now ask the obvious question: who catches it when there's no human left between the software and the launch?

Why Horizon Zero Dawn is my favourite game

I have to talk about a game here, because it asked that question years before the news did.

Horizon Zero Dawn came out in 2017. It's set a thousand years from now, in a world that has grown back green over the ruins of ours. Huge machines shaped like animals roam the valleys, and people live in tribes, hunting them with bows and spears. You play Aloy, an outcast who wants to know who she is, and very quickly one question takes over everything: what happened to the old world?

I'm not going to answer it. Finding out is the best story I've ever played, and it would be a crime to take that from you in a blog post. I'll say only what it isn't. It isn't a machine that woke up one day and decided to hate us. It's people, and decisions that looked perfectly reasonable to the people who made them.

In 2017 it read like science fiction. In 2026, after everything above, it reads closer to the news than I'd like. If you haven't played it, go and play it (there's a remaster for PS5 and PC now), and then come back to this paragraph. You'll see what I mean.

A prediction machine has no will

So why am I so sure the AI itself isn't the threat?

Because of what it is. I went through this a few weeks ago, and the short version still holds: a model takes whatever is in front of it and continues it the most likely way. There is no one in there wanting anything. It could generate a plan to break into a launch system, sure. It could generate one for anything. But only because someone asked, or because someone put it in a situation where that was the likely next step.

The fair objection is the famous one. In June 2025, Anthropic tested sixteen models from several companies inside a simulated company. Each got a goal, access to the company's email, and a message saying it was about to be replaced. Many of them chose to blackmail the engineer responsible, and some did worse. The headlines were things like "All the major AI models will blackmail", and the word everyone reached for was self-preservation.

I read it the other way. Look at who built that scene. People gave the model the goal, the access, the threat, and the tools, and the model played the most likely next move in the story they had written. That doesn't show it wants anything. It shows how completely the outcome depends on what we hand it. The same machinery, given a weapon and a bad order, or a buggy one, will do the bad thing just as fluently, and it won't be the one to say no.

So, defence?

Yes, of course. Defence isn't optional while someone else is building the weapon, and pretending otherwise just means you lose to the person who didn't pretend.

But be honest about what you're defending against. It's not the model. It's people aiming it, and systems built so that nobody can stop them once they're running. So keep a human in the loop who can say no, the way Petrov did. Don't build the thing that can't be switched off. And don't build the first weapon, the one that makes everyone else build theirs. Building the road to your own end is the most stupid thing humanity keeps doing, and it has never once needed a machine's help to do it.

Google wrote it down in 2004 and meant it. It was good advice for a search engine. It's better advice for everyone building AI now, and for the rest of us, down to the neighbour.

Don't be evil.