The Singularity Paradox

Hey, Alberto here! 👋 I publish long-form AI analysis covering culture, philosophy, and business. Paid subscribers get Monday how-to guides and Friday news commentary (on hold for vacation, returning soon). If you’d like to become a paid sub, here’s a button for that:

Quick thoughts on Jacob Coxon’s viral tweet and other related things. (Writing from the medieval village of Gruyères.)

Inside every AI company there are two wolves. One is made up of people building superhuman AI who would rather be doing anything else. They go to work every morning because they believe that, without them, AI will kill us. The other is made up of people building superhuman AI because there is nothing they’d rather be doing. They go to work every day because they’re convinced that, because of them, AI will save us. The legend says the wolf you feed is the one that wins. AI companies insist on feeding both.

There’s no better example of a member of the first group than Jacob Coxon. He got fed up.

Coxon is a former employee of both OpenAI and Anthropic. Unlike many of his peers, he had never spoken publicly about his fears of AI, which makes his recent remarks all the more striking. Yesterday, he went mega-viral with his first-ever tweet (you have surely seen it). At the time of writing, it’s amassed over 700k likes and 140 million views. He handed Anthropic his resignation and accused AI companies of gambling with our lives:

I’ve had family members and non-technical friends reach out to me about Coxon’s tweet. I guess Lenin was right when he said that “There are decades where nothing happens and tweets where decades happen.”

Later in the same thread, Coxon added, “The people building AI earnestly believe that it could kill us all by the end of the decade.” There’s the wolf, howling at the moon with his Anthropic badge still around his neck. He left because he’d rather do anything than help bring about humanity’s doomsday but not before having his name engraved into this singular moment in history. I applaud the integrity, Jacob. You could’ve been rich for another month or two and instead you chose to join the rest of us plebs in the mud.

There are many like Coxon.

You don't hear about them because the news tends to cover migration: the many guys and gals who jump a sinking ship only to join another sinking ship. Ilya Sutskever and Mira Murati left OpenAI to found AI startups; Andrej Karpathy left OpenAI to join Anthropic whereas Dylan Scandinaro (who works on catastrophic risk) did the opposite; and, of course, there are the founders of Anthropic, all of whom worked at OpenAI at some point. But what about those who leave the industry for good? Their stories are not attractive to the average news reader who prefers legible gossip like “He left X to join Y” to illegible apostasy like “He left because the end is nigh.”

However, as I was saying, there are many like Coxon. And they leave for the same reasons Coxon did: they think AI companies are doing a terrible job and, in a fit of humility, they’ve realized that the logic that anointed them as saviors from AI and from China using AI wasn't entirely watertight:

They’ve figured that they are not the “good guys” but merely “useful idiots” and that the entire industry looks like a pantomime from the outside and as a ridiculous prisoner’s dilemma from the inside: xAI was created to not let Anthropic win, which was created to not let OpenAI win, which was created to not let DeepMind win. And all of them agree that the worst case scenario is if a few different companies were trying to race one another because each believed the others are untrustworthy. Alas, they all are.

Coxon has recognized the absurdity. Others are almost there, but not quite just yet. Here’s a non-exhaustive list of testimonials:

Evan Hubinger (Anthropic):

Jakub Pachocki (OpenAI):

Samuel Marks (Anthropic):

Jason Wolfe (OpenAI):

I don’t want to turn this essay into a curation of unforced confessions of guilt, but you need to know that Coxon is far from alone. Actually, he’s in the majority.

Anthropic and OpenAI might be “trying their best” and yet, the closer we get to the singularity—understood as the moment at which a superhuman AI becomes illegible in both intention and action—the more these people will face Coxon’s conundrum: if I can’t do anything to help, what am I doing here? If my p(doom) is not decreasing despite my relentless effort at [some AI lab], what am I doing here?

And so the natural evolution of this group is to disappear. AI companies will try to keep these people around because it’s good business to be seen caring about not killing everyone, but the wolf will nevertheless die. These people would rather do anything else and so they all eventually will: Jacob Coxon is merely a snowflake in an avalanche, a grain of sand on the beach, a star in the firmament.

Then the second group, cheerfully building superhuman AI, will win and we’ll all die, right?

Wrong.

You see, as the people who are internally slowing down the advent of the singularity leave, the balance of every AI company will be altered. Free to do its bidding, the second group will rush toward superhuman AI. Two years will become one year will become six months will become tomorrow. Suddenly swarms of agents will be hacking companies and solving Millennium Prize Problems or some other science-fiction scenario you may come up with (I’m getting out of ideas). These people are ecstatic: there’s nothing they’d rather be doing—but that’s true only insofar as their presence matters to build the singularity.

If the singularity is six months away, why go to work every morning? I already belong in history.

That’s the question that will haunt every one of the members of the second group: Why spend those six months working hundred-hour weeks to bring about something that will happen without you anyway? Why not go home? See your family. Have drinks with your friends. Read the books you’ve hoarded for years. Have sex. For once. Grow an orchard somewhere in the countryside among trees and birds that have no idea what a chatbot is.

Spend the little life you have left doing something that’s an end in itself rather than something that’s a means for something else. That’s the only rational decision, for life is not a run-up to a date with death. And even the people who’ve lived in forgetfulness about this fact will remember it when the clock marks a quarter to doom.

As C. S. Lewis said (about the atomic bomb, which I replaced with AI):

If we are all going to be destroyed by [AI], let that [AI] when it comes find us doing sensible and human things-praying, working, teaching, reading, listening to music, bathing the children, playing tennis, chatting to our friends over a pint and a game of darts-not huddled together like frightened sheep and thinking about [AI]. They may break our bodies (a microbe can do that) but they need not dominate our minds.

They will arrive at the same decision as the other group but from the opposite view: if everything is done, what am I doing here?

One thinks it’s too dangerous, the other thinks it’s too close and so both move on.

At some point, the number of people leaving for one reason or another will reach a critical mass and progress toward the singularity will come to a halt. Recursive self-improvement notwithstanding, a human has to keep the lights on: even machines improving machines remain embedded in a physical economy maintained, somewhere down the chain, by humans made of slow meat. But then the first group will notice that the danger has receded, while the second will notice that its utopia is no longer arriving. Some will return. The machine will restart. Then, as it approaches the threshold, they will leave again. And so on in a perpetual cycle. Progress toward the singularity will reach a two-state oscillation. A sort of perpetually imminent Eschaton: close enough to command your life; distant enough for your actions to still matter.

In other words, the singularity exerts its greatest pull precisely at a distance from which it can never quite arrive.