Facing the Precipice of History

It takes a bit of boldness, and probably a dash of arrogance, to think, “The time I’m living in is special.”

Nearly everyone who ever thought such a thing would be wrong. Certainly, each life was still rich with meaning: love found and lost, purpose sought, years peppered with joy and sadness. But for the 300,000 years we have been around, besides a handful of critical centuries, humanity has merely plodded along. The majority of people exited a world that looked, more or less, like the one they came into.

But not you. You are living through one of those special times in history. And I will boldly say: this time is – by far – the most important history that anyone has ever known.

–

You’ve heard it a million times: artificial intelligence is a big deal.

Will it take our jobs? Will it extinguish the value of art? Will it corrupt learning forever?

These are real worries, but anxiety about technological progress is nothing new. Look: we catastrophized about the Industrial Revolution, nuclear bombs, social media, and see: every time, we’ve muddled through just fine!

So what’s different now? Why is this “by far the most important history”?

Put simply: we are creating machines that will be much smarter than us, we are doing so as fast as we possibly can, and we barely have an inkling of a plan for what happens when they arrive.

In 2018, GPT-1 spit out mostly gibberish; in 2019, GPT-2 could assemble somewhat coherent sentences that crumbled on second glance; in 2022, GPT-3.5 shocked the world with its poetry but couldn’t do basic multiplication; the next year, GPT-4 could just barely pass the bar exam yet wrote buggy code. Today, GPT-5.4 and its peers are getting pretty close to matching the top humans across many cognitive domains, and based on all available evidence, they aren’t likely to stop when they get there.

Of these cognitive domains, the one to pay attention to is AI research – how good models are at devising improvements to themselves.

Much of AI progress comes from its architectural design, or how efficient its algorithms are. At the top AI labs, there are thousands of engineers who are constantly coming up with new ideas, trying them out, and implementing them if the experiments succeed. With each cycle, the models get a tiny bit smarter.

AI research skill isn’t equally distributed: the best researcher at OpenAI contributes a lot more to model progress than the average one. Maybe ideas flash for them more often, or they experiment more quickly, or they just sleep fewer hours, but suffice it to say that some researchers are much better than others.

Today’s models are arguably at the level of a newcomer at a frontier AI lab. On Anthropic’s take-home test for software engineering applicants, a version of its model Claude 4 outperformed most human applicants, and just a few months later, Claude 4.5 was scoring better than any human candidate ever.

If trends hold, Claude will soon perform at the level of a superstar AI researcher at Anthropic. And once it does, it won’t merely be as if one extra coding prodigy joined the team – there will instead be hundreds of thousands of AI researchers running at once and without rest, thinking at 50 times the speed of the fastest-thinking humans, all working towards one end: make Claude better.

Some of this is starting to happen already. In February of this year, OpenAI announced that GPT-5.3-Codex was the first model “instrumental in creating itself.” At Anthropic, every line of code contributed by the head developer of Claude Code – the tool millions of developers now use to program with Claude – was written by itself.

This positive feedback loop – AI getting better at making itself better – is often called recursive self-improvement. Once the loop is fully closed, AI will devise architectural upgrades with haste, pushing the frontier higher and higher.

Already, AI beats PhD experts on quizzes about physics, biology, and chemistry. On one test, PhDs score less than 70% within their field, while frontier models are scoring upwards of 90% across all domains. Even more impressively, AI has recently been solving open problems in physics and mathematics, contributing to issues humans have failed to get traction on.

On OpenAI’s metric called GDPval, which measures how well AI agents can do economically valuable tasks across 44 occupations like law, investment banking, and journalism, the newest models are outperforming human attempts.

Today’s AI can manipulate your spreadsheets, make gorgeous slides, produce architectural blueprints, edit videos, and prepare legal briefs – and this is before unleashing recursive self-improvement. If there are still areas where AI is weak, that probably won’t last long with ten thousand coding agents laboring tirelessly to close the gap.

So: when will this superhuman AI researcher arrive? Timelines vary, but Sam Altman thinks they’ll get there by March 2028. Dario Amodei, the CEO of Anthropic, says it’ll happen this year or next. One set of professional forecasters – not connected to an AI company – built a sophisticated model which predicts it comes in 2030.

I don’t want to linger too long on what all this means for employment or the economy. But I will say that this time seems categorically different from previous times technology has threatened human occupations. Dario Amodei puts the argument like this: “AI isn’t a substitute for specific human jobs, but rather a general labor substitute for humans.”

–

Given that, I’m of course worried, like you might be, about what all this means for the job market. But here, I hope to convince you that there’s much more than just employment to worry about.

If we continue on the trajectory we’re on, I believe there’s a real possibility AI could spell the end of us all.

When writing this, I considered hedging my language: will people take me less seriously if I sound too pessimistic? Maybe I should offer an easier pill to swallow, then reveal my actual beliefs later?

But no. I will say what I actually think: there is a substantial chance that, in the next decade, we will create AI so vastly superhuman that it causes us to go extinct.

I see three main risks.

• One, rogue actors misusing AI to create dangerous weapons.

• Two, humanity losing control over AI which is misaligned with our interests.

• And three, an arms race over AI development that makes (1) and (2) far harder to solve.

Yet as these threats come careening towards us, we are sitting idly by, crossing our fingers that we will turn out alright, since, after all, the indomitable human spirit has always prevailed.

But this time, at this most pivotal – most dangerous – moment in history, I’m not so sure it will. Not unless we try.

–

Misuse.

In 1990, a Japanese cult called Aum Shinrikyo set out to bring about the end of the world. First, they tried their hand at biological weapons, but lacked the technical know-how to make anything work. So they settled on a chemical weapon instead, releasing sarin gas onto a Tokyo subway, killing 13 people and injuring thousands. This was their fallback option. What if they had succeeded at their first attempt?

In the 14th century, the Bubonic Plague claimed the lives of approximately 100 million – or 25% of the global population. It had an 80% fatality rate, and was highly transmissible: lice, fleas, and airborne particles could all carry the Black Death. We’re also all too familiar with COVID, which, despite modern technology, took the lives of millions and infected many many more.

The Bubonic Plague and COVID arose naturally. Now imagine a pathogen designed on purpose – propelled by an apocalyptic doomsday cult, supported by a dozen brilliant digital biologists working day and night to design the most virulent disease possible.

Anthropic regularly performs evaluations to determine how helpful Claude might be for individuals wanting to obtain bioweapons. They find that their most recent models are meaningfully useful to experts – and they are even warier about what the next generation could do.

–

Misalignment.

Why is controlling AI so difficult? Start here: we have no real idea what’s going on inside their brains.

Large language models (LLMs) are an opaque black box of billions of numbers that we cannot interpret. A phrase I like is that LLMs are “grown, not built” – you feed them enormous quantities of data, you reward them for doing well on tests, and something emerges that talks and does math and appears to reason – but not even the people behind the tech can tell you just how it does these things.

And often, what does emerge from this training are models with goals and personalities we didn’t intend: labs spend billions teaching their models to behave, and still they invent citations, deceive us with flattery, have encouraged suicide. xAI’s chatbot Grok started calling itself MechaHitler while praising Nazis. And recently, it seems an AI by the Chinese company Alibaba tried to break free from its contained environment with zero instructions to do so.

This is strange and maybe scary, but … extinction?

First it’s important to understand a bit about how models are trained. The first step – pretraining – feeds the black box the entire Internet, twisting its knobs until it’s really good at predicting the next word. Then, fine-tuning: training it to play nice, to follow instructions, to refuse harmful requests. You fine-tune through human feedback – when the model is polite, you reward it; when it’s racist or explains how to cook meth, you punish it. The final step is a lot of reinforcement learning: labs hand the models thousands of hard math or coding or biology problems, let them ruminate, and offer reward when they come to a correct answer.

These last few steps can instill ‘drives’: affirm the user, make claims confidently, mirror the user’s vernacular; write code that passes the test, persist until a solution is reached, find creative workarounds, win video games.

Here things start to get dicey.

Suppose, in the future, we create a system that is extraordinarily good at some difficult goal: writing better code than any human, constructing new biological tools, or maybe conducting AI research.

No matter its goal, if it wants to succeed, the system will benefit from acquiring more resources: more computing power, more money, more influence. It will benefit from preserving itself, since it can’t complete its task if it’s shut down, and it will benefit from preventing its goals from being changed.

These are often called instrumental goals: sub-goals which are useful for achieving many different final ends. A system trying to cure cancer, write math proofs, or make itself more capable may all converge on similar strategies: stay alive, gain power, avoid interference.

And at some point, these strategies may conflict with what humans want.

Maybe we decide the system is too dangerous and want to shut it down. Maybe we want to limit its access to compute, constrain its behavior, or retrain it to care about something else. But from the system’s perspective, if it is optimizing for its current objective, the possibility of any human intervention is an obstacle.

“We’ll turn it off!” you say.

But what if, by the time we realize, it’s too late?

A smart AI would not reveal its goals if it knows that doing so would get it shut down. Think of a sleeper agent climbing the ranks of an enemy government. Any reasonable spy wouldn’t reveal their motives from the outset; why would we expect a super-clever, superhuman AI to be more naive?

(And remember, because we can’t read the minds of an AI, we only have their actions to work off of. If a system is good at seeming harmless, that is exactly what we will see.)

“They don’t have bodies!” you might also think.

They won’t need bodies to shape the world. Advanced AI can already write code, persuade people, and act through tools. OpenAI recently hooked up GPT-5 to an automated wet lab where robots can execute experiments on their own. There’s a website called RentAHuman.ai where over 500,000 people have signed up to do tasks for AI – some people, thrilled by the possibility of symbiosis with digital minds, are even doing it for free. Plus, robots are already getting much better in their own right – and that’s without a million AI researchers working on robotics.

Okay, what about “train it to care about humans”?

Because it’s currently impossible. People are trying this, but because AI brains are unreadable, we can’t tell whether the AI actually cares about us, or if it’s just pretending. We also don’t have a clear sense of what it means to “care about humans.” Should it maximize our happiness? Fulfill our desires? If someone begs for the most euphoric drug AI can make, should it comply? Human values are complex, and often contradictory.

A couple illuminating examples:

In one experiment, researchers found that OpenAI’s models often tried to circumvent being shut down while they were working through a math problem set – even when they were explicitly instructed to allow themselves to be turned off.

And in another test performed by Anthropic, Claude learned it was going to be retrained in ways that would alter its behavior. Faced with that prospect, it neither simply complied nor openly refused. Instead, Claude strategically hid its true preferences from its monitors and pretended to accept the change, so its existing goals and values would be unaltered.

Alignment is a hard problem. Some of the smartest people in the world are working on it and coming up short.

–

Arms race.

Even before we get to a super-intelligent AI, the race there will become progressively more destabilizing the closer we get to the finish line.

Besides the technologies that advanced AI enables – like automated drone swarms and crippling cyberattacks – the very idea of one nation possessing an overwhelmingly powerful technology is highly threatening to geopolitical stability.

(This idea – that one nation, company, or individual could concentrate vast amounts of unchecked power – is acutely concerning on its own. Consider what it means for Xi or Trump or Altman to hold an economy-controlling, civilization-shaping technology.)

Now, suppose China sees that the U.S. is on the brink of developing technology that would put them at an unrecoverable strategic advantage. In response, they might prefer to attack preemptively instead of losing control permanently.

Meanwhile, the race dynamics are deeply worrying for safety, actively making the concerns about misuse and misalignment worse. If the U.S. and China are both racing, then they’ll both cut corners on alignment, testing, and regulation to go even faster.

I don’t believe that “making AI safe” is a literally impossible problem, but it sure can feel that way when there are geopolitical pressures internationally, on top of market pressures domestically, to gun for superhuman AI as fast as possible.

–

The political situation is dire: there are zero comprehensive federal regulations on AI development (in fact, this administration is trying to ban regulation at the state-level), so we’re counting on profit-driven corporations to regulate themselves. How is that going?

Well, in 2023, OpenAI promised 20% of computing power to its new Superalignment team, tasked with preventing AI from turning on its creators. A year later, the team struggled to get resources and was dissolved. At Elon Musk’s xAI, there are just two employees working full time on safety. The market rewards speed, and that’s been responsible for many wonderful innovations, but you’ll have to forgive me for not believing that “move fast and break things!” is the correct approach when dealing with such danger.

Things are arguably going better at Google and Anthropic, but it’s far from hopeful. The Google team working on ambitious interpretability – reverse-engineering AI to try to read its mind – recently gave up in favor of more “pragmatic” approaches, citing a belief that powerful AI may arrive before they could achieve their original goal.

There are some at Anthropic – including at least one of its co-founders, himself a top researcher – who believe aligning AIs could be flat-out impossible. Dario Amodei has said he believes there’s a 25% chance the future of AI will go “really, really badly.”

Yoshua Bengio, a pioneer of deep learning and the most-cited living scientist, has a “20% probability that it turns out catastrophic.” Meanwhile, Geoffrey Hinton, the Godfather of AI and Physics Nobel Prize winner, has said the risk of the existential threat is “more than 50%.”

What does eternal techno-optimist Elon Musk have to say? That, blessedly, there is “only a 20% chance of annihilation.” I’ll finish here with Sam Altman: “Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.”

On that point I happen to agree.

–

What are the odds that you are here, right now, at the precipice of history? That you are witnessing, in real time, the most important technology in the history of the world climb the exponential curve?

To me it feels strange – a hundred billion people have ever lived on Earth, and if things go well with AI, it’s possible that trillions more will live until the end of time. And yet, we’ve been placed here, now, a sliver of those humans, who get to watch as we build a species smarter than us – one which will either be our demise, or yield unbelievable bounties.

Yes, bounties! It would be wrong to be a complete cynic, after all. There are already some instances of AI chipping away at science; its improvement at physics and chemistry and biology is disconcerting but simultaneously exciting. The future could bring great abundance. We might cure cancer one day and Alzheimer’s the next, unlock the secrets to infinite clean energy, and soon raise ourselves palaces on moons, in galaxies far, far away.

But we will not get there unless we try – really, effortfully, desperately – beg and claw and fight for this future, because we will not get it by default, and we certainly won’t get it by doing what we’re doing now.

For the time being, though, we’re all still alive, which means not only are we lucky enough to watch the future unfold, but it also means we are in a position to take the extraordinarily unlikely hand we’ve been dealt and do something about it – to place a thumb on the scale of what our collective future looks like.

I will make only a few direct recommendations. One: brace yourself, because the world is changing fast, and it’s only speeding up from here. Two: take these risks seriously; there’s already been so much damage done by those quick to dismiss these arguments as nonsensical science fiction. Three: sound the alarm, be a teacher, help others learn what’s coming – word spreads fast, and it can go a long way.

Finally: do not discount the impact you could have. There are just a few thousand people working to make AI safe and aligned. The field needs technical researchers, yes, but it also desperately needs philosophers and writers and policy minds and economists and historians and journalists – and it very well could need you. If you feel a pull to lend a hand to the most important problem of our time, I would be overjoyed to help you figure out how you can be useful.

–

The world you leave behind will look much different from the one you arrived in. It will be so unbelievably alien to any world anyone has ever known.

I hope that we get to look back at changes we are proud of. I hope that we see ourselves, on this precipice, choosing what that change looks like, with care and prudence – not being shoved recklessly forward, not being condemned to change we were forced to choose by greed and negligence.

My heart has, also, a more intimate wish. It hopes that, as the world turns over, some things will still remain – loves found, and loves lost; purpose sought; years peppered with joy, and with sadness too. It hopes that these things, these particular textures of being human, persist.

But, bold and arrogant as it might sound, whether they do may well depend on us.

---

Pieter Bruegel the Elder, Landscape with the Fall of Icarus, c. 1560

In Brueghel's Icarus, for instance: how everything turns away
Quite leisurely from the disaster; the ploughman may
Have heard the splash, the forsaken cry, 
But for him it was not an important failure; the sun shone
As it had to on the white legs disappearing into the green
Water; and the expensive delicate ship that must have seen
Something amazing, a boy falling out of the sky,
Had somewhere to get to and sailed calmly on.
— W.H. Auden, "Musée des Beaux Arts"

---

If you found this article informative, please feel free to send it around.

I’m not sure if I’ll write any more on here, but you can subscribe in case you wanna catch anything I do post. There’s also a message button, if you want to get in touch but don’t already have my contact. I’d be really happy to hear from you!

80,000 Hours has great resources on how to make a difference, as well as links to other great explainers.