AI Scaling Panic: Everyone's Wrong But Especially The Doomers

Scott Aaronson dropped a truth-bomb last week that nobody asked for but everyone needed: "If AI scaling is to be shut down, let it be for a coherent reason." Boom. Mic drop. Queue the sound of a thousand Twitter philosophers screaming into the void.

Here's the setup. We've got two camps screaming at each other across the AI battlefield like it's 1999 and we're arguing about Napster. Camp One: the doomers. They want GPT-5 canceled, GPU clusters dismantled, and Sam Altman put in timeout. Camp Two: the accelerationists. They want GPT-7 by Thursday and think AI will literally solve death by Q2 2025. Both sides are incoherent. Aaronson, to his credit, sees through the noise.

Let's talk numbers before we talk theology. GPT-4 launched March 2023 with an estimated 1.76 trillion parameters across a mixture-of-experts architecture. Training cost: somewhere between $78 million and $100 million in compute alone. Claude 3 Opus dropped in March 2024, reportedly matching or beating GPT-4 on major benchmarks while Anthropic burned through $7.3 billion in total funding. Google's Gemini Ultra launched in December 2023 after reported training costs exceeding $191 million. These are not startup numbers. These are "small nation-state GDP" numbers.

And for what? MMLU scores went from 86.4% (GPT-4) to 86.8% (Claude 3 Opus) to 90.0% (Gemini Ultra). We're spending 2x more money to gain 0.4 percentage points on a benchmark most humans couldn't pass. This is the AI scaling equivalent of paying $500 for a pair of Labubu figures because the last one had a slightly different shade of pink.

The doomer argument goes like this: AI might become superintelligent, kill everyone, therefore stop. But here's the problem, and Aaronson nails it—this argument could apply to literally any technology. Nuclear physics gave us both cancer treatments and Hiroshima. The internet gave us Wikipedia and 4chan. You don't stop progress because of hypothetical worst cases. You build guardrails. You test. You iterate. You don't burn down the lab because the beaker might explode.

Then there's the pause letter. You remember—the one Elon Musk signed in March 2023 while simultaneously founding xAI to build the exact thing he wanted paused. The letter called for a six-month moratorium on training models larger than GPT-4. Six months! As if existential risk from superintelligence operates on a quarterly earnings schedule. "Sorry, Skynet, we need you to hold off on the apocalypse until Q3. Budget reviews."

The real reason scaling might stop isn't doom or acceleration. It's money. Pure, boring, capitalist money. Training GPT-5 is estimated to cost over $1 billion. That's not R&D spending. That's a strategic bet the size of a small acquisition. Microsoft drops $13 billion on OpenAI. Amazon throws $4 billion at Anthropic. Google casually mentions they're spending billions on Gemini. At some point, someone in a boardroom asks the forbidden question: "What's the ROI on this?"

And the honest answer is: nobody knows. We're in the middle of the most expensive science experiment in human history, and the business model is still "figure it out later." Sound familiar? It should. It's the exact same playbook as crypto in 2021, VR in 2016, and the metaverse in 2021. Hype curve goes brrr, venture capital follows, products ship, and then... crickets. Or at best, niche adoption that doesn't justify the valuation.

Here's what Aaronson gets right that both extremes miss: coherence matters. If you want to slow AI scaling, give me a concrete reason. Not "maybe possibly perhaps something bad might happen." Give me: "Training runs above X parameters create measurable safety risks Y, and here's the peer-reviewed evidence." Or give me: "The energy consumption of frontier AI training violates climate commitments by Z gigatons of CO2." Those are coherent reasons. Those are reasons you can build policy around.

The incoherent reasons? "AI might become conscious and feel sad." "AI might decide humans are inefficient." "AI might do something we can't predict." These are science fiction plots, not policy positions. And I say this as someone who writes about AI for a living and genuinely worries about misuse. But worrying about misuse is different from demanding a global pause because you read Nick Bostski's book and got spooked.

Meanwhile, the actual harms are mundane and happening now. Copyright infringement. Deepfake porn. Algorithmic bias. Job displacement. These aren't sexy existential risks. They're boring, quotidian harms that require boring, quotidian solutions. Regulation. Enforcement. Liability frameworks. But those don't get you on the podcast circuit or land you a $200K speaking fee at Davos.

So where does this leave us? AI scaling will continue until it doesn't. The brake won't be pulled by philosophers or petition-signers. It'll be pulled by CFOs when the numbers don't work, or by regulators when the harms become too obvious to ignore, or by engineers when the scaling laws finally break and we hit the wall that every physics student knows is coming. You can't double compute forever. Eventually, you run out of atoms.

Until then, enjoy the show. GPT-5 is coming. Claude 4 is coming. Gemini 2.0 is coming. They'll be marginally better, massively more expensive, and hyped within an inch of their lives by people who should know better. And we'll be here, calling it like we see it, scanlines and all.

Aaronson's right: if we're gonna stop, let's stop for a reason that makes sense. But don't hold your breath. The hype train has no brakes, and the conductor is asleep at the wheel.