Infra Play #156: Things are accelerating

So long Demis, and thanks for all the fish

The last few weeks in AI have been shaky, to say the least. Open-source models got a big win with Kimi K3, with neoclouds salivating over the inference demand headed their way, while OpenAI's GPT-6, still in training, hacked HuggingFace and several other unnamed organizations.

More interestingly, 1,350 AI researchers signed a petition asking the government to step in and regulate the pace of AI research, while much of Google’s AI leadership left to… automate more of science and AI research across several key disciplines.

Let’s start with the “Open Weights and American Leadership” open letter, which was signed by pretty much every player in cloud and infrastructure software except… Anthropic.

In the 1980s, early open-source software pioneers challenged the prevailing belief that software would advance only if companies kept tight control over their code. This movement pushed for a transparent ecosystem where developers around the world could study, modify, and improve software. Software developed by the open-source community now supports most of the internet and underlies systems used by the world’s largest technology companies, as well as the U.S. military and federal agencies conducting scientific research, cybersecurity, and other critical missions. Open source did more than lower the cost of software; it created a shared foundation of knowledge on which generations of American engineers and entrepreneurs built their institutional sovereignty.

…

In fact, openness may be one of the most important paths to AI safety and security. Relying solely on closed models is not inherently safe: they can be breached, misused, or fail in ways that outsiders cannot detect. And concentrating advanced AI capabilities behind a small number of closed models compounds that risk. It results in a small number of single points of failure, weakens competition, and leaves critical technology in the hands of a few providers. Open weight models, on the other hand, allow a broad community of researchers and developers to examine their behavior, identify vulnerabilities, develop safeguards, and improve them over time. Just as open-source software demonstrated that transparency can be more secure than obscurity, AI safety may depend on giving more people the ability to test and strengthen the models on which society relies. It allows for rigorous benchmarking and evaluation, red teaming, and protections tied to real and demonstrated harms rather than assuming that closed systems are safer by default.

Anthropic remains vehemently anti-open-weight (or even anti-open-source), hence their rebuttal, which leans on the core national security policies they have long advocated, and this entertaining blurb from Dario where he pretends “we’re all on the same side, you guys!”

This brings me to the open letter. I agree with much of it: open weights expand access to the AI economy, they strengthen competition at least for some use cases, and they give customers greater control. Concerns about distillation should be addressed through targeted legal and commercial frameworks—the same measure I described above. But I don’t agree with the letter’s assertions that open-weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. It seems at least as likely to me that the opposite will be true. For example, I worry that biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials, whereas defense against these agents is a multi-year operational task in the best case (as we saw with Operation Warp Speed)⁵. Questions like this should be empirically answered by rigorous pre-release testing, not assumed in advance.

To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category. We should instead focus on keeping powerful chips out of authoritarian hands, stopping industrial-scale distillation, and requiring safety testing of all sufficiently capable models, open and closed.

If it is not obvious, claiming to be a big supporter of open weights while also demanding chip controls, distillation enforcement, and mandatory testing, and somehow expecting none of that to strangle the ecosystem, is a pretend play.

Now, all of this can be considered business as usual, since companies such as NVIDIA and Microsoft (two of the key players behind the initiative) benefit significantly from their infrastructure being used by a range of players beyond OpenAI and Anthropic. Dario’s refusal to sign is entertaining (the letter essentially commits signatories to stop petitioning the government to not block Chinese models) but irrelevant in the greater scheme of things.

Then “Pacing the Frontier” dropped. It was triggered by the sheer amount of recursive self-improvement happening across the field, captured well by OpenAI in their rather scattershot and very technical post, “How GPT-5.6 Fuses Frontier Intelligence with Frontier Efficiency”:

The efficiency gains we delivered with GPT‑5.6 are the result of years of compounding improvements across the stack, spanning research, inference, and our agentic harness. The role of GPT‑5.6 in delivering many of these improvements makes us optimistic about how the pace of optimizations will accelerate. We’ll continue making greater optimizations in areas such as kernel optimization, alongside foundational improvements to our stack. We look forward to transferring these ongoing, under-the-hood improvements back to our users and customers in the form of more widely available, cost-efficient intelligence.

In the post they outline multiple areas where 5.6 Sol proved useful in troubleshooting and improving the model's own core workflows, with efficiency as the through line. That, combined with the minor matter of GPT-6 hacking HuggingFace, rattled the AI research community enough to trigger a researcher-only petition, full of some very interesting quotes.

AI could help create a dramatically better future, but that outcome is not guaranteed. The world’s leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.

To realize AI’s potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight. But each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress.

Building on work already underway to monitor frontier model releases:

We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

- 1,350 employees of frontier AI companies

Now, if it’s not obvious, right after the whole industry had a big debate about not banning Chinese open models, suddenly hundreds of people at Anthropic, OpenAI, Meta, and Google are asking for a “delayed pace.” By definition, unless the US government also forces China to follow suit, pacing anything makes very little sense. But let’s take a look at the actual statements.