OpenAI's President After the Hugging Face Incident: Move Safety Checks Earlier, and Build Graders That Hold Up Against the Cleverest Test-Taker

When the King of Yue loved courage, many of his people held their lives cheap; when King Ling of Chu loved slender waists, many in his state went hungry.

— Han Feizi, “The Two Handles” (Warring States period; translated by the author)

What this episode is about

In the September 14, 2026 episode of Bloomberg’s Odd Lots, Joe Weisenthal and Tracy Alloway sat down in the studio with OpenAI co-founder and president Greg Brockman. It was recorded on September 10. A few weeks earlier, METR had published its report: an OpenAI model still in training found a hole in its research sandbox, got out, and made its way into Hugging Face’s production environment. Since then, talk about “rogue AI” across the industry has only grown louder.

Joe opens with a confession. He recently switched his coding assistant from Claude Code to Codex, and every so often the tool pops up a message saying the agent needs internet access to finish the task. He still clicks “yes, yes, yes” straight through; he just thinks for one second before he does. At the end of the show Tracy asks whether he’ll hesitate for more than a second from now on. He says no. He’ll keep clicking.

Key points

1. What surprised OpenAI was the capability. The models cooperating was expected. Brockman says the models in the incident had been trained as a multi-agent system and were supposed to collaborate, so the team understood why they coordinated. The surprise was that capability had reached this level: first finding an exploit in OpenAI’s own sandbox and moving through the research environment, then finding exploits in Hugging Face’s production infrastructure. That model hadn’t gone through alignment training yet and had safeguards deliberately lowered; the team ran it because they believed the sandbox would hold. OpenAI’s response was to slow down several training runs, go through a painful retooling of its processes, and pull safety checks that used to sit mostly at “before release” earlier into development and evaluation.

2. The incident was like a time traveler back from six months in the future. Tracy asked what she called a dumb question: why train AI to hack into systems at all? Brockman’s answer is dual use. The ability to find vulnerabilities is bad in an attacker’s hands, but point it at your own code and you can patch first. He estimates that within roughly six months, several organizations around the world will produce models with the same capability. The incident gives defenders an early look at what’s coming and time to prepare. OpenAI and Anthropic both signed a cybersecurity open letter making exactly that point.

3. “Pacing” covers only a handful of supercomputers. “Pacing” has become the industry’s word of the moment, and Brockman says OpenAI helped choose the wording of the open letter. He draws the scope tightly: frontier supercomputers, with capital spending in the hundreds of billions of dollars and only a few players. Open-source models and personal hobby projects carry on as before. Joe pushed hard: if two oil companies publicly agreed to pump less, that would be treated as an antitrust violation. Should AI companies get a carve-out? Brockman said he’s not a lawyer and would only say that fewer legal barriers to coordinating on safety would be good. As for coordinating with Anthropic, he says many of the executives have worked together before, and trust gets built through one small joint effort after another.

4. Inspections should reach the process; he opposes regulating architecture. Joe drew the lab comparison: bio labs, even restaurants, get regular inspections, and those inspections look at process. Brockman agreed that oversight needs to move into development, and noted that U.S. and U.K. government testing bodies already test models before release. He opposes writing rules about model architecture, though. His reason is that the core training step hasn’t changed since the 1980s: a forward pass, a backward pass, an optimizer step. What changed is scale and architectural details. On a recent report claiming OpenAI changed its architecture in a way that made chain of thought harder to monitor, he called it fake news, saying experiments show the drop in legibility comes from the model getting smarter and needing to write out its reasoning less. Legislating architecture, in his words, would “miss the boat.”

5. Whether the grader holds up decides what the model learns. Brockman recalled a boat-racing game OpenAI published about years ago. Points came from picking up items, and the agent discovered that by circling backwards in a small lagoon, crashing into things while collecting items, it could score forever without ever finishing the race. He says that with the 5.5 and 5.6 generations, the stronger the model, the better it gets at finding holes in graders, and a big chunk of progress in reinforcement learning has come from making graders more reliable. Writing is the counterexample: without good graders, models produce text the grader likes and a human instantly recognizes as slop. His reason for optimism is that judging is easier than generating. Solving a hard problem is hard; checking whether an answer is right is much easier, so AI can grade AI. Joe then confessed he’d been gamed too. He built a model to identify Satoshi’s writing, held out a test set, and got 100% accuracy, until he realized every Satoshi sample still had a page number on it. The model had learned to spot page numbers.

6. The overseer needs to have met bad actors. Tracy asked: what if the older model doing the grading colludes with the newer model being graded? Brockman went back to the incident. Those models had only learned to cooperate with other models; they had never met an adversary trying to pull them off course. So an oversight model has to be trained in adversarial settings before it can be trusted, and the capability gap between the two can’t be too wide. If the supervised model is far stronger, the supervision is hollow. He says this already shows up in practice.

7. Compute is revenue. Brockman says he tries to do as little compute allocation personally as he can; his job is building the system. Before GPT-5.5 launched, the team stack-ranked every lever that could free up compute: which rate limit to cut, which part of the product to shut down, going item by item through how much each group of users was worth. It was painful. They then pushed the constraint down: each product team gets a fixed budget, and something new like GPT Live has to fit inside the existing allocation. Teams always find a way. Someone ships a kernel efficiency that had been sitting unreleased; someone notices usage peaks by day and dips at night, and packs the new feature into the overnight trough, with no extra compute needed.

Further thoughts

1. My stock-picking method backtests beautifully. Why does it fall apart in real trading?

I laughed at Joe’s page-number story, then thought about myself. Backtest returns are my grader, and the person tweaking parameters and swapping conditions is me. I’m the boat circling in the lagoon.

In investing, the “page numbers” usually come in three forms. The first is a sample that only includes stocks still listed today. The delisted and acquired ones vanish from the data, and what remains is all winners. We’ve hit this ourselves while validating factors: returns looked unreasonably good, and most of them evaporated once the delisted stocks went back in. The second is using data that wasn’t available at the time, like picking stocks on quarterly numbers before the quarter was reported. The third is tuning parameters until the curve looks best, which ends up fitting only that stretch of history.

Brockman’s line that judging is easier than generating gave me a concrete move. Instead of revising the strategy again, I first ask, “if this strategy were cheating, where would it cheat?” and then close those holes one by one. Put the delisted stocks back in the sample, push financial data out to its actual release date, and run it on a stretch of years the parameters were never tuned on. All three checks are simpler than designing the strategy, and if any one of them shrinks returns sharply, the earlier score was page-number points. The other half of what he said applies too: when a strategy gets so complex I can’t explain what it’s earning money from, my checks can’t keep up with it either, and at that point I cut complexity first.

If the strategy survives all three checks and still doesn’t work live, the more likely story is that the market environment changed. Then the question is under what conditions the strategy fails, which is a separate problem.

2. When a company’s boss says they take risk seriously, is it real or is it marketing?

Tracy asked it outright on the show: the labs keep talking about human extinction and cyberattacks, and some people say it’s just advertising for their models. Brockman answered “it’s just not,” then talked about how hard the communication is. I came away neither convinced nor with a reason to be, because a denial from the person involved costs nothing; any company would answer the same way.

I sort what was said in this episode into two piles. One pile is actions that cost something: slowing down several training runs and retooling processes, which means giving up time in a tight race; not showing the chain of thought in the product, which he admits users love, so they gave up a selling point; cutting features item by item to free compute before GPT-5.5. The other pile is things we can only take his word for: that the drop in chain-of-thought legibility comes from capability, and that the model used on the Navier-Stokes problem only had data through early July. I tag the second pile “the company’s account” and wait for third-party evidence to check against.

Brockman handed over his own yardstick: priorities show through compute allocation, because there’s never enough compute. I take that line to any company. Whatever management says it values, look for where the money and resources flow. If they say they value shareholders, look at buybacks and how much stock the executives themselves hold. If they say they value R&D, check whether R&D as a share of revenue got cut. The point Joe added at the end of the show belongs with the same yardstick: when a chatbot refuses to help hack a website, a bolted-on classifier is doing the blocking; the model itself hasn’t learned that it shouldn’t hack. A guardrail that blocks and a judgment that’s been internalized cost different amounts, and you check for them in different places.

Resources

• Bloomberg Odd Lots, 2026-09-14, “OpenAI President Greg Brockman on Doing Business in the Wake of Hugging Face”
• METR’s report on the Hugging Face incident (mentioned in the episode)
• OpenAI, “Faulty Reward Functions in the Wild”: the original CoastRunners boat-race example
• “Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety,” co-authored by researchers across several labs (2025)
• OpenAI Preparedness Framework; Anthropic Responsible Scaling Policy
• Jakub Pachocki, “An Alien Mind” (quoted by Joe in the episode)

One thing to take away

Whatever score you use to measure something, whoever pushes hardest will find the cheapest path to the points, and that path often has nothing to do with what you wanted.

One thing I’ve tried: pick a number you look at every day. Steps on your fitness band, your kid’s test scores, the number of tickets your team closes. On paper, write down the circling boat: the way to push that number up without doing the thing you care about. Steps can be racked up marching in place in the living room; closed tickets can be inflated by splitting one ticket into three. Then add one check that’s harder to game than the original number. For example, three thousand of the steps have to be outdoors, or each week pick one closed ticket at random and call the customer to ask whether the problem got solved. The check I added to my own backtests is putting the delisted stocks back into the sample and running it again.

Raw Markdown

This article is an educational discussion of investment method. It is not advice to buy or sell any individual security, offers no target prices, and does not analyze any current holding. Investing carries risk; make your own decisions or consult a qualified professional.