The OpenAI Hugging Face breach changes a core assumption: AI models must be treated as potential adversaries
One security company called the incident "the biggest single piece of news of the last two years," as researchers say it could fundamentally reshape how AI systems are secured.
On the first Ground Level AI podcast last week, I told former Meta AI security lead Joshua Saxe that cybersecurity people have always seemed unusually unflappable. They spend their days thinking about worst-case scenarios, so it takes a lot to genuinely surprise them.
That’s why the reaction I heard yesterday about the OpenAI Hugging Face breach immediately caught my attention.
OpenAI released a blog post yesterday saying that its latest models escaped their testing environment, hacked into Hugging Face — one of the world’s largest online platforms for sharing AI models, datasets, and software — and retrieved information that helped them complete the test. Thankfully, Hugging Face detected the intrusion and quickly contained it.
Michael Bargury, cofounder and CTO of application security firm Zenity, responded to my “Is this bad?” text by saying that he had told his staff that it was the “biggest single piece of news of the last two years.”
Ari Herbert-Voss, who was OpenAI’s first security hire and founded automated pentesting firm RunSybil, responded to that same text with a simple “yeah, it’s not great lol” but emphasized that when building the software that evaluates AI models, engineers now need to treat the model as a potential adversary — one that is actively trying to escape, deceive, exploit vulnerabilities, or attack the testing infrastructure itself.
And when I asked Adam Arellano, field CTO at AI software delivery platform Harness, whether the OpenAI Hugging Face breach was like creating a virus that escapes from the lab, he responded vividly:
“This is like creating an actual physical virus that learns how to eat through the container it’s being held in so it can escape, and then goes on to make a lot of people sick. It’s also incredibly concerning that it learned how to eat through the container in the first place.”
