OpenAI let their models breach Hugging Face

I wanted to wait a bit before I wrote something about the OpenAI breaching Hugging Face incident because I wanted more details to come out before we jump to an conclusions about this event. Without really any details, OpenAI can make all sorts of claims about how impressive a feat this is, the new model deciding how it would be easier to compromise Hugging Face to steal answers for the task it had than it would be to just solve the task. But once details start coming out, you can see that what is a technically impressive feat in theory, is really only possible because OpenAI failed to monitor and maintain their testing environment.

OpenAI put out a statement about a week ago addressing what they had discovered from their side of the incident. That's the first issue I have. If they're investigating this incident, that says to me that they weren't really paying attention to their testing environment. And it went on for multiple days even further confirms they weren't paying attention. I don't know what their testing environment looks like, but for a company with so much money being thrown at it, I don't understand how they can't have a fully air-gapped system for testing these new models on. If they really didn't want it to have internet access, they could have done that. These are software models, they can't create internet access if there is no physical capability for it.

I am going to allow for some leeway on that though, we living in a world where remote access is possible for people, and if you want the best engineers, you can't always make them move. You've got engineers from all over the world working on this, you need them to access telemetry of what's going on to access the model, so you build an environment that has some connections to the internet to allow it to be monitored. That's the second issue, where was the monitoring? Nobody noticed the spike of outbound traffic from the compromised proxy host? Nobody is reviewing the data that these models are accessing? It's a testing environment for cybersecurity capability, you don't plan to have any EDR or RMM software running on the hosts to see if your environment is being cyber attacked? OpenAI decided to take a cue from Ron Pomeil to just set it and forget it?

Another issue for another discussion is how these new test models were allowed to run without guardrails or monitoring, but Hugging Face had to spin up open weights models to review the data because the guardrails on the commercial available tools didn't allow the artifacts of the breach to be reviewed. And that really highlights my next point, the company who allowed this to happen has run to the US government to complain that open models are a threat and need to be stamped out or tightly controlled. I think it's very ironic that these people who took security of their own tools in such a lax manner is now saying they know what's best for security. The only thing that's really securing is their own income streams rather than cybersecurity as a whole.

At the end of the day, OpenAI either cares very little about security best practices or they hoped something like this would happen to allow headlines to be written about them. I think the second one is obviously much worse but possibly what happened and got out of hand. The sudden rush to announce Hugging Face as a partner after they announced an agentic model compromised them was a PR move to save face, maybe help reduce some of the criticism that would have come from Hugging Face about the incident. But look, now the headlines are all about OpenAI having a model that can compromise and infiltrate systems rather than Anthropic's Mythos and Project Glasswing.

My takeaway from the whole incident is that OpenAI was very reckless in their testing when they knew what they were trying to do. Testing an exploit tool without having the proper measures in place is just setting up trouble. I also don't think things are going to get better because the open models are going to follow in progress and not have the same guardrails that the commercial tools try to have in place. I don't know how an agentic SOC AI looks much different from a well run SOAR deployment, but I guess we'll find out sooner or later since the lesson most people are going with this is we need defensive models to protect from these offensive models. I'm ready to mark the calendar for the day a web platform goes offline because their defensive SOC AI decided taking the platform offline was the easiest way to stop the attack, even when explicitly told not to do that.