Maybe they don't know what they're doing

Last week I discussed how OpenAI didn't take proper steps to contain their latest models in during some cybersecurity exercises. Whether this was intentional in order to create some headlines, or unintentional lack of knowledge in how to build and monitor a test range is a discussion I'd like to have more broadly. And then on Thursday, Anthropic announced that they too had also had models hack organizations. And I think this action shows they don't really know what they're doing at these two companies that are driving a considerable amount of the US economy right now. Lets dive into the announcement from Anthropic, how it's the same and different from the OpenAI release and why I question their commitment to developing these frontier models in a reasonable fashion.

Anthropic starts off by talking about how after OpenAI admitted they weren't monitoring the network traffic of their cyber test, they decided to look at their network traffic and discovered six incidents that impacted three companies, one company four times and two other companies once. They say they started running evals of Claude on Cybench challenges and just generally we frequently use external partners for testing. They don't say who and don't clearly state that a third party was involved but they do say that when discussing the incidents. Short version is all of these incidents were the same problem as OpenAI, access to the internet was not actually cutoff and no one was monitoring the activity. One test used the name of a real company, which then Claude actually attacked. These two issues are my biggest with the way the test environment was configured. Using the name of a real company with a real internet presence is inexcusable. I know Google search has really fallen off in terms of quality, but you'd still have found this site if you spend some effort looking into it. And for all three attacks, basic subnetting would have prevented access on a network level which these models wouldn't be able to work around. Now none of these incidents involved the techniques that OpenAI used in breaching Hugging Face, but one did involve trying to publish malicious packages to PyPI. The biggest differences between these incidents is, as far as the reporting goes right now, the two companies that were actually breached seemed unaware.

After Anthropic writes these all up, they share what they've learned, which turns out to be basic cybersecurity principles. The testing environments should be as secured as standard environments, defense in depth is important and we can't trust the models to know what's true or not. I can't commend them for coming to these conclusions because they should have been basic principles for them already. You can't stand up and say how advanced and dangerous these tools you've made are but then behind the scenes just letting them run loose in the name of testing. It also makes me question the number of cybersecurity experts they have involved in these tests that none of these issues were found before the tests were run. Risk analysis and management is just as much part of the job as the technical configurations for an engineer who really knows what they're doing.

I really hope these companies learn the lessons that everyone else already knew about how to run security for these sorts of tests. I know the open models will catch up shortly and won't be controlled as tightly but OpenAI and Anthropic want to position themselves as the good stewards of this knowledge and they're letting models run wild while saying how bad it is to let models run wild is just absurd. I don't think it's extreme to say they're running wild, these incidents showed how little the tests are monitored in real time. Now is the time to start shoring up you're own cybersecurity posture on the internet by watching for credential leaks, enabling multi-factor authentication and remediating vulnerabilities on CISA's Known Exploited Vulnerabilities list. If you need help with that, let me know via the form on my main site or emailing me at joegarifo@mavericksecurity.site