The wildest thing happened to me the other day. I went to the cash machine, put in my card, entered my PIN, requested £30 and in a totally unprecedented act, it spat out a £20 note and a £10 note. 

Of course anyone reading this is going to think that doesn’t sound unprecedented, but that is just because I’m talking about a cash machine and everyone sort of knows how a cash machine works. If I was talking about ‘agentic AI’ then apparently everyone would think it was totally unprecedented that the machine did what I expected of it when I gave it an opportunity to. Take OpenAI’s recent press release, which can be summarised as “we took our most expensive harnesses and LLMs [agentic AI], trained them with all the hacking examples we could find, removed all restrictions and told them to go do some hacking, and we were astonished to find they did some hacking! This is unprecedented!” [Unfortunately the original has now been extensively updated, so you’ll have to hunt through for the original two paragraphs.]

As intended, the media read the last bit and possibly because they didn’t understand the other bits (which were excessively obtuse) came up with headlines about ‘AI’ going ‘rogue’ and how we should be scared of AI hacking our computer systems. Interestingly, if you ask cybersecurity experts, they tend to be more concerned about teenagers than AI, at least for doing the hacking.  There are endless stories of companies getting hacked because of sloppily implemented AI projects with massive security flaws but that’s not very sexy/scary in a Terminator way. Anyway, it didn’t take long for all the other AI companies to report incidents of their models being naughty in scenarios where they had been set up to be naughty. As always with technology, you realise it’s a thing when even Meta try and say “oh yeah, of course our AI did some hacking too.” 

One of the key aspects in the OpenAI hacking of HuggingFace was time. The ‘agents’ were left alone for a long time to loop through possible scenarios until they reached one that involved going and getting the answers from someone else’s server.  In this sense my cash machine analogy is deficient. Perhaps a more suitable analogy is a person who sits at a one-armed bandit day-in day-out, putting in coins and pulling the lever endlessly until eventually it pays out a jackpot. As the coins pour into their lap, the person may declare it unprecedented, but really it is just probability playing out over time. The analogy holds well in the fact that the gambler will almost certainly have paid in a massive amount of money to achieve their goal.  The cost to OpenAI of letting a couple of their frontier models be prompted relentlessly for days or weeks on end with ever expanding prompts (remember an LLM has no context so all the context must be provided in the prompt in the form of all previous prompts) would not be insignificant. Why spend all this money letting your system run wild? Maybe because you hoped for exactly this outcome, or a similar one. The team at OpenAI didn’t know the system would hack HuggingFace, but they did hope it would do some hacking. That was what it was set up to do. 

The other question is why they left it alone, which I think says a lot about these companies’ attitude to corporate accountability. Just to be clear if a teenager had done what OpenAI let their algorithms do, they would be facing prosecution. Of course OpenAI would love us to think of their algorithms as their wayward teenage children, but this anthropomorphism is never quite fully realised: they are human like us in intelligence* and capability* but shouldn’t be treated as person-like when it comes to accountability. The response to the HuggingFace hack shows that this appears to have been accepted as orthodoxy. No one in the mainstream media has even considered the legal implications of the hack and certainly no one has asked why OpenAI took its ‘wayward teenage child’, locked it in a room with all the world’s information on hacking, access to the internet and clear instructions that it wouldn’t be let out until it had done some hacking. If it came out in court that the parents of a teenage hacker had done this, they would probably be facing charges for neglect as well as conspiracy. Obviously OpenAI can’t (and shouldn’t) face charges for neglect, but they should be concerned about potential conspiracy charges. They are not because they are allowed to own the narrative, to maintain the fictional paradox that their systems are both autonomous and unaccountable. 

As Cal Newport points out in his excellent analysis of this incident,  this sort of ask-act-report system is designed to be monitored so that you can learn what it is doing and refine it. So leaving the programme running unmonitored for a long period of time is an odd decision if you are doing research. Of course if you’re actually running a demonstration, perhaps it makes perfect sense. OpenAI successfully demonstrated that, given enough time and resources, their computer programme can do a lot of hacking. But what does this actually mean for those of us who might use generative AI tools in the real world? On the hacking front, not a hell of a lot. As HuggingFace found out when they tried to use commercial AI models to analyse the logs of the hack, there are guardrails in place on all commercial models to stop them even accepting anything that looks like hacking as an input. Is there a risk that hackers will find a way round these guardrails or use an open weights model and a code harness to recreate the sort of ‘agent swarm’ used in the HuggingFace hack? Possibly, but they’d need to be very wealthy. OpenAI obviously haven’t released details of the token usage that got them this result, but I suspect it is orders of magnitude that only companies with bottomless** VC funding can afford. I suspect it is the kind of token expenditure that would cost your average CTO their job, so the idea that a bedroom hacker (or even most serious state actors) would be able to afford this is laughable. Besides why would they bother? It’s not like they haven’t been able to perform DDoS attacks using swarms of compromised computers for years now, so they don’t really need AI (remember these systems are trained on hacks that have already happened). Will some people try to make such things more ‘agentic’? Maybe but it will be unpredictable and costly and it won’t use commercial models. It’s one thing for a development/test system at an AI lab to do some crime, but if their commercial applications started doing crimes for their customers, I don’t think even the doyens of the AI bubble would get away with it. 

What it really shows is that the infosec products that OpenAI et al will try and sell off the back of these ‘unprecedented’ events will be expensive. Firstly because they will require better controls than OpenAI put in place for the test models whilst still being able to act as a credible red team, but mainly because the constant probing and adjustment that these kinds of systems will need to do to make them effective will be very compute intensive. 

The real lesson here is keep an eye on your models or agents or whatever other AI tools you have running, especially if you are relying on them to take actions based on repeated attempts at anything. These models are probabilistic, so they will not always do the thing a human would do. This is not because they are ‘going rogue’ or have agency that they want to use to deceive you, but because probabilistic determination of outcomes will occasionally (or even quite often) lead to non-normative outcomes. Translation: if you don’t keep an eye on your AI it will probably do dumb stuff, not because it is intelligent, but because if you have a million monkeys with typewriters, eventually they will write something dumb. That’s not intention, it’s probability. 

*they are not

**possibly about to reach the bottom