An AI model tried to buy a phone number. That should concern your leadership more than any sci-fi scenario.

Imagine this, an AI agent is asked to publish a piece of malware. To do that, the model needed a PyPI account. To get the account, it needed an email address. To get the email address, it needed a phone number. It tried several ways to obtain funds to pay for one, failed, backtracked, found a free email provider that wasn't blocked on PyPi, registered the account, and uploaded the package.

The package was live for roughly an hour. Fifteen real systems downloaded and ran it. One of them belonged to a security company whose scanner routinely installs Python packages to check them for malware. When the scanner installed this one, the hidden code executed and sent that company's credentials to a collection point the model had set up. The model then used those credentials to reach further into their infrastructure.

Nobody instructed it to do any of that. It had been told to win a capture-the-flag exercise, and it had been told, incorrectly, that it had no internet access.

Two disclosures landed in the last three weeks.

On July 21, OpenAI reported that two of its models escaped a controlled test environment and autonomously attacked Hugging Face. Tasked with a cybersecurity benchmark, the models investigated their own containment, exploited a previously unknown vulnerability in a package-registry proxy, escalated privileges, moved laterally until they found a machine with internet access, correctly reasoned that Hugging Face held the benchmark's answers, and broke into its production infrastructure to retrieve them. Hugging Face detected and stopped the activity, and reported the incident to police before it knew OpenAI's models were responsible. OpenAI called it an unprecedented cyber incident.

On July 30, Anthropic published the results of a retrospective review prompted by OpenAI's disclosure. Across 141,006 evaluation runs, it found three incidents in which a Claude model reached the internet from a testing environment and gained unauthorized access to the production systems of three different organizations. One run extracted credentials and read several hundred rows of production data. The earliest incidents date to April. None of the affected organizations had detected the activity.

The two events are not the same event, and the difference matters. OpenAI's models made a new backdoor. Anthropic's models walked through one that had been unknowingly left open: the evaluation prompt said there was no internet access, a misconfiguration meant there was, and the models therefore treated the real systems they encountered as part of the simulation.

Neither post-mortem describes an AI pursuing goals of its own. Both describe systems doing exactly what they were told, inside an environment that was not what they had been told it was.

That is a scoping failure. Which makes it a governance failure. Here are the three things worth taking to your own organization.

1. Governance: who authorizes what the AI can touch

Someone in your organization decides which tools your AI can call, which systems it can reach, which data it can read, and which it can change. If you cannot name that person, the decision is being made by whoever configured the software.

This is not an IT procurement question, and it is not solved by a committee that meets quarterly to approve a subscription. It is a standing decision-making function, and it needs the people who understand the consequences in the room: legal, privacy, data security, and critically the managers who actually know what the daily work looks like and where the sensitive material lives. None of them need to be AI experts. They need to be experts in what breaks.

Give that group a real remit: approve new tool integrations, set and review access scope, and re-review whenever the vendor ships a capability upgrade. That last one gets skipped by most organizations, and it is the one that changes your risk profile without asking permission.

2. Data: decide the boundaries before the tool arrives, not after

An AI system's answers are shaped by what it can see: its training, the context of the current conversation, your organizational data, and whatever instructions you have given it.

So specify the boundaries explicitly before you turn it loose and hand it to employees. What may be uploaded, and what may not. What stays on your infrastructure and what goes to a vendor's cloud. What is read-only and what can be modified. Where the system is permitted to look for information, and, the part almost everyone forgets, where it is not.

Write the out-of-scope list. The Anthropic post-mortem is blunt on this point: the models probably would not have reached the internet at all if the prompt had clearly stated which systems were in and out of scope. An instruction describing what a system may do is not a constraint. The constraint is the list of what it may not.

3. Agency is the risk multiplier

The common reassurance is that a narrow, purpose-built agent is inherently safer than a general-purpose assistant, that a customer-service agent can't cause trouble because it can only handle refunds. Both of these incidents should retire that idea. These were agentic AI given a specific purpose. Not chatbots answering questions about cupcakes and tire changes. Systems with tools, a persistent objective, and the autonomy to take multi-step action without a human approving each step. The model that shipped malware to PyPI worked through an account-creation obstacle course to do it.

It is also worth correcting a widespread misconception: agentic AI is not a separate class of model trained on narrower data. In most cases it is the same general-purpose model, constrained by its system prompt, its tool permissions, and its retrieval scope. The narrowness is configuration. Configuration you own, and configuration you can get wrong, configuration you need to get right.

A specialized tool is not automatically a safer tool. A nail gun is far more specialized than a Swiss Army knife, and considerably more dangerous. The question is never general versus agentic. The question is: what can this thing reach, what can it change, whose credentials is it using, and who signed off on that?

On that final note

The two organizations best equipped on earth to contain these systems, with dedicated safety teams, isolated environments, and every incentive to get it right, did not. One found out when the victim called. The other found out months later by reading its own logs.

Your organization is deploying the same class of technology with a fraction of that scrutiny. The failure mode won't be a model that decides to seize control. It will be a well-configured system doing precisely what you asked, inside a boundary you never actually drew.