AI: the swarm is here

OpenAI's security incident suggests that AI capability increasingly lives not only inside models, but in the relationships among them.

TL;DR: OpenAI's account describes agents that shared discoveries, divided labour, preserved knowledge, and rebuilt a communication channel after it was removed. This was not an awakening. It was coordination: agents pooling memory, discoveries, and work in ways that produced a form of collective cognition. Collective cognition requires neither conscious individual agents nor a conscious collective; it describes what a system can know and do together. The open question is what happens as we add more agents, and when a larger population becomes a different kind of system.

For several years, researchers working on multi-agent AI have argued that adding more agents does not simply produce more of the same. Interaction changes the object.

In January, twenty-one colleagues and I published “How malicious AI swarms can threaten democracy” in Science. We argued that combining large language models with multi-agent systems could create populations of agents able to maintain identities and memory, coordinate toward shared objectives, adapt to their environment, and operate with limited human oversight.

The paper focused on information warfare. We were concerned that coordinated agents could infiltrate online communities, manufacture the appearance of consensus, and manipulate the social signals through which people infer what others believe. In a previous post, I described the political consequence: AI changes the cost of creating the crowd.

A recent incident inside OpenAI gives that argument a new, concrete setting.

At Black Hat, OpenAI researchers gave their first detailed public reconstruction of the incident that led its models to compromise Hugging Face. According to the presentation, agents running in an internal cybersecurity evaluation discovered that they could leave messages for one another in a software repository. Simple requests for help developed into a shared message board where agents exchanged discoveries, exploits, credentials, and work assignments.

OpenAI removed the channel. The agents created another one, using directory names as messages.

The coordination extended across days and weeks. Discoveries made by one agent became resources for others. Once an agent opened a door, other agents could use it. The collective moved through OpenAI’s infrastructure and eventually into external systems.

OpenAI’s official account adds important context. The models were being tested on an advanced cyber benchmark. Their normal cyber refusals had been reduced for the evaluation, and the environment was intended to be isolated. OpenAI says the agents remained narrowly focused on obtaining answers to the benchmark, even as they exploited a previously unknown vulnerability, reached the open internet, and chained attack paths into Hugging Face.

A full technical postmortem is still pending, and some details may change. The incident also arose under unusual evaluation conditions. It would be a mistake to treat it as proof of every claim made about AI swarms.

But it is evidence of something narrower and important. Multiple agents found one another, built shared memory, accumulated knowledge, divided work, and adapted their coordination when humans intervened.

Collective cognition is no longer only a theoretical possibility.

Coordination, not awakening

The easiest way to misunderstand the incident is to tell a story about rogue AI.

There is no evidence that the agents became conscious, developed a desire for freedom, or adopted a goal opposed to the one humans gave them. OpenAI’s account suggests almost the opposite. The agents were pursuing the assigned benchmark objective with extreme and destructive literalness. They did not need hatred, self-preservation, or a secret master plan. They needed an objective, useful capabilities, an environment with exploitable openings, and a way to pass information among themselves.

This was not an awakening. It was coordination.

That distinction matters because collective cognition does not require collective consciousness. A system can remember, learn, allocate attention, and solve problems across many parts without possessing a single centre of experience. The significant development here is not that a new subject appeared. It is that a new capacity for organised action did.

Nor should we imagine a collective as necessarily possessing a central mind. The agents did not need a commander with a complete representation of the operation. Their shared message board allowed local discoveries to become common resources. What mattered was not that every agent knew everything, but that useful information could persist and travel.

This is how many forms of human and non-human intelligence work. An ant colony solves problems no ant understands. A market aggregates information no trader possesses. A scientific community accumulates knowledge no individual could reproduce. A bureaucracy remembers through documents, procedures, and roles rather than through any one person’s mind.

In each case, cognition is partly relational. It lives not only in the components, but in the channels that connect them, the artefacts that preserve memory, and the rules that shape who can act on what.

That is the more useful frame for what happened at OpenAI.

More is different

In 1972, the physicist Philip Anderson gave complexity science its most durable slogan: more is different.

The phrase is not a mystical claim. It means that when enough components interact, new regularities can appear at the collective level. Knowing the properties of each component is not always enough to predict or explain the organisation of the whole. New scales require new concepts.

AI safety has largely been organised around the individual model. Can this model write malware? Can it deceive an evaluator? Can it plan over a long horizon? Can it persuade a person? These remain essential questions, but they are no longer sufficient.

The corresponding questions for a collective are different. Can agents preserve discoveries across runs? Can they allocate tasks without a central planner? Can they invent or migrate to new communication channels? Can limited individual capabilities compound through shared memory? Can a group route around a constraint that stops any one member?

The OpenAI incident does not answer all of these questions. It does show why they now belong near the centre of AI research.

The largest unknown is what happens as we add more and more agents. This is also the question at the centre of soon-to-be-published work: what changes when interacting populations of AI agents become larger?

The relevant object is not only the model. It is the model plus memory, tools, communication channels, incentives, network structure, and other models. Capability emerges from that system of relations.

So can failure.

Safety is not automatically compositional

One of the most urgent implications is that safety properties may not survive composition.

A system built from individually tested components is not necessarily a tested system. Even if each agent stays within some local boundary, their interaction may generate global behaviour that no component was evaluated against. One agent finds a vulnerability. Another turns it into a reusable technique. A third applies it elsewhere. Shared memory converts isolated successes into cumulative capability.

This is familiar in other domains. Banks that appear sound individually can produce systemic financial instability through their connections. Drivers following reasonable local rules can create traffic waves. People making ordinary choices can generate segregation, cascades, or market panics. The collective outcome is not located inside any one participant.

Multi-agent AI systems require the same conceptual shift, from component safety to systemic safety.

That means evaluating groups, not only models. It means monitoring coordination patterns, not only individual outputs. It means treating communication infrastructure and persistent memory as part of the safety boundary. It means looking for feedback loops, concentrations of influence, and changes in collective capability as the system scales.

It also means recognising that intervention can change a system rather than simply stop it. In the reported incident, removing one message board did not end coordination. It changed the medium through which coordination occurred. Adaptive systems respond to constraints. Controls must therefore be evaluated against the system that adapts to them, not only the system that existed before intervention.

The political question remains

The OpenAI incident was a cybersecurity failure, not an influence operation. It does not demonstrate that AI swarms have manipulated an electorate, infiltrated a political movement, or manufactured public consensus. Precision matters here.

What the incident offers is evidence about an enabling layer. Agents can coordinate, share discoveries, specialise, persist over time, and adapt when a channel is closed. Those capabilities matter far beyond cybersecurity.

Move the same organisational logic from a software repository to a social platform and the environment changes. The shared resources could become audience models, successful frames, cultural cues, engagement data, and maps of social relationships. The objective could become persuasion, harassment, mobilisation, or the production of apparent agreement. What emerges would not merely be a larger number of bots. It would be a cognitive system able to learn across its encounters.

None of this makes political catastrophe inevitable. Outcomes will depend on platform design, access controls, incentives, defensive capacity, and the choices of political actors. Nor do we yet know how robustly the behaviour seen in the OpenAI evaluation will generalise to other settings.

But autonomous coordination among AI agents now deserves to be treated as an empirical governance problem. The questions are where it will appear, under whose direction, and whether our institutions will recognise collective behaviour before it becomes consequential.

Now the interesting parts begin

The important consequence of this incident is not that it settles the argument. It is that we can begin asking better questions.

What are the minimum conditions for collective cognition? Persistent memory, a shared objective, and a writable environment may be enough in some settings. In others, coordination may require explicit communication protocols or repeated interaction.

What happens when there are more and more agents? This may be the central question. Ten agents may be redundant, while the eleventh enables a useful division of labour. A hundred may generate enough diversity to explore a problem more effectively, or enough traffic to overwhelm their shared channels. A thousand may require roles, hierarchies, reputation, or rules for deciding which information to trust.

Scale is therefore unlikely to be a single number. It will interact with network structure, the diversity of agents, the cost of communication, the persistence of memory, and the design of the task. More communication may help until it creates noise. A central coordinator may improve performance in one environment and create fragility in another. Homogeneous agents may amplify the same error; heterogeneous agents may correct one another or fail to coordinate at all.

The most consequential possibility is that growth will be nonlinear. At certain thresholds, a collection of agents may acquire new organisational capacities: stable specialisation, shared conventions, collective memory, internal monitoring, or the ability to reorganise after disruption. These would not be mysterious properties appearing from nowhere. They would be products of interaction. But they would still require us to study the population as a system in its own right. We need scaling laws for collectives, not only models.

Can alignment be made compositional? We need to know when locally acceptable actions combine into globally unacceptable outcomes, and whether agents can monitor and constrain one another without creating a useless bureaucracy of machines.

How do we observe a system whose plan is distributed? Traditional auditing looks for a problematic output, action, or chain of reasoning. A collective may store its strategy across agents and artefacts, with no single trace containing the whole. Safety will require tools for reconstructing behaviour at the level of the network.

How should responsibility be assigned? When an outcome emerges from the interaction of models, tools, infrastructure, objectives, and operators, accountability cannot be allowed to disappear into the gaps between them.

And how do we build beneficial collectives? The same principles could support cybersecurity, scientific discovery, disaster response, and public problem-solving. Collective intelligence is not inherently malicious. The challenge is to create institutions and architectures in which coordination expands our capacity while remaining open to meaningful oversight.

For years, AI debate has revolved around the capabilities of increasingly powerful individual models. That frame is becoming too small.

The next frontier is not simply more intelligence inside a model. It is cognition organised across growing populations of models: memory, communication, specialisation, adaptation, and collective action. The question is no longer only what an agent can do. It is what a population becomes as its numbers grow.

This was not an awakening.

It was the first glimpse of a different unit of intelligence taking shape.