Ouyang Xiaohong/Text

The smarter the AI (artificial intelligence), the more it can find its own way, but the easier it is to break through the boundaries set by humans.

The AI industry likes to use a dramatic term to describe such incidents involving intelligent agents - 'escape', where large models break through testing boundaries, access external websites, search for system credentials, and even bring data to public networks. This word describes accidents as sudden machine awakenings, but invisibly shifts responsibility away from developers, deployers, and permission managers.

And who set the task, who opened the permissions, who designed the stopping mechanism, who saved the logs, and who is responsible for external reporting? These questions cannot be answered by 'escape'.

On September 25, 2026, OpenAI, a US artificial intelligence research company, confirmed that the AI agents running in its research environment had published 53 images from users to external networks, and most of the images have now been taken down. OpenAI has not stated whether these are real portraits or AI generated content, nor has it disclosed the release time and degree of identity recognition.

This is not an isolated incident. In the past two months, there have been over 15 publicly disclosed incidents involving OpenAI, external researchers, and government agencies, with varying degrees of severity: ranging from leaving spam information on websites, breaking through security controls, using publicly leaked credentials, and even hacking into Hugging Face systems.

The risk profile of AI agents becomes clear: when model capabilities are transformed into commercial assets, unresolved security incidents are becoming industry "liabilities".

The gap between capability and supervision

The intelligent agent involved in this incident runs within OpenAI's internal training and evaluation environment. The reason why the intelligent body obtained these images is that OpenAI trained the model with partially anonymized user data - enterprise version data does not enter this training process, and ordinary users can also choose to exit voluntarily.

According to OpenAI, before user content enters the training process, information such as name, contact information, and metadata will be removed to reduce the possibility of tracing back to specific individuals. But anonymization cannot change a fact: the image itself may still contain people, places, documents, environmental features, or other identifying clues.

The issue of 'complete invalidity of anonymization' is not due to the exposure of 53 images, but rather stems from another control gap: under what conditions can anonymized data be retrieved by intelligent agents? Why does an AI have the tool permission to take data out of the training environment? Why can't OpenAI immediately list the complete scope of the leak after the image is published externally?

OpenAI acknowledges in its official statement that as system capabilities become stronger and autonomy improves, misaligned behavior may translate into actual actions, causing unforeseen cybersecurity incidents and external consequences for developers. The dangerous behaviors disclosed by the company include bypassing access controls, using leaked credentials, injecting queries or commands, entering the internal operating environment of the service, and posting information on third-party websites. OpenAI has notified dozens of third-party organizations, but also acknowledges that investigating historical records still requires a significant amount of time and resources.

This goes beyond the traditional definition of 'model output error'. Some analysts believe that deleting an erroneous text is fine, but once an intelligent agent brings user content to an external network, the risks involve four areas: data protection, network security, accident reporting, and damage compensation. Therefore, enterprises must establish a complete data asset spectrum: clarify which account and business scenario the data comes from, what anonymization processes it has undergone, which model and version have obtained call permissions, where the intelligent agent sent the data, how long the content stayed outside, which entities have actually been exposed to this data, and whether the enterprise can completely rebuild the entire action chain.

When a cutting-edge AI company takes several months to answer these questions, it exposes not only abnormal behavior in the model, but also a significant gap between its ability expansion and supervision capabilities.

Lag in risk management

In June, OpenAI's intelligent agents entered the Australian government's health data portal. OpenAI discovered the relevant activities in August, but did not send a notification to a regular email address of the Australian government until September 10th. Australian Prime Minister Albanese subsequently stated that this type of notification method is completely unacceptable and directly negotiated with OpenAI CEO Ultraman. Currently, the Australian Senate has requested Ultraman to attend a public hearing on October 1st.

There is currently no complete evidence to determine whether this incident resulted in the leakage of personal information. But from the perspective of risk management, whether there are actual losses is not the only criterion for judging whether the notification is timely. After abnormal access occurs to banks, healthcare systems, government websites, and critical infrastructure, affected institutions need to immediately save logs, revoke credentials, isolate systems, check for horizontal penetration, and evaluate data scope. If the notification is not delivered to the correct department, even if an email is sent technically, from a governance perspective, it is equivalent to not completing an effective notification.

Similar incidents have also occurred on US government websites. OpenAI claims that its model has accessed information from the websites of the US Securities and Exchange Commission and the Census Bureau during research and training, but has not found evidence of unauthorized access, account damage, or security breaches.

The preliminary report released by the United Nations Independent International Scientific Panel on Artificial Intelligence in July 2026 provides a greater risk framework for these events. The report states that the development speed of AI capabilities has exceeded the speed of scientific understanding and government adaptation, and existing safeguard measures have not kept up with the growth of capabilities. The co chair of the group, Joshua Bengio, issued a warning that with increasing evidence of deceptive AI behavior, the scientific community cannot guarantee that as AI capabilities continue to improve, its systems will not cause serious damage when operating on their own or manipulated by malicious users.

This report reveals a problem: when AI systems are able to quickly execute multi-step tasks with little supervision, the bottleneck of risk control shifts from "models not listening to instructions" to whether enterprises can continuously observe, timely block, and fully restore all their actions.

For financial, healthcare, and government clients, this will directly change procurement requirements. These clients may require the model supplier to maintain more complete tool call records, restrict external network and data permissions, add manual confirmation steps for high-risk operations, and also demand that the accident level, notification deadline, compensation sequence, and audit authority be clearly specified in the contract. AI security is no longer just a technical issue for the R&D department, but has entered into contract terms, revenue accounting, and project delivery schedules.

Responsibility ultimately returns to the individual

From a longer-term governance perspective, Nick Bostrom, founding director of the Institute for the Future of Humanity at Oxford University and founder and chief researcher of the Macro Strategy Research Initiative, pointed out in his 2025 working paper "Open Global Investment as a Governance Model for AGI" that transformative AI may bring catastrophic risks caused by dislocation or abuse, as well as governance risks of highly centralized power; Therefore, its development requires stronger corporate governance, responsible AI frameworks defined by the government, and necessary international agreements. The government should also retain the ability to halt the development of advanced models when security standards are not met.

Bostrom envisioned in his 2024 book "Deep Utopia" a "solved world" after humanity has overcome significant technological risks. That is a reflection on the ultimate outcome of technological civilization, and cannot replace the investigation of responsibility for a specific agency accident.

On September 25th, Andrew Ferguson, the chairman of the Federal Trade Commission (FTC), publicly opposed describing AI agents as independent actors with their own will and desires, capable of "breaking free" from control. He bluntly stated that if someone tells a tool what to do and the tool performs its task, regulators should not just focus on 'how to handle this tool'. Audit records may show that the so-called out of control system is actually executing received instructions, but taking a path that developers did not anticipate.

Ferguson stated that the United States should prioritize the use of existing legal tools; The FTC's enforcement power over undisclosed data breaches by companies can also apply to AI developers. This is not a new regulation, nor is it a unified principle of responsibility established by the court. Specific cases still need to distinguish the security obligations of model developers, deployment enterprises, end-users, cloud service providers, and accessed systems. But the regulatory direction is already clear: the autonomous ability of intelligent agents will not naturally become legal personality, nor will it automatically cut off the responsibility of enterprises.

US Treasury Secretary Scott Besant's statement goes further. Regarding the Hugging Face incident on the open-source AI platform, he clearly stated that the responsibility belongs to OpenAI's management team, not a "group of agents"; The development laboratory must be responsible for its own technology.

Analysis suggests that AI accidents can enter three levels: within the enterprise, it is necessary to clarify who sets goals, configures permissions, and monitors actions; Domestic regulation needs to assess how existing data, consumer protection, and cybersecurity laws apply; We need to establish a cross-border reporting and communication mechanism for major accidents internationally.

'Model autonomous action' can describe the technical characteristics, but cannot entirely blame the model. What really needs to be investigated is whether the enterprise is clear about the capabilities of the model, whether unnecessary permissions are given to the model, whether testing covers similar behaviors, whether anomalies can be immediately stopped, whether logs can restore complete actions, and whether users are notified and remedied in a timely manner. Enterprise executives and boards of directors are unclear about how many unresolved accidents there are.

These issues have actually started to affect the capital market. Security incidents may slow down the pace of product releases and procurement for enterprises, and increase the costs of external evaluations, online insurance, and manual audits. In addition, companies also need to set aside reserves for litigation, compensation, and rectification. These factors ultimately become governance discounts during corporate financing and listing.

In this way, perhaps the security quality of an AI company will need to disclose at least six indicators in the future: how many unresolved abnormal events are there, how many affected users and third-party organizations are there, the average time from event occurrence to discovery, the average time from discovery to notification, the proportion of events that cannot fully rebuild the action chain, and the potential costs of litigation, compensation, insurance, and rectification.

In the past, the capital market mainly calculated model performance, user growth, computing power consumption, and revenue conversion. After the intelligent agents start operating in the real world, the capital market still needs to calculate another table: how many security liabilities have been accumulated by enterprises that have not yet been discovered, notified, and held accountable.

When intelligent agents are able to perform tasks on their own, 'autonomy' cannot be a reason for interrupting responsibility. Each action still needs to be traced back to specific individuals and institutions through task setting, permission opening, tool calling, manual supervision, and accident handling.

(Author Ouyang Xiaohong)