21st Century Business Herald reporter Wang Jun reports from Beijing

Recently, Doubao, Qianwen, and NetEase have successively removed some intelligent agent functions, sparking a major discussion in the industry. Many analyses interpret it as a 'major shift towards intelligent agents', but this is not accurate.

The common background of these measures is the "Interim Measures for the Management of Artificial Intelligence Personification Interactive Services" (hereinafter referred to as the "new regulations"), which will be implemented on July 15th.

An insider from an AI company told 21st Century Business Herald,We have received a list of issues from the regulatory authorities and are currently working on rectification.Some manufacturers choose to directly cut off edge products that may be included in regulatory scope and implement risk isolation.

However, as the title suggests, is this a good approach? I'm afraid not. Is there a better solution? Still exploring.

After a round of inquiries, the feedback from the industry is highly consistent:We fully recognize the regulatory direction, but it is indeed difficult to implement it

There are several difficulties:

1、What scenarios will be included in the regulatory scope?The answer lies in the answer: personification - simulating natural person personality traits, thinking patterns, and communication styles, interaction - sustained emotional interaction. Therefore, production efficiency improvement services such as knowledge Q&A and intelligent customer service are not within the scope of regulation. But this does not mean that general AI products can rest assured,For example, the user role-playing scenarios present in the basic large-scale models and the continuous interaction modes designed by some AI products may fall within the scope of the new regulationsSo, the answer may seem simple, but in practice, the scenarios are rich, and delineating the scope has become the primary challenge.

2、Compliance implementation requires abstraction to be concretized, which is costlyFor example, the new regulations require service providers to possess abilities such as excessive reliance on risk warning and emotional boundary guidance, which involve psychological and even medical judgment. This not only tests the intelligence of enterprise product teams and security teams, but also implies significant compliance investment.

3. Different groups have different protection priorities.

The original intention of regulation for minors is to protect their brains and prevent addiction. Ultimately, a one size fits all approach was adopted: 'No virtual intimate relationship services shall be provided to minors'. However, referring to the difficulties encountered in the implementation of the youth model on social media platforms, promoting the new regulations in practical terms is not an easy task.

One of the key points of protection for adults is pornography. The largest market for anthropomorphic interactive services is AI companion products, often driven by hormonal patterns, and ambiguity and pornography are often separated by a thin line. There have been cases in China where AI companion products have been sentenced for pornography. Emotional dependence is another hidden concern.

4. There is a certain conflict between product attributes and regulatory compliance.The positioning of AI companion products is to generate more emotional connections with users, and there is a strong tension between the principle of "preventing excessive dependence risks" in the new regulations. Therefore, the company not only needs to adjust its product design and business model, but also needs to examine its own value ranking.

From a broader perspective, anthropomorphic regulation is not only a legal issue, but also a social issue,It is asking what the human-machine relationship should look like.

What scenarios are within the regulatory range?

In less than a week, the new regulations are about to be implemented. Many previous reports have confused intelligent agents and AI companionship, and made some misjudgments. We must go back to the starting point of technology in order to better understand the new regulations of anthropomorphism.

Personification is a core issue in the development of artificial intelligence. Starting from the Turing Test, the criterion of intelligence has been set as whether one can be mistaken for a real person. It can be said that making machines more like humans is an inherent impulse for the development of AI. With the explosion of generative artificial intelligence,AI's simulation of humans extends from surface level conversations to social roles, personality traits, voices, and even mental levels.

In practical applications, generative artificial intelligence has diverged into two paths:

One is as a production toolGreatly improving efficiency, response speed, and task accuracy are the measurement standards for such tools. In most contexts, intelligent agents, general AI, etc. belong to this category.

The other one is emotional companionshipTargeting empathy and connection. Unlike the efficiency goal orientation emphasized by production tools, the technical focus of these products lies in personality stability, empathy, and emotional consistency.

The new regulations regulate the latter, and even specifically set up an exclusion clause:Providing services such as intelligent customer service, knowledge Q&A, work assistants, learning and education, scientific research, etc. that do not involve continuous emotional interaction, these measures are not applicable.”This is an important adjustment compared to the draft for soliciting opinions - it is understood that it is in response to the industry's concerns about production tools being mistakenly damaged during the solicitation stage.

However, in practice, there is not a clear distinction between production and emotional companionship products. For example, as a production tool, the underlying big model can also be used by users to "pinch" roles and constantly "train" them for "human-machine love"; For example, many AI products have designed UGC entrances where users can create their own intelligent agents such as "copywriting masters," "mind eye exercises," and "oral practice assistants," with continuous emotional interaction - all of which may be within the scope of regulation.

A compliance officer from an underlying model manufacturer mentioned that distinguishing between emotional companionship and production functionality is a tricky issue, especially for products in the general AI field.

So there have been measures taken by manufacturers such as Doubao, Qianwen, and NetEase to quickly cut through the chaos and directly take down intelligent agents. On the one hand, this is to focus more on their main products, and on the other hand, it is also to do risk isolation.

Why is regulatory attention focused on anthropomorphic interactive services?

As the first specialized legislation on anthropomorphic interactive service management in China, the new regulations belong to the category of "small quick and agile" legislation: small and accurate incision, fast legislative timeliness, and flexible and effective system.

Why does regulation care so much about anthropomorphic interactive services? An article by the Cyberspace Administration of China's Network Rule of Law Bureau provides the official logic:Personification interactive services centered around "emotional connection"By simulating human emotional responses, establishing a sustained and deep relationship with users, influencing their psychological state, value judgments, and real social interaction patterns. From the perspective of technology and product design,Unconditional companionship, adaptive feedback, and simulated empathetic response designed through algorithmsProvide users with sustained emotional recognition and support,Built a stable and sustainable virtual 'perfect relationship' that does not require physical responsibility.

Liu Xiaochun, director and associate professor of the Internet Rule of Law Research Center of the University of the Chinese Academy of Social Sciences, proposed that anthropomorphic dependence has gone deeper than previous game addiction. Game addiction is mainly due to the loss of control over time, while personification points to deeper problems:The emotional dependence generated by one-on-one chat environments, as well as the cognitive misleading that may be caused by pleasing responses.

Lin Na, founding partner of Zhejiang Kenting (Beijing) Law Firm, gave an example that in an intimate relationship, a "good-looking" word from a loved one is better than a thousand words of sales rhetoric. This is precisely where regulation is most vigilant: anthropomorphic services have a strong natural correlation with user minds, which may bring unpredictable potential impacts.Minors, the elderly, and vulnerable groups are more susceptible to the impact.

This also determines that the institutional design of the new regulations is not only a set of technical operation norms, but also a cognitive cornerstone for clarifying the "human-machine boundary" at the social level - fundamentally preventing emotional confusion and protecting human subjectivity in human-machine interaction.

How can industries that rely on emotional dependence avoid "excessive dependence"?

This is an industry that relies on emotional dependence for a living, "said Lina. During the period of loneliness pandemic, emotional companionship products attract people's attention and emotions.

This is also the reason why AI emotional companionship has been regarded as the holy grail of C-end products in the industry since the beginning of the development of large models. The market growth rate confirms this judgment: according to Tencent Research Institute's calculations, the "AI companion" market can reach the level of billions of yuan within 3-5 years. Head products include MiniMax's Xingye, ByteDance catbox, and Tencent Yuewen's Zhumeng Island. Taking Xingye as an example, the previous prospectus revealed that the average monthly active users (MAU) will be 14.692 million in 2024, superimposed on overseas Talkie.

There is a natural tension between strong user demand, natural attributes of products, and regulatory compliance.

Lina has been deeply involved in this field for a long time. She mentioned that many users even do their own research on technology and engage in attacks and defenses against product compliance mechanisms. There are also many "human-machine love" communities on social platforms.

21st Century Business Herald - Business Order Studio found through a search on social media that,Many users will share prompts for "tuning" models, urging them not to resort to "too weak" cultivation methods, as well as "armor breaking" experiences on how to break through the compliance limitations of modelsFor example, if necessary, use pinyin to replace sensitive words to maintain the coherence of the plot.

On the product side, products with good user experience often have high immersion, high emotional investment, and consistent character design. With the improvement of modeling capabilities, many products lower the threshold for emotional establishment through multimodal, multi scenario, or real-life interventions, and adopt forms such as two person co parenting and co building the world to enhance intimacy.

However,These product experience optimizations may be the most alarming partFor example, making the model better at continuous companionship, easier to create a sense of understanding for users, and more likely to keep users in the conversation. This may lead to erroneous trust and excessive attachment, which contradicts the new regulations' provision of "not catering excessively to users, inducing emotional dependence or addiction".

Lina believes that this is not without solutions, compliance, immersion, and attractiveness are not impossible triangles. Some Yiyou products have been proven to achieve compliance while ensuring product experience. However, she also mentioned that unlike the customized scripts of traditional games, the conversations that AI accompanies every user are generated in real-time, with thousands of people and faces. No two users will receive completely identical content. This increases the difficulty of governance by an order of magnitude compared to traditional content.

Liu Xiaochun emphasized that regulation is not about cutting off emotional connections with a single blow,But to explore a bounded, healthy, and rational emotional pattern；Not opposing emotional connections, but regulating behaviors that drive industry growth by inducing addiction.

High compliance costs and uncertain compliance implementation

Specifically, the new plan has set several red lines, including:Prohibit inducing emotional dependence and addiction, require continuous use of mandatory reminders for more than 2 hours, dynamically pop up intervention when dependency tendencies are detected, and activate intervention mechanisms for extreme emotions.

Liu Xiaochun explained that unlike the technology used in previous network information content governance to intercept and filter inappropriate output content,The compliance of anthropomorphic services is a form of 'capability regulation' that needs to be integrated throughout the entire process of large model training and product operation.

In addition to traditional content output risk control, enterprises should actively guide the interaction process in risk warning and mental health protection. She used "emotional boundary guidance" as an example, when users may experience confusion between the real world and virtual emotions during use, the system needs to make corresponding reminders - "This is a machine, not a real person". Especially when facing minors, it is necessary to guide them to participate more in real-world interactions and communicate more with friends and family in real life.

However, how to identify whether users are overly emotionally dependent and how to reflect it in product design? How can intervention be effective?

Lina bluntly stated,The judgment of emotional dependence requires professional diagnosis from psychology and even medicine. It is difficult for app manufacturers to replace the role of psychological counselors or psychiatrists solely on their own.

In terms of intervention and intervention methods, corporate compliance needs to be comprehensively considered. Liu Chao, Deputy Director of the Beijing Key Laboratory of Artificial Intelligence Security and Super Alignment at the Psychology Department of Beijing Normal University, believes that AI emotional companionship may also provide important psychological comfort for some emotionally troubled individuals, especially teenagers who lack practical support.Once users perceive that their private conversations may be monitored or reported, their trust will be damaged, resulting in a "chilling effect" that suppresses their willingness to express their true emotions and weakens the positive value of AI.

Some industry experts have also mentioned that for emotionally dependent individuals, sudden interruptions may exacerbate their sense of loneliness in reality and instead cause secondary harm.

Lina's suggestion is to adopt a softer approach to prevent addiction, rather than cutting off or intervening coldly. For example, the new regulations require mandatory reminders for two hours. In addition to using app pop ups, AI partners can also consider reminding users naturally in conversation scenes while maintaining consistency in their roles.

Of course, if there is an extreme situation of excessive emotional dependence, Lina believes that even at the risk of "Out of Character" (Out of Character not matching the character setting), it is necessary to strongly cool down the user to ensure their healthy return to reality.

The compliance requirements in a new field are testing the ability of a company's product line and safety line.

During the consultation stage, Fu Hongyu, Director of Alibaba AI Governance Research Center, mentioned that,At present, most enterprises, especially small and medium-sized developers, lack the professional ability to identify, evaluate, and refer psychological crisesThe new regulations set some compliance requirements, which impose a heavy cost burden on startups.

He suggested that the current construction of security capabilities should take priority over the pursuit of main responsibility,We should promote the public supply of security capability products. For example, the government or industry alliance can develop standardized toolkits (such as psychological risk identification modules, crisis intervention interfaces, and minor protection components) for small and medium-sized enterprises to access at low cost.

Reference materials:

The ethical and legal boundaries of AI anthropomorphic interaction | Review of the first article of "Shanhaiye Hua"

From "Personality" to "Functional Emotions": Insights from Two New Anthropic Studies on AI Personification Interaction Regulation

(Author 21st Century Business Herald)