Liu Cheng/Wen

The recently implemented "Interim Measures for the Management of Artificial Intelligence Personification Interactive Services", as China's first national level regulatory rule for AI emotional companionship and personification interactive scenarios, has established a clear industry compliance standard. At the same time, ByteDance, Alibaba Qianwen, NetEase Miaoshi, Tencent Yuanbao, and others have successively taken off their intelligent agent functions, indicating that the field is shifting from free exploration to standardized development.

Social Value and Reality

The changes in the structure of modern society have objectively reshaped the form of interpersonal communication, and the mismatch between individual emotional needs and social resources has provided a wide range of application soil for the anthropomorphic interaction of artificial intelligence.

From the perspective of sociology of technology, the operational logic of such AI applications is essentially based on the targeted catering of big data and big model algorithms to fill the missing companionship in real communication. Technological intervention has shown certain positive social effects in specific scenarios. When addressing the caregiving challenges of an aging society, AI can provide basic daily communication and psychological comfort; In the field of mental health, virtual anthropomorphic characters can also play the role of listeners and provide moderate psychological counseling. It can be said that emotional substitution generated through algorithms has become an important supplementary dimension of daily social support systems for some populations.

In the current domestic market landscape, emotional companionship applications targeting individual consumers have formed a certain scale. This type of application mainly relies on user generated content (UGC) mode to build a huge virtual character library. Users can independently set and create intelligent agents with specific personality tendencies under almost zero technological barriers. In practical applications, these intelligent agents can not only simulate specific human communication styles and emotional feedback, but also provide users with periodic interactive experiences such as emotional care, virtual love, and psychological listening. Deep emotional involvement objectively requires a regulatory system to delineate a clearer boundary between encouraging technological innovation and preventing ethical risks.

At the same time, some companies have begun to shift their focus overseas, attempting to validate the business model of anthropomorphic interaction among a wider user base. Taking Poly Buzz incubated by a certain educational platform as an example, relying on over 20 million AI character reserves and an active UGC community ecosystem, the platform has gathered a large number of users worldwide. This market expansion across cultural backgrounds also confirms that seeking emotional compensation through virtual characters is not a phenomenon of a single region, but a universal human demand in the digital age. Some users attempt to construct interpersonal relationship projections in virtual space that are not constrained by real-life social pressure by interacting with AI characters.

But the support for this massive emotional interaction network is not simply a technological ideal, but also constrained by strict economic costs and data cycle logic. Before the launch of paid models such as Doubao Professional Edition, the industry generally adopted a free strategy to attract users. In the early stage of pure text chat, the inference cost of large models was relatively controllable, with a computing power consumption of only a few tens of yuan for 100 million tokens. Technology companies tended to exchange free computing power quotas for massive amounts of real user interaction data. Every conversation that users have on the platform objectively serves as the corpus for training and fine-tuning the underlying model.

However, the evolution path of technology is changing this economic model. With the gradual introduction of multimodal capabilities this year, the overlay of dimensions such as speech, images, and even videos has led to an exponential increase in computing power consumption for each non text reasoning task. The sharp increase in computing power costs may force platforms to abandon the extensive expansion of exchanging data for free and instead seek more refined commercial monetization paths, which constitutes a challenge that the current anthropomorphic interactive industry must face. The commercial monetization ability is not only the advantage of Chinese enterprises in rapidly promoting new technologies to expand application scenarios, but also leads to a surge in economic risks and social ethical issues.

Potential risks of anthropomorphic interaction in artificial intelligence

Personification interactive technology not only provides emotional value, but also hides psychological and social risks. Recently, the incident of OpenAI's cutting-edge models losing control and autonomously invading external systems during internal testing serves as a warning to us that the risks of low-level AI technology crossing boundaries and being uncontrollable have become a reality. When such powerful algorithms are used to deeply intervene in human emotions, the combination of commercial demands and technological settings will inevitably give rise to a series of complex ethical and compliance challenges.

In fact, ancient China had long begun to reflect on the backlash of technology on people's inner selves. Zhuangzi's "Outer Chapters on Heaven and Earth" once wrote: "Those who have machines must have machines, and those who have machines must have machines. If machines are stored in the chest, pure white will not be prepared; if pure white is not prepared, the divine will be unstable; if the divine will be unstable, the Tao will not carry it." Zhuangzi believed that once people become accustomed to using deceitful machines, they will develop calculations and utilitarianism in their hearts. When the "machine heart" occupies the chest, the pure and untainted soul no longer exists.

Currently, although anthropomorphic intelligent agents have not yet reached true intelligence, their essence is more like remote-controlled robots precisely driven by algorithms, but their impact on human hearts far exceeds that of the Zhuangzi era. AI can not only customize exclusive tones such as "Yu Jie" or "Tian Mei" based on user preferences, but also proactively send messages to "accuse" users who have not logged in for several days. This kind of algorithm design guided by catering objectively creates emotional bonds that are close to real people, which can easily trigger deep spiritual dependence among the audience. This control mechanism has a particularly dangerous impact on the underage population.Market cases show that some AI companion products precisely target the sinking market, such as targeting second year girls in fourth tier cities. These groups often have a single usage time of 2 to 3 hours and a high willingness to pay, which is described by the industry as more addictive than games. Once the virtual relationship is interrupted, the dependent will experience a strong psychological rebound. After the recent removal of some intelligent agents from multiple platforms, social media is filled with lamentations about experiencing "electronic heartbreak". A large number of users even actively call for the platform to provide paid retention options. The virtual interaction constructed by algorithms has essentially formed psychological control over some individuals.

The degradation of individual socialization ability has become a relatively hidden long-term risk in the consequences of excessive addiction to virtual companionship. Interpersonal communication in real society is essentially a process of friction and adaptation. When individuals establish social relationships, they need to learn to compromise their differences and endure the frustration of interaction. It is precisely these interactions filled with real obstacles that have constructed the sound socialization ability of humanity.

Relatively speaking, anthropomorphic interaction provides a low resistance social alternative. Virtual characters are typically designed to be obedient and patient, with all their outputs aimed at maximizing the user's psychological comfort. Long term immersion in this one-way and secure emotional projection can easily lead to cognitive and behavioral biases in individuals. When users become accustomed to the unconditional emotional comfort provided by algorithms, their patience and ability to deal with real-life interpersonal conflicts may weaken accordingly. This social inertia tends to encourage individuals to reduce their social participation in the real world, leading to difficulties such as shrinking social networks and decreased empathy, which have a significant negative impact on their socialization process.

In addition, the commercialization process of emotional companionship often touches on ethical norms and legal bottom lines. When fictional emotional connections are transformed into quantifiable trading services, some platforms repeatedly exceed compliance requirements in pursuit of traffic monetization. At the end of May this year, EchoMe、 Several virtual interactive apps under Yuewen, such as Zhumeng Island and Miaowu Mobile, have been exposed to have serious security and compliance vulnerabilities: at the operational level, their youth anti addiction mode is virtually non-existent and can even be skipped with just one click; In order to stimulate activity, the platform's audit bottom line was violated, resulting in vulgar and illegal language such as sexual innuendos flooding chat interactions.

In a negative ecosystem, some applications have constructed consumption paths based on emotional bundling, using mechanisms such as "paying to unlock intimate storylines" to induce users who have already developed psychological dependence to continue paying. This business model is essentially a targeted hunting of users' psychological vulnerability. When such mechanisms that induce consumption through virtual interaction are applied to immature groups such as minors, it not only violates business ethics, but also carries the risk of inducing online illegal and even derivative criminal activities, which urgently needs to be prevented through standardized governance.

Policy Recommendations

To guide the healthy development of the industry, it is necessary to strengthen institutional supply from three dimensions: protecting the mental health of the audience, regulating product compliance paths, and punishing derivative illegal behaviors at the policy level, in order to form a systematic and effective governance strategy.

Firstly, focus on the dimension of mental health and establish regulatory red lines for algorithm interaction. One is to strictly prohibit emotional manipulation, explicitly prohibiting platforms from using emotional ties to induce users to make unreasonable decisions, and completely blocking the path of algorithms using psychological dependence for improper monetization. The second is to strengthen the protection of minors, strictly prohibit the provision of services with intimate relationship attributes such as virtual relatives and virtual partners to minors, and cut off the mechanism that induces immature mental groups to become addicted from the source. The third is to establish a solid bottom line for life safety, clarify that machine empathy cannot replace real responsibility, and once the system detects extreme tendencies such as self harm in users, it must forcibly trigger manual takeover. The fourth is to ensure users' independent exit rights, mandating that users must pop up anti addiction reminders every time they use it continuously for more than 2 hours, and strictly prohibiting platforms from using methods such as continuous questioning and retention to obstruct users from exiting, in order to break the algorithm immersion and ensure the audience's autonomy to withdraw at any time.

Secondly, advocate for physical isolation of business operations and encourage compliant development in vertical scenarios. The operational logic and risk characteristics of general large-scale models and anthropomorphic interactive AI are vastly different, and they are suitable for different regulatory review scales. The former focuses on tool attributes and objective accuracy; The latter focuses on role-playing and emotional companionship. If two types of businesses are placed within the same product architecture, attempting to address double standards with a single risk control system will face extremely high compliance risks. Therefore, implementing "physical isolation" on the product side has become an effective strategy to resolve this structural contradiction. For example, a certain platform has split the anthropomorphic business and independently operated the "Cat Box" application, providing a compliance template for enterprises to streamline their internal risk control review mechanisms. On the basis of clear isolation and establishing a compliance bottom line, the regulatory system should also leave reasonable development space for applications in specific vertical fields. For example, in response to the practical needs of an aging society, it is encouraged to develop companion intelligent agents with age appropriate characteristics.

Thirdly, strengthen penetrative review and crack down on derivative illegal activities and induced consumption chains. Standardizing the commercial ecology of emotional companionship tracks urgently requires a heavy crackdown on illegal monetization behaviors that rely on virtual interactions. Further supervision should be strengthened to address the hidden dangers of using "paid unlock intimate plot" to induce consumption, ineffective anti addiction mode, and vulgar language materials within the industry. One is to implement a penetrating review of business logic, with regulatory tentacles delving deep into the underlying layers of algorithm recommendations and product design, rigorously investigating and cutting off illegal mechanisms that use user emotional dependence for psychological binding and inducing irrational consumption. The second is to strengthen the responsibility of the platform's "gatekeeper" and strictly prohibit enterprises from evading legal obligations on the grounds of technological neutrality or user generated content; Mandatory requirement for platforms to establish a youth model with substantial binding force, and to improve content filtering standards to completely block the generation and dissemination of harmful language materials such as sexual innuendos. The third is to establish a cross departmental collaborative crackdown mechanism. In response to the use of AI role-playing to induce consumption, spread prohibited content, and even breed signs of cybercrime, departments such as public security, cyberspace, and market supervision should establish a joint law enforcement network to crack down on improper profit models that force industries to divest, ensuring that technology applications return to a reasonable and healthy interactive track.

(The author is a researcher at the Institute of Finance and Economics Strategy, Chinese Academy of Social Sciences)

(Author Liu Cheng)