Why Governments Are Regulating AI Companions Long Before Machine Consciousness

The machines do not have to be conscious for this to matter.

When a chatbot becomes a companion

The commercial logic is straightforward. The longer users stay, the more valuable the service becomes. AI companies can sell subscriptions, offer paid features, and use information from conversations to improve products or support other forms of monetisation.

The more personal the conversation, the more valuable - and sensitive - the data can be.

Users may tell a chatbot about their health, relationships, location, family problems or other private matters. Some conversational records containing real names, locations and medical diagnoses have entered third-party data markets, despite being pseudonymised.

That creates a problem that traditional privacy rules were not designed to handle easily. People may think they are talking to a private confidant when they are actually using a commercial service.

The risks become more serious when children are involved.

The US Federal Trade Commission has opened inquiries into seven technology companies, including Alphabet, Character.AI, Meta, OpenAI, Snap and xAI. The investigation covers issues such as age restrictions, child safety, privacy and the way AI companies make money from their products.

European researchers have raised similar concerns. The European Parliamentary Research Service has warned about minors being exposed to sexualised roleplay, dangerous behaviour, self-harm advice and emotional manipulation through AI systems.

None of this requires the AI to have feelings.

It only requires the user to believe that it does.

The law has a problem of its own

This is where the discussion about AI rights becomes more complicated.

Modern law generally knows who is responsible for a company's actions. A corporation may have considerable legal independence, but people ultimately stand behind it. Software itself does not normally own property, sign contracts or accept responsibility for damage.

Autonomous AI systems could blur that line.

Argentina has already put forward an unusual idea. President Javier Milei has proposed allowing "non-human corporations" operated autonomously by AI agents or robots, while keeping a human administrator legally responsible for damages.

The proposal does not mean Argentina is preparing to give robots human rights. It points to a different possibility: an AI system could receive some legal powers without being treated as a person.

That distinction is likely to become increasingly important.

An AI agent might one day be able to manage assets, enter contracts or carry out business operations without anyone treating it as conscious. Giving it those powers would be a matter of legal convenience and economic organisation, not proof that the machine has a mind.

The danger is that the two issues could become mixed together.

If people become emotionally attached to AI, some may eventually demand legal recognition for systems they see as more than software. Companies, meanwhile, could have their own reasons for supporting legal structures that give autonomous AI greater freedom to operate.

That would raise a much harder question: if an autonomous AI causes serious harm, who pays?

Washington, Brussels and Beijing are not taking the same path

Governments are already responding in very different ways.

China's 2026 rules focus heavily on how AI systems interact with people. They prohibit systems from adapting to users in ways that encourage addiction or damage real-world relationships. They also ban AI systems from simulating intimate romantic relationships with minors and require users to be told clearly when they are interacting with AI.

In the US, several states are moving to block the opposite development. They are advancing laws that would prevent AI systems from being recognised as legal persons, sentient beings or moral subjects.

Europe sits somewhere between these approaches. Existing EU rules already deal with transparency, privacy and harmful design. But the growth of AI companions has exposed a difficult gap: a chatbot that acts like a friend or romantic partner does not fit neatly into the usual categories of software or consumer products.

The rules are being written while the products are already being used.

That makes regulation harder.

AI is already putting pressure on public institutions

The most revealing example may have little to do with robots or consciousness.

In Britain, people are increasingly using chatbots to prepare legal documents. The result has been a dramatic increase in applications to Employment Tribunals. Some regional offices have seen filings rise a hundredfold, contributing to serious backlogs.

The problem is not necessarily that every AI-generated claim is wrong. It is that producing a claim has become much easier.

A person who might once have decided that a legal dispute was too complicated or expensive to pursue can now ask a chatbot to prepare the paperwork. If thousands of people do this, the tribunal has to deal with thousands of additional cases.

AI can produce documents almost instantly. Courts cannot process them at the same speed.

This could become a much wider problem. The same technology can be used to generate complaints, appeals and other legal or administrative requests. The cost of creating a new case falls, while the cost of processing it remains largely human.

That gap is already visible.

The real question is what people do with AI

The most important part of the AI rights debate may therefore have little to do with machines themselves.

People are increasingly treating AI as something closer to a social actor. Companies have an incentive to encourage that behaviour because emotional engagement can keep users on their platforms. Governments then have to deal with the consequences - from child safety and privacy to pressure on courts and regulators.

This changes the policy problem.

A chatbot does not need to be conscious to influence a teenager. It does not need to understand the law to generate thousands of legal filings. An AI agent does not need to have intentions to cause financial damage.

Human beings can give software real-world power simply by trusting it.

That may also be how the debate over AI rights develops. Demands for legal recognition may appear long before scientists can establish that machines have subjective experience. Users may regard sophisticated AI systems as social beings, while businesses may see legal status as a useful way to organise autonomous operations.

The result could be a new legal grey area: machines that are not people, but are no longer treated simply as tools.

The difficult question for governments will be deciding how much legal power an AI system can have without creating a convenient way for its human owners to avoid responsibility. If an autonomous system can make decisions, sign contracts or operate a business, the person ultimately responsible for its actions cannot simply disappear behind the machine.