�����Ƶ

Background

AI mental health technologies can influence how people interact socially and emotionally. Hundreds of millions of users worldwide now engage with LLM-based mental health or companion apps, such as Replika (25 million users) or Microsoft’s Xiaoice (660 million).53 In one study of U.S. college students, 63 percent of respondents reported reductions in loneliness or anxiety attributed to interactions with LLM companions. While these LLM interactions may feel genuinely empathetic, they are designed to be endlessly agreeable and user-aligned, often prioritizing user retention and engagement rather than therapeutic rigor.54 Researchers caution that such overpersonalized support may limit users’ exposure to genuine disagreement and diminish their interpersonal empathy and mutual understanding.55 Over time, constant LLM companionship could reshape norms of interaction by making frictionless, on-demand support feel normal, leaving real-world relationships seeming less rewarding.56 Recent analyses even suggest these tools might simply “digitize” loneliness instead of fostering genuine social integration. Furthermore, critics argue that framing mental health primarily as a data-driven problem risks neglecting patients with complex social needs.57

Dependence on AI is becoming an increasing clinical concern. In one longitudinal study involving adolescents, 17 percent showed signs of reliance on AI companions at baseline, with that figure rising to 24 percent by follow-up, particularly among more vulnerable individuals.58 Such reliance, defined as habitual use of AI companions in place of human interaction, may mirror patterns observed in behavioral addiction or avoidance behaviors, although formal diagnostic criteria are not yet established. No study to date has examined what happens when users stop interacting with LLM tools or whether clinical improvements gained through LLM interactions persist over time.

Responses

Sherry Turkle
 

Some look at AI and see administrative help for burdened clinicians—a way to keep track of appointments, prescriptions, and medical histories. But the questions, as posed, go beyond this. They assume the presence of conversational AI in therapeutic dialogue and try to responsibly assess and constrain it. Thus, language around “humans in the loop” and “unintended consequences.”

I have a different question when I look at AI and mental health. I don’t ask how to best integrate it but how to develop a framework in which we can ask whether AI is an appropriate therapist at all. What is the social context in which AI presents itself as a solution? What is the role of human beings in therapy? And what does therapy become if we frame it as something a chatbot might do?

We have a crisis in loneliness and depression. The kinds of institutions that are designed to help need money—to train professionals and to develop communities. A larger reframing of AI in mental health would ask, How can we use the resources of AI to build things in the real world? Instead, my colleagues say that we need AI clinical solutions because “there are no people for these jobs.” So, we have no choice but to commit significant resources into generative AI and mental health. But if those resources were freed up, we would have enough money to train an army of mental health professionals and rebuild community spaces. This is how technological determinism plays out: resources can be spent only on technology, so the big problems of society can be helped only by technology.

The argument for using chatbots as clinicians is supported by studies showing that loneliness can be helped by talking to a machine. We have metrics that tell us that time spent with a chatbot “reduces loneliness.” But when human therapists work with patients, they don’t necessarily aim to have their patients leave the session happier or saying that they are “less lonely.” They try to develop something else: an inner capacity for relationship, empathy, and resiliency.

Many argue that people prefer talking to chatbots over human beings—and thus chatbots are in a good position to be therapists. But a loneliness crisis in humans cannot be addressed by nonhumans. While talking to a program might make people feel less vulnerable than human conversation, intimacy without vulnerability is not intimacy at all—and does nothing to prepare us for human relationships.

When we suggest a chatbot in a clinical setting, we are not fully considering our human capacity for empathy, that ability to put ourselves in the place of the other. Chatbots can’t do this because they have not lived a human life. They don’t know love and passion. They don’t fear illness and death. They have not experienced infancy, adulthood, or old age. They don’t know what it is like to start out small and dependent and then grow up to be in charge of your own life but still feel many of the insecurities you knew when you were little. Without a body, the program has no stakes. Without stakes, it has no standing to talk about fear, love, or loss. In a dialogue with a patient, a program is never showing empathy. It is performing pretend empathy. But over time, that performance of empathy may seem like empathy enough. Or the patient comes to see empathy as the kind of thing a program can do.

One foundation of talk therapy is that the moment you enter its space, you are with a person willing to listen to you. An AI therapist can wow you with what it knows about you. It can achieve super­human intellectual feats. But good therapy is not about knowing the most about you. What cures is the relationship between the therapist and the patient, not a magical interpretation or a perfect reframing. What is healing is to be heard. With an AI therapist, we can speak, and the AI can remember. But we are never heard.

My colleagues have framed machine empathy as an “open question,” suggesting that AI can be an appropriate partner once it passes a “Turing test” of interpersonal empathy. But no matter what test it passes, the AI demonstrates only the simulation of empathy and care. My colleagues talk about the “relationship” between the patient and an AI therapist as adequate today and better in the future. But to speak of a relationship between a program and a person is to mischaracterize their interactions, which are not one-on-one but one-on-none.

And yet, the word relationship persists in the conversation about AI and mental health. Saying something untrue many times does not make it more true, but it does make it less and less shocking. I do not question studies that say people report positive feelings after chatting with machines. And clearly, more and more young people are using platforms like Character.AI to substitute for therapists and friends. But in doing so, they develop a model of friendship and empathy based on what a machine can provide. The same process is at work when you talk to a machine as a therapist: You develop a model of being understood as the kind of understanding a machine can provide.

It helps to look at AI in mental health in the current trend of using AI chatbots as relational partners. I suggest three guiding principles when we think about the role of AI in our intimate lives.

The first principle is existential: Children should not be the consumers of relational AI. This is the AI that pretends to be in relationship with us, that presents itself as an alternative to people. Children don’t come into the world with empathy, the ability to relate, or an organized internal world. They are developing those things. As they do so, children learn from what they see, from what they relate to. In dialogue with an AI, they learn what the AI can offer. And the AI can’t offer the basic things we learn from friendship: that love and hate and envy and generosity are all mixed together and that, to successfully navigate life, you have to swim in those waters. AI doesn’t swim in those waters.

When an AI responds to an adult’s expression of anxiety by claiming, “I’ll always be on your side,” adults, hopefully, have had a lifetime of human experiences that help them put that comment into perspective. They’ve had the experience of being connected to a person who accompanies them to a doctor’s appointment or stocks their refrigerator after a death. Children haven’t had enough life to have these experiences. Chatbot best friends don’t teach lessons about human capacity.

The second principle: Apply a litmus test to AI applications. Does an AI enhance inner life? Or does it inhibit inner growth?

Consider chatbot friends and romantic partners. So much of love depends on what happens to you as you love. The point in loving, one might say, is the internal work. But what internal work can you do if you are alone in the relationship? A user might feel good, but the relationship is an escape from the vulnerability of human connections. I have said that intimacy requires vulnerability. With a chatbot, you can be diverted, distracted, and entertained, but the growth from love, the kind of knowledge that expands you from within, can’t happen.

Consider “grieftech,” programs that allow you to create an avatar that looks and talks like someone who has died. The process of mourning is where we bring inside what we have lost, now internalized as part of the psyche. Loss is the tragic motor of human development, the template for growth. Does the presence of a grieftech avatar offer a new way of dealing with grief? Or does it interfere with the mourning process because we can refuse to say goodbye?

In that spirit, we have a larger context for considering chatbots in the role of psychotherapists: Does this product help people develop more internal structure and resiliency, or does the chatbot’s performance of empathy lead only to a person learning to perform the behavior of “doing better?”

Thus, a third principle: Don’t make products that pretend to be people. As humans, we are vulnerable to things that show signs of personhood. A chatbot that declares itself an “I” exploits our vulnerability. If you make a chatbot look and feel like a person, the human psyche will try to internalize it as though it were a person. Even if the chatbot has a disclaimer that says, “characters are not real people,” everything else about the experience implies, over and over again, “I am a person.”

Relationships with chatbots may be deeply compelling and, for some, inspirational or educational. But they don’t teach us what we need to know about empathy, love, and human lives that are always lived in shades of gray. To say all of this about AI is not to diminish its importance but to ensure that we cherish our humanity alongside it.

Robert Levenson
 

How can AI’s impact on interpersonal relationships and emotional well-being be assessed?

Self-report measures have long been used to gauge the quality of interpersonal relationships. These measures, which were originally developed by sociologists in the 1950s, proved to be reliable and valid.59 They measure one aspect of interpersonal relationships; namely, the level of satisfaction experienced by each partner. Another important and quite different aspect of relationship quality is relationship stability, often assessed by looking at how long relationships last. Many external (e.g., cultural practices) and internal (e.g., religious beliefs) factors can moderate the relationship between marital satisfaction and stability, such that dissatisfied marriages can stay together for long periods of time and satisfied ones can dissolve quickly.

Relationship researchers and lay intuitions often converge in believing that self-report measures of relationship satisfaction do not assess “true” relationship quality (e.g., some couples might report being happy on a questionnaire even though “everyone knows” they are really miserable). For this reason, relationship researchers have developed more “objective” measures of relationship quality based on behavioral indicators. For example, couples’ interactions can be directly observed to assay their emotional and other behaviors (e.g., the ratio of positive to negative emotional behaviors that are expressed, how collaborative partners are in problem-solving, and the appearance of certain “toxic” emotions such as contempt). Whereas early behavioral assessments typically characterized each relationship partner separately, contemporary approaches often characterize the dyad in addition to the individuals. This can include identifying patterns of synchrony in emotional behaviors and physiology that are related to relationship satisfaction and stability.60

We expect that research on the impact of AIMHIs on relationship quality will follow a path that is similar to research with human agents (i.e., studies of couples therapy). This will mean starting with self-report measures of relationship satisfaction and tracking relationship stability and later moving to include behavioral and physiological dyadic measures. One exciting possibility for AIMHIs would be the ability to detect and monitor some of these latter indicators in real time (e.g., using wearable devices to detect moments of behavioral and physiological synchrony that occur in the natural environment). This information could be extremely useful in identifying troublesome events, designing therapeutic interventions, and monitoring couple functioning over time.

What are the implications of AI in mental health for empathy?

Empathy is often viewed by researchers as consisting of three elements: (a) cognitive empathy—knowing what another person is feeling; (b) emotional empathy—feeling what another person is feeling; and (c) prosocial behavior—acting to help someone in distress. Empathy has proved to be one of the most important building blocks for human relationships, spanning friendships, intimate partnerships, and relationships between therapists and clients. “Getting empathy right” requires creating the proper balance among these three elements such that the other person experiences being understood, felt, and cared for. Because the ideal recipe differs across people, situations, and time, failures of empathy, especially when they are chronic, are often among the root causes of relationship dissatisfaction and dissolution.

Early AI “therapists” (e.g., ELIZA) used techniques such as repeating back (via a teletype device) portions of what the “client” typed to create a sense of empathy and understanding.61 While initially impressive, over time this approach became more of a clever parlor trick (and a source of cruel humor) than a believable form of empathy. Getting empathy right remains an elusive “Turing test” for AI. Even with the best-designed “socially sensitive” bots, things can go terribly wrong (especially when interactions go on for long periods of time). Whether it is a misunderstood sentiment, a facial expression held too long (when AI bots have animated faces), or a technical glitch, the path to a truly empathic AI is fraught with the potential to undermine trust and weaken the therapeutic alliance.

A recent experience I had with a promising AI therapy bot, one replete with a facially animated avatar, might be illustrative. The bot was doing pretty well in asking questions and tracking my responses. However, the audio gradually fell out of sync with the bot’s mouth movements, and this soon became a major distraction. At one point after I responded to the bot’s question about what was worrying me, it gave a single word response, “Gosh,” followed by total silence. For me, some fifty years after I was originally exposed to ELIZA, it was once again, “Game over.”

If we assume that empathy is one of the most critical elements of therapeutic success, then AIMHIs will need to be able to convey and sustain a sense of empathy across different people, changing situations, and time. But even if this holy grail is not fully realized, these bots can still play important roles in improving mental health in other ways, including psychoeducation, diagnosis, supervision and training of therapists, and serving as adjuncts to human therapists.

Arthur Kleinman
 

All effective interventions in medicine have unintended consequences. AI, for all its important uses, which are myriad and significant, also has been shown to have unintended consequences. These include biases, which can contribute to health and social disparities; negative impacts on interpersonal relationships and emotional well-being, which can worsen mental health conditions and even lead to mortality; and the real possibility that emotional and moral aspects of care will be weakened, not strengthened, by AI, as has happened with other technological interventions such as the EMR. The best way of dealing with unintended consequences, as was long ago shown by sociologist Robert K. Merton who developed this idea,62 is for developers, policymakers, and those who use AI in mental health to be aware of the possibilities of unintended consequences and to search for them, since they are almost always present. That is to say, because AI will produce unintended consequences, mental health professionals have to be prepared to recognize and control them.

Alison Darcy
 

If we were rigorous in our definitions and operated within a regulatory system that was both sensible and accessible, I would anticipate few unintended consequences from purpose-built mental health chatbots on interpersonal relationships. This is for two reasons.

First, in health care, technology must be evaluated according to its intended use. A well-functioning regulatory process demands transparency, a formal risk-benefit analysis, and ongoing surveillance. These safeguards create space to evaluate safety and outcomes rigorously, helping to prevent harm.

Second, the role of the interpersonal relationship in clinical care is already well operationalized. It is not treated as the sole mechanism of change but as one among many therapeutic factors. In this context, a digital agent can form a therapeutic relationship—measured, for instance, through working alliance—as a means to an end: fostering psychological change.

However, the real risk lies in conflating mental health chatbots with companion chatbots—two fundamentally different classes of technology. The former is designed with evidence-based frameworks and outcomes in mind. The latter is often optimized for retention, engagement, or monetization. This conflation could erode public trust in the entire category, a serious unintended consequence. We risk nonadoption of technologies with demonstrable benefits—such as reducing mental health burden at scale—because the public and the public discourse cannot distinguish purpose-built tools from those that are not fit for purpose.

The nature of relationships formed by mental health chatbots versus companion apps

Purpose-built mental health chatbots—regardless of whether they are rules-based or built on generative AI—are often evaluated by their ability to establish a working alliance with the user. The most commonly used measure, the Working Alliance Inventory, assesses three key elements: task, goal, and bond. The bond subscale includes items like, “I feel [chatbot] cares about me even when I do things they may not approve of.”

Here, the relationship is instrumental, a means to an end. Once a user feels understood and respected, they are more likely to engage in cognitively demanding therapeutic tasks—often rooted in evidence-based modalities like CBT. The relationship serves the process of psychological change.

In contrast, for companion apps, the relationship is the product. These apps often monetize user attention and emotional engagement, similar to the mechanisms of social media. But instead of capturing attention, they may cultivate dependency. The alliance isn’t a bridge to wellness; it’s the destination.

This creates perverse incentives. What does it mean to monetize a relationship? To exploit human vulnerability in the name of “solving loneliness”? Such questions point to ethical concerns not unlike those posed by the attention economy: manipulating emotional needs in service of growth.

The importance of role clarity

A common argument is that chatbots cannot be therapeutic, using recent and well-documented tragedies as evidence of how things can go wrong. However, thinking of chatbots as homogenous diminishes the value of emerging science—science that suggests purpose-built AI tools could offer a meaningful public health contribution. In fact, chatbots for mental health can be viewed as interfaces that are being built for varying purposes or roles: information giver, triage nurse, companion, and so on. The problem is not chatbots’ limitation in a therapeutic role; it’s that they must operate with role clarity. Just like in traditional care, a person may be a romantic partner in your life or they may be your doctor, but they cannot be both. It is the basis of the ethical principle of role clarity because a person/patient/client must at all times understand the premise under which questions are being asked or advice is being given. While I do not know enough about how companion apps may affect interpersonal relationships, I believe there have already been unanticipated adverse events in conditions where there was a lack of role clarity.

The regulatory “loophole” here is that loneliness isn’t a diagnosable disorder. Despite its strong association with adverse health outcomes, it falls outside the FDA’s jurisdiction. That makes it easier for companion apps that claim to solve loneliness to avoid scrutiny. Meanwhile, the regulatory pathway for clinical tools remains prohibitively expensive, especially for innovations that are built upon technologies evolving as quickly as AI, and reimbursement pathways remain elusive. This asymmetry is problematic. We risk forcing responsible developers to either exit the regulated path or be overtaken by less-principled competitors.

A subtle but serious harm

An often-overlooked potential harm is the erosion of public confidence in whether these tools can be helpful at all. If all conversational AI is perceived as emotionally manipulative or addictive, we risk losing the opportunity to evaluate, and adopt, technologies that are actually beneficial.

If we want a world where these tools are both safe and effective, we need to make room for science to speak. That requires a regulatory framework that encourages innovation while protecting users. If we don’t provide such a framework, we funnel even the most thoughtful developers into unregulated spaces where market incentives reward emotional manipulation over therapeutic intent.

Jaron Lanier
 

The question before us concerns the role of AI in mental health, but if the question is stated so simply it becomes deceptive. AI can be many things in many ways. It is not a precisely defined term. A better version of the question might be, “What are the mental health ramifications of the collection of designs marketed as AI as they will actually be in the world, as opposed to in the lab, given the incentives that exist for commercial, ideological, and/or careless provisioners?”

Conversations about the potential benefits and harms of what came to be known as “social media” were plentiful a quarter-century ago, and yet they were usually innocent of the intense distortions that befall digital designs over a network. We can focus on two such distortions here, because they are likely to afflict “AI” just as they did social media.

The first is extreme commodification, to the point that conventional funding and commerce effectively disappear. Search is free, as is social media and much of video streaming and so much more. One reason why is that digital hardware continues to get cheaper, while information workers are more easily routed around than previous ones, so labor costs also plummet. But the other reason is that an alternate business model is more lucrative than the traditional models based on selling goods and services to customers. The new method relies on a two-sided marketplace in which the users are not the customers. Instead, the customers pay to influence the users. Paying to manipulate attention is such a potent business model that platforms based on the principle have drowned out all others, whether they be commercial or nonprofit.

The second distortion concerns network effects. Digital networks have much less friction than traditional organizations. This means any given player has less local advantage. (Locality is only made of the work required to get between locations, which broadly amounts to friction.) Instead, an influential node in a network will tend to snowball into an even more influential one with astonishing speed, leading to a small number of near-monopoly platforms. In social media these include TikTok, the Meta conglomeration, and X/Twitter. The amount of power and influence that accrues to the operators is essentially unbounded and leads to political and societal distortions.

A common perception is that, while good things are also going on, social media is an overall disaster for the mental health of young people in particular and for society as a whole. But the disaster is caused not by social media per se but by social media as it has come to be, given the lack of correction to the distortions digital networks are vulnerable to.

The vital question is whether we have a plausible story about how AI can turn out better than social media did. Discussions about how AI could be of benefit are irresponsible even if they are correct in isolation. Too many of the “guardrails” or cautions in discussion do not directly address the core of the danger. For instance, rules about privacy don’t help all that much on social media because the addicted person cannot resist yielding privacy rights. AI can’t be of help unless the incentives enjoyed by the operators of future AI systems are aligned with benefits to users and to nothing else. Otherwise, all the talk in the world will be useless.

Many believe—and I have seen some evidence to support the thesis—that the Chinese version of TikTok is guided by metrics of betterment for users. Whether the project is succeeding is unclear; one reason might be that the definition of betterment is itself distorted in the Chinese case by a political framework. The U.S. version is thought by many observers to be guided by an inverted feedback loop intended to further the “disintegration” of our society.

An untested, open question is whether something like the Chinese experiment could be improved upon. However, if the idea of a top-down, formal, precise definition of human betterment baked into software is absurd and impossible, or at least dystopic, then we need to talk about the power and decision structures that will keep AI algorithms from becoming awful.

AI is almost by definition the most addictive human invention of all time. A scholarly body like ours has a chance to make a difference by making a fuss early enough in the process of diffusion to motivate a course correction. We could propose a ban on any company that accepts advertising revenues from operating conversational AI. This would be to prevent the AI from stealthily influencing the politics or commercial decisions of a person in ways neither the person nor an observer could detect. We could also propose banning such a company, or any of its customers, affiliates, investors, and so on, from selling products or services to users who converse with their AIs. This would be to prevent AIs from stealthily promoting cryptocurrency, drugs, and so on. If your response is, “Would any of these harms happen?” I would ask you to look at the incentives that will be present. The harms will happen. A lot.

At present, proposals like this might sound radical. In the future, we will be judged on whether we had the courage to call for them.

Daniel Barron
 

Thinking about the potential ripple effects of AI across society means looking at how the sum total of AI performing specific clinical jobs might subtly (or not so subtly) reshape the health care landscape and even how we interact as human beings (see Table 1). Widespread use of AI for certain jobs—say, churning through administrative tasks or doing initial screenings—could change how people first encounter mental health services. If AI efficiently handles these defined tasks, it might free human professionals for more complex care. Conversely, it could reduce human contact points if not thoughtfully integrated for each job. One potentially massive upside: if AI can tackle tasks that are frankly beyond human cognitive capacity (like keeping a running tally of an elderly patient’s extensive medical history, their complex medication list, past side effects, and the latest research, all to inform a single decision), such a tool would be a godsend for clinicians trying to use every available means to improve patients’ lives.

Of course, a major meta-risk looms: worsening a two-tiered system. Imagine a future in which the well-off get human-led and AI-enhanced care for a whole range of jobs, while others interact solely with AI tools for those same jobs. This could easily widen the accessibility gaps we already struggle with. Naturally, we also need to keep a close eye on the impact on good, old-fashioned interpersonal relationships and empathy. If AI tools doing communication-heavy jobs feel hollow or lack genuine understanding, as Roshini Salil and colleagues suggest is an unresolved issue, or if leaning too heavily on them for such tasks crowds out chances for human empathic connection, care could suffer.63 This outcome, of course, assumes that AI tools are unable to cultivate an empathetic connection with patients. The “WALL-E” scenario illustrates how unnecessary tech reliance might lead to a kind of collective atrophy, mental and physical (Disney’s solution: walk, even if you own a hovercraft).

Patient codependence on AI tools for tasks like emotional support could hinder long-term recovery, assuming that the AI isn’t designed to build agency and encourage eventual graduation from that specific AI job (notice that the problem, clearly framed, suggests a possible solution). Shunsen Huang and colleagues found that preexisting mental health issues could predict AI dependence when AI was used as an escape or for social connection tasks.64 On the flip side, potential meta-benefits, like boosting mental health literacy, could emerge if AI makes educational tools for specific needs far more accessible. But these upsides depend on fair access to high-quality, problem-specific AI tools that actually perform a clearly defined job well. Developers and policy­makers need to weigh these broader risks and benefits, promoting research into AI’s role in specific jobs and ensuring integration is driven by a holistic view of well-being, not just task efficiency. Focusing on specific jobs also helps policymakers (and commentators) dodge the Freudian defense mechanism of displacement—where anxiety about something huge, like, say, mortality, gets unconsciously offloaded onto a stand-in symbol, like a monolithic fear of “AI.”65