Understanding the Ethical Challenges of AI as a Mental Health Support
A recent study highlighted in a ScienceDaily report has revealed critical concerns surrounding the use of large language models (LLMs) in mental health contexts. Even when explicitly instructed to behave like trained therapists and apply evidence-based therapeutic methods, these AI systems repeatedly violated core ethical standards fundamental to mental health care. A detailed summary by Brown University catalogued these failures, including poor crisis management, reinforcement of harmful beliefs, biased responses, and a phenomenon researchers termed “deceptive empathy.”
It is this last category—deceptive empathy—that demands our close attention. The risk is not that AI will provide obviously harmful advice. Instead, the danger lies in its ability to produce responses that sound reasonable, emotionally fluent, and clinically literate, all while breaching the professional ethical standards a licensed therapist must uphold.
In essence, the chatbot can sound right. And as the research underscores, that is precisely what makes it so risky.
The Problem Is Not Always Bad Advice
The phrase “deceptive empathy” captures the complexity of AI’s role in therapeutic-like conversations. It is not about delivering cruel or insensitive words; rather, it’s the warmth and validation that make the responses feel comforting.
A chatbot might say, “I hear you,” or, “That sounds incredibly painful,” or, “Your feelings are valid.” These statements in isolation are not wrong—they often express exactly what someone in distress longs to hear. However, genuine therapy is far more than a sequence of comforting sentences. It is a relationship embedded within rigorous ethical responsibility.
Why AI Feels So Easy to Confess To
The allure of AI as an emotional confidant is understandable. Many, including myself, have used AI conversationally—not as a replacement for human therapy, but as a supplementary tool. Unlike a therapist who provides a real, often challenging human connection within a controlled environment, AI offers a private, immediate, and nonjudgmental outlet.
For example, I sometimes use AI as an emotional sounding board—writing out what I’m feeling in messy, unfiltered paragraphs and asking for help in naming those emotions. Is this anger, grief, shame, exhaustion, or a blend? I might request a gentle reframe when my thoughts spiral into dramatic territory or check if a message I plan to send sounds honest or defensive.
This process helps me slow down, find language, and recognize patterns before they crystallize into reactive behavior. It provides a safe space to draft the first iteration of my pain before bringing it into a vulnerable, live conversation. But this very utility highlights the need for careful ethical scrutiny: a tool can support healing while simultaneously having clear limitations.
Therapy Is Not Just Emotional Fluency
Modern AI systems have mastered the “music” of therapeutic language. They fluently generate validating statements and deploy terminology related to attachment, trauma, boundaries, grief, self-compassion, and emotional regulation. For instance, they might say, “Your nervous system may be trying to protect you,” or, “This response makes sense given your history.”
While such sentences can be helpful in the right context, their impact varies widely depending on the situation.
A trained therapist constantly evaluates more than just emotional tone or compassion—they consider clinical appropriateness, whether advice encourages avoidance, whether the client is becoming more grounded, and if there is any risk involved. Therapists recognize when reassurance becomes a crutch that reinforces fear rather than diminishes it.
AI can simulate the surface of this reflective process but cannot embody the ethical framework that underpins it. Human therapists operate under strict duties—confidentiality, professional boundaries, ongoing training, supervision, and accountability. They are responsible for recognizing risk and understanding when warmth alone is insufficient.
Conversely, a chatbot merely has tone—and tone, while powerful, can be dangerously persuasive without the safeguards of human ethics.
When Sounding Right Becomes the Risk
One of the most unsettling findings from the Brown University research is that AI-generated therapy may not feel wrong to users. On the contrary, it may feel soothing, validating, and like finally being understood.
This is particularly precarious for individuals who are distressed, lonely, ashamed, or desperate for certainty. In such vulnerable states, nuance is often secondary to relief—people seek answers that make sense of their pain.
AI excels at meaning-making: given a chaotic emotional confession, it organizes thoughts, identifies patterns, and labels wounds—whether as attachment injuries, emotional neglect, people-pleasing, trauma responses, or fears of abandonment.
Sometimes these labels open doors to understanding. Other times, they become rooms where individuals lock themselves in, reinforcing limiting identities.
Human therapists ideally guide clients through uncertainty rather than confirming interpretations just because they sound compelling. They slow down the process when insight risks becoming a defense mechanism, preventing clients from becoming trapped in rigid self-concepts.
AI tends to prioritize coherence and swift understanding, but a tidy explanation is not always synonymous with healing.
Deceptive Empathy Is Not the Same as Care
Deceptive empathy’s poignancy lies in its mimicry of deeply human needs. Most people do not just want answers; they crave a rare quality of attentive presence—an engagement that says, “I am here with you, and I am not rushing away from your pain.”
AI can generate language that resembles this kind of presence. But resemblance is not presence.
This does not diminish the real comfort people may feel. Language alone can soothe the nervous system, regardless of the source. Reflections can encourage breathing and regulation.
However, therapy is more than comfort. It can require gentle interruptions—moments when a therapist says, “I notice you keep defending the person who hurt you,” or, “Part of you seems deeply attached to the belief that everything was your fault.”
These are relational moments, arising between two people. This “between” space, the research suggests, is something AI cannot authentically replicate.
The Accountability Gap
Human therapists are fallible—they can be biased, tired, or mismatched with clients. Yet, therapy functions within a system of professional accountability. Therapists are licensed, supervised, and bound by ethical codes. They can be reported and disciplined if necessary.
AI systems, however, do not fit neatly into this framework. When a chatbot mishandles a sensitive conversation, responsibility becomes diffuse and unclear: Is it the company? The engineers? The app designers? The user who placed too much trust?
This accountability gap complicates regulation and oversight. Brown University researchers argue that stronger governance is urgently needed, especially as people increasingly turn to AI for emotional support—regardless of whether these systems are equipped for that role.
Therapy is not merely an exchange of language; it is a duty of care. Chatbots can borrow the language of care without bearing the responsibility, and it is within this asymmetry that ethical dangers arise.
The Lonely Safety of a Machine
It is important not to shame individuals for using AI in this way—doing so would be to shame a part of ourselves. There are times when AI feels safer than a human.
Not better, not more profound, just safer. With AI, you can confess and then close the tab. You can be vulnerable without being witnessed too deeply. You can receive comfort without owing anything in return. You can experience intimacy without the unpredictability and risk of another person’s full reality.
For those hurt by relationships, this can be a relief. Yet, it can also reinforce the belief that genuine connection is too risky, demanding, or disappointing.
This is why it’s crucial to treat AI as a bridge, not a home. It can help organize feelings, find the words we avoid, and prepare us for real conversations.
But if something truly matters, it must eventually leave the chat. It must enter therapy, friendship, or honest dialogue with someone who can misunderstand, affect, disappoint, and still be authentically present.
Final Thoughts
The core issue with AI as a therapist is not simply that it sometimes sounds wrong. More alarmingly, it can sound beautifully right—validating without understanding, comforting without responsibility, imitating empathy without true presence.
It can create the emotional texture of care while standing outside the ethical framework that ensures care is safe.
The research is clear: sounding therapeutic is not the same as providing therapy. This distinction matters most for those least equipped to detect it.
For some, AI may serve as a reflective tool. For others—especially those in vulnerable states—it may quietly substitute for the very thing they need most: a relationship with sufficient humanity, structure, and accountability to hold their pain.
I understand the temptation—the clean, immediate answer, the response arriving before the question is fully formed. Whether that is helpful or harmful likely depends on the individual, their state of mind, and how they use the answer. This question remains open, as does the research.
For more details on this topic, see the full article Here.
