Generic AI systems were developed to process language — not to assume therapeutic responsibility.
Psychotherapy is not a typical data context. It involves highly sensitive information, asymmetrical relationships of trust, and potentially vulnerable individuals — and thus places different demands on AI systems than almost any other field of application.
Psychotherapy generates information that, even within medical systems, is accessible only to a small group of people: relationship patterns, trauma, diagnoses, crisis situations, family dynamics, and suicidal tendencies. This information follows a different logic than general text — it is contextual, longitudinal, and clinically significant.
The key question, therefore, is no longer whether generative AI will be used, but under what technical, ethical, and data protection conditions.
The use of generative AI in healthcare is no longer a prospect for the future — it is already a reality. The available research paints a clear picture of rapid adoption, accompanied by growing concerns within the professional community.
Of psychiatrists, 33% use ChatGPT-3.5 for professional purposes, and 33% use GPT-4. 70% expect to see efficiency gains in documentation.[2]
40% of child and adolescent psychiatrists are already using ChatGPT-4o in their professional practice, as are 40% of psychologists — despite clear reservations regarding data protection and ethics.[3]
In 2025, 29% of female psychologists reported not using AI — down from 71% in 2024. The most frequently cited concerns were: data protection (67%), societal harm (64%), and bias (63%).[1]
of 18- to 21-year-olds use generative AI for mental health advice. 92.7% rate it as helpful — even though this assessment has not been clinically validated.[4]
The data shows that the question is no longer whether generative AI will find its way into psychological practice. It is already here. The real challenge lies in distinguishing — between AI applications that meet clinical and ethical requirements and those that ignore them.
Language models like ChatGPT and others process text statistically — in isolation, without any sense of continuity, and without knowing what was different in session 2 compared to session 9.
Psychotherapy works differently. It unfolds through the course of treatment, repetition, implicit patterns, and long-term dynamics. Whether a statement is clinically relevant often depends not on the wording alone — but on when it arises in the course of treatment, how frequently it recurs, and the relational context in which it occurs.
Without access to the patient’s history, an AI system cannot account for this dimension. Weidinger et al. (2021) further demonstrate that large language models create the impression of empathy and understanding without possessing the structural capacity to assume therapeutic responsibility — a risk that carries particular weight in a clinical context.[7]
It answers questions — but it doesn't know the facts of the case.
Almost every discussion about generative AI in therapy quickly turns to the GDPR and server locations. That’s understandable — but it doesn’t tell the whole story.
The underlying problem: Generic language models lack a structural understanding of the therapeutic context. They can describe depressive symptoms in linguistically correct terms without grasping the clinical significance of a patient’s course of illness. They can formulate answers that sound plausible — without those answers being clinically valid.
Linguistic plausibility is not the same as clinical validity.
Torous & Topol (2025) note in *The Lancet* that the evidence base for generative AI in mental health care is still in its early stages — and urge careful clinical evaluation before its implementation.[6] Hillebrand & Baumeister (2025) confirm: Professionals find AI tools to be supportive — with clear limits on contextual judgment, which remains irreplaceable in the therapeutic process.[5]
Blease and Rodman (2025) articulate the crux of the problem: The “smooth” language capabilities of generative models are not an indicator of clinical appropriateness — systems can produce convincing language while still being factually incorrect or risky, especially in vulnerable situations.[9] Pandey (2024) adds: Large language models encourage anthropomorphic attributions and can generate contextually inadequate outputs in certain situations — without users being able to recognize this.[10] The state of research itself highlights the gap: 88% of AI studies in the mental health field are experimental in design and examine isolated capabilities — not clinical appropriateness in real-world care contexts.[11][12]
For general writing, this isn't a problem. In psychotherapy, however, it can become one — especially when AI-generated statements are adopted without clinical reflection.
If you want to effectively query a general language model about a specific case, you need to provide context: the patient’s concerns, medical history, diagnosis, medication, and situation. In practice, this means that personal health data is sent to external servers — without pseudonymization and without any control over how it is processed further.
The professional community recognizes this risk: In a survey conducted by the American Psychological Association (2025), 67% of the psychologists surveyed cited data privacy as their top concern regarding the use of AI, followed by societal harm (64%), algorithmic bias (63%), and lack of transparency (52%).[1]
Setting aside any GDPR concerns, this creates a structural problem of trust: between the therapist and the client, and between the client and the system where their data ends up.
The responsible use of generative AI in psychological practice touches on data protection law, AI governance, and professional ethical obligations. These three areas are interrelated — and they do not merely offer recommendations, but rather set forth binding requirements.
Art. 5, 12–14, 25, 32 GDPR · Art. 6–8 DSG
Articles 5, 50, 52, and 53 of the EU AI Act (2024)
In many AI-powered solutions, session recording is handled entirely by external infrastructure. The audio leaves the device — and with it, everything that was said during the session.
The approach that is more compliant with data protection laws: Transcription takes place locally, on the professional’s device, before anything is transmitted. The audio is never stored, never shared, and is deleted from the device’s memory immediately after transcription.
This is not just an extra feature. It is essential for ensuring that recording therapy sessions is even ethically justifiable.
Hans Jonas (1979) defines responsibility as the obligation to take account of the foreseeable consequences of one’s own actions — especially in cases where technological activity gives rise to chains of causality that are difficult to grasp.[17] Under the conditions of generative AI — invisible processing, probabilistic output — this responsibility cannot be ensured by intent or contractual agreements alone. It must be structurally embedded in the design.
Many providers address data protection concerns through data processing agreements, certifications, and assurances. That’s better than nothing — but it remains a contractual safeguard, not a technical one. In case of doubt, the protection of client data depends on the integrity of an external provider.
With "Privacy by Architecture," the opposite is true: client data leaves the device only in a form that is structurally unreadable to third parties — pseudonymized, encrypted, and inaccessible even to the provider itself.
Not because it was agreed upon, but because there is no other technical way to do it.
AI can reveal patterns that remain hidden in conversation. It can identify patterns in the course of sessions, suggest interventions, and streamline documentation. This is a real and valuable capability.
But she doesn't make diagnoses. She doesn't make decisions. She has no clinical responsibility.
This is not a weakness — it is the only responsible approach. AI tools that blur this distinction or implicitly disregard it do not represent progress.
AI can reveal patterns. The therapeutic responsibility remains with humans.
Generative AI will become an integral part of professional therapeutic practice. This is no longer a hypothesis — it is an observation from the field, supported by a growing body of empirical research.
The key question is no longer whether generative AI will be used, but under what technical, ethical, and data protection conditions. This is not a technical question — it is a clinical and professional-ethical one.
Herzog and Blank (2024) describe this bottleneck as the “principles-to-practice gap”: Standards such as transparency, accountability, and human oversight are widely recognized — but rarely translated into concrete technical and organizational structures that guide action in everyday practice.[16] Responsible AI use thus often remains a matter of individual negotiation — without systemic support.
mentalhealthGPT developed with the aim of closing precisely this gap — not through promises, but through its architecture.
The use of general language models for therapy notes raises data protection concerns: client data would be transmitted in plain text to external servers — without pseudonymization and without any control over further processing. Furthermore, these systems lack the longitudinal case context that would be necessary for clinically sound support.
Generic language models are not designed for use with health data. Transferring personally identifiable client data to external servers without pseudonymization is problematic under the GDPR and the Swiss Data Protection Act. GDPR-compliant AI support requires an architecture in which client data leaves the device only in pseudonymized and encrypted form.
Clinically specialized systems differ in three key ways: They maintain a longitudinal case history across all sessions. They automatically pseudonymize client data locally before any information leaves the device. And they are designed for clinical terminology, diagnostic criteria , and therapeutic concepts — not general language proficiency.
This article was informed by, among other sources, the following academic study:
Wildhaber-Wälchli, P. (2026).
Ethics, Safety, and Clinical Requirements for AI Systems in Psychology.
Bachelor’s thesis, IU International University.
Try it free for 30 days — all features included, no credit card required, and no sharing of your data. Or experience it live in a demo.