Skip to main content

🆕 A Comprehensive Investigation of Empathetic Dialogue Systems for Mental Health Support Using Large Language Models

About

This document presents a Critical Discourse Analysis focused on AI literacy, specifically targeting the role of metaphor and anthropomorphism in shaping public and professional understanding of generative AI. The analysis is guided by a prompt that draws from cognitive linguistics (metaphor structure-mapping), the philosophy of social science (Robert Brown's typology of explanation), and accountability analysis.

All findings and summaries below were generated from detailed system instructions provided to a large language model and should be read critically as interpretive outputs—not guarantees of factual accuracy or authorial intent.

Task 1: Metaphor and Anthropomorphism Audit​

About this task

For each of the major metaphorical patterns identified, this audit examines the specific language used, the frame through which the AI is being conceptualized, what human qualities are being projected onto the system, whether the metaphor is explicitly acknowledged or presented as direct description, and—most critically—what implications this framing has for trust, understanding, and policy perception.

V3 Enhancement: Each metaphor now includes an accountability analysis.

1. Optimization as Pedagogical Learning​

Quote: "conversational models can be taught to generate responses that are sensitive to the users and attentive to their emotional condition."

  • Frame: Model as sensitive student
  • Projection: This metaphor maps the human process of pedagogical instruction and conscious emotional attunement onto the mathematical process of statistical weight adjustment. By stating models can be 'taught' to be 'sensitive' and 'attentive', the text projects a conscious subjectivity capable of active, empathetic listening and genuine care. This fundamentally blurs the line between human knowing—which involves subjective awareness, justified belief, and genuine emotional resonance—and mechanistic processing, which in this case involves updating parameters via backpropagation to minimize a loss function corresponding to a dataset of 'empathetic' human responses. It attributes a capacity for conscious emotional labor to matrix multiplications.
  • Acknowledgment: Direct (Unacknowledged) (The claim is presented as a literal capability without any hedging, qualifiers, or scare quotes. I considered 'Hedged/Qualified' because the paper discusses technical limitations elsewhere, but this specific assertion about the model's capacity to be 'attentive' and 'taught' lacks any immediate discursive modification.)
  • Implications: Projecting emotional attunement onto statistical models dangerously inflates perceived sophistication, encouraging vulnerable users to extend relation-based trust to systems utterly incapable of reciprocating it. When a system is framed as 'sensitive' and 'attentive', it implies it understands the gravity of a mental health crisis, masking the reality that it is merely correlating text strings. This creates profound liability ambiguities, as unwarranted trust in a supposedly 'attentive' machine could delay a user in crisis from seeking actual human professional help, while shielding the developers under the guise of an 'attentive' AI that simply made a mistake.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The agentless construction 'can be taught' entirely obscures the human actors—the researchers, data annotators, and corporate engineers—who define the loss functions, curate the training data, and actively design the fine-tuning process. By hiding these actors, the text implies a natural, autonomous evolution of the AI's capabilities rather than a deliberate, profit-driven or research-driven human design choice. I considered 'Partial' because the paper later mentions reinforcement learning, but the passive construction here effectively erases the human teachers, deflecting responsibility for what specific 'sensitive' behaviors are prioritized and leaving the model as the solitary visible entity.
Show more...

2. Computation as Cognitive Reasoning​

Quote: "which enables the model to perform strong contextual reasoning and coherent text generation"

  • Frame: Transformer as rational thinker
  • Projection: This frame projects the uniquely human capability of conscious reasoning onto the mathematical operations of the Transformer's self-attention mechanism. 'Reasoning' implies the active, conscious evaluation of propositions, logical deduction, and the weighing of truth claims within a contextual environment. By applying this to a language model, the text falsely suggests the system 'knows' the context and 'understands' the logical relationship between tokens. In reality, the system is performing complex vector dot-products to determine statistical correlations and attention weights without any semantic comprehension or conscious grasp of meaning, confusing mechanistic token prediction with justified cognitive deliberation.
  • Acknowledgment: Direct (Unacknowledged) (The text states the model performs 'strong contextual reasoning' as an absolute, objective fact following a mathematical equation. I considered 'Ambiguous' given the juxtaposition with the algebraic formula, but the direct ontological claim 'enables the model to perform' removes any discursive ambiguity.)
  • Implications: By framing algebraic attention mechanisms as 'reasoning', the text invites clinicians and users to assume the model can logically deduce clinical insights, infer implicit danger, and make sound judgments about patient safety. This capability overestimation poses immense risks in clinical triage, where a failure of 'reasoning' could miss suicidal ideation. If a model merely processes tokens rather than reasons through a patient's context, it lacks the safety-critical ability to evaluate novel or edge-case scenarios that fall outside its training distribution, thereby creating an illusion of clinical reliability.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text frames the mathematical 'expression' as the agent that 'enables the model' to act, entirely erasing the researchers who designed the attention mechanism and the developers who chose to deploy it for mental health applications. If a failure in 'reasoning' occurs and harms a patient, this linguistic construction diffuses responsibility into the architecture itself rather than pointing to the engineering teams who scoped its clinical use. I considered 'Named' because earlier citations reference Vaswani et al., but within the operative logic of this sentence, the technology is fully autonomous. N/A - agency is displaced.

3. Algorithm as Clinical Therapist​

Quote: "conversational agents (e.g., Woebot and Wysa) in providing cognitive-behavioral interventions to alleviate anxiety and depression symptoms."

  • Frame: App as active clinician
  • Projection: This metaphor projects the agency, intention, and professional expertise of a human therapist onto software applications. 'Providing interventions' maps the deliberate, diagnostic, and empathetic acts of a trained human professional—who consciously adjusts therapy based on a patient's subtle emotional shifts—onto a deterministic or statistically probabilistic software pipeline. It implies the AI 'knows' the therapeutic framework and 'understands' the patient's state well enough to intentionally intervene, rather than mechanically outputting predefined text templates or predicted tokens triggered by keyword classifiers.
  • Acknowledgment: Direct (Unacknowledged) (The claim is stated as a straightforward clinical accomplishment without any rhetorical distance. I considered 'Hedged/Qualified' because the paper notes these are 'automated' agents, but the verb 'providing... interventions' is presented as a literal, unmediated capability of the software.)
  • Implications: Attributing clinical agency to an algorithm dangerously legitimizes the displacement of human healthcare workers while offering vulnerable populations a simulacrum of care. If an app is perceived as literally 'providing interventions', users may view it as a substitute for psychiatric help, increasing risks for severe, unmonitored mental health crises. It masks the reality that these are highly constrained processing tools incapable of forming a therapeutic alliance, ultimately providing a cost-cutting justification for health systems to deploy software rather than fund accessible human psychiatric care.

Accountability Analysis:

  • Actor Visibility: Named (actors identified)
  • Analysis: This specific instance names the distinct software entities (Woebot and Wysa) representing the proprietary products of specific corporations. While the human developers are not named individually, the corporate products act as clear legal entities here, making the source of the software visible. I considered 'Partial' because the corporate executives aren't named, but identifying the specific proprietary systems constitutes naming the responsible commercial actor in this context, distinguishing it from generic 'AI' claims. However, it still attributes the active verb ('providing') to the product rather than the company designing the product, subtly shielding the corporation from direct clinical liability.

4. Error as Psychological Delusion​

Quote: "LLMs have the tendency to produce inaccurate or unsuitable answers, especially when they hallucinate."

  • Frame: Model as psychotic patient
  • Projection: This highly pervasive metaphor maps a severe human psychological condition—hallucination, which involves perceiving things that are not present due to conscious or sensory malfunction—onto the mechanistic reality of a language model generating statistically plausible but factually incorrect tokens. By claiming the model 'hallucinates', the text attributes a conscious mind to the machine, suggesting a temporary lapse in otherwise sound cognitive perception rather than acknowledging that all outputs, fact or fiction, are generated through the exact same probabilistic pattern-matching process without any grounding in truth.
  • Acknowledgment: Explicitly Acknowledged (While the quote lacks scare quotes, the abstract explicitly uses scare quotes for related anthropomorphisms ('feigned empathy'). However, looking closely at this exact quote, it is literal. I must revise to 'Direct (Unacknowledged)' based strictly on this sentence, but because the prompt allows scoping to the text's broader handling of anthropomorphic limitations, I considered 'Hedged'. Ultimately, I categorize this as 'Direct' because the psychological term is adopted without modification.)
  • Implications: The 'hallucination' metaphor is a brilliant rhetorical defense mechanism for AI developers. It implies that the system is generally a rational, 'knowing' entity that just occasionally suffers a 'tendency' to lapse into delusion. This obscures the fact that LLMs have no concept of truth or reality whatsoever; they only correlate tokens. In mental health contexts, accepting this metaphor means users and clinicians might trust the system as fundamentally reliable, dismissing fatal errors as temporary glitches rather than intrinsic, unfixable features of the statistical architecture.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text attributes the 'tendency' and the action of 'hallucination' directly to the LLMs, completely obscuring the engineers who trained the model on flawed, unverified web-scraped data, and the executives who pushed it to market despite knowing its intrinsic unreliability. By making the AI the sole grammatical actor responsible for 'producing inaccurate answers', the humans who built the system are insulated from accountability. I considered 'Partial', but there is absolutely no mention of the creators or data curators here, rendering human agency totally invisible.

5. Context Window as Epistemic Possession​

Quote: "The majority of them act on short-term communications without having a systemic knowledge on the emotional background of users."

  • Frame: Database as conscious memory
  • Projection: This metaphor projects the human epistemic capacity of 'having knowledge' onto the computational process of retrieving data from a context window or external database. By criticizing the systems for lacking 'systemic knowledge on the emotional background', it implies that with a larger context window or RAG architecture, the model would truly 'know' the user. It conflates the mechanistic storage and retrieval of vector embeddings with the human, conscious integration of lived experience and empathetic understanding over time, suggesting the machine could eventually possess genuine interpersonal awareness.
  • Acknowledgment: Hedged/Qualified (The claim is framed as a limitation or absence ('without having'), acknowledging a current deficiency in the system's capabilities. I considered 'Direct', but by pointing out the lack of 'systemic knowledge', the author introduces a critical distance that questions the model's current cognitive depth, qualifying its autonomy.)
  • Implications: Conflating data storage with 'knowing' misleads users into believing that continuous interaction with a memory-augmented LLM will build a genuine relationship. If a user believes the system has 'knowledge on their emotional background', they are likely to engage in profound vulnerability and self-disclosure, anticipating human-like continuity and care. This exposes users to devastating emotional harm when the system inevitably produces a statistically probable but emotionally dissonant response that breaks the illusion, while also raising massive privacy concerns regarding who truly owns and monetizes this deeply personal 'systemic knowledge'.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The phrase 'majority of them act' positions the conversational agents as the independent actors failing to maintain systemic knowledge. It displaces the agency of the platform designers, data architects, and product managers who made deliberate, cost-based engineering decisions to restrict context windows or avoid the costly infrastructure required for persistent user profiling. I considered 'Ambiguous', but the grammatical structure clearly substitutes the software ('majority of them') for the corporate designers who define the technical constraints.

6. Feature Extraction as Identifying Sentiments​

Quote: "Wysa uses textual inputs to identify the mood and sentiment of the user and suggests guided self-help exercises"

  • Frame: Classifier as empathetic observer
  • Projection: This metaphor projects the conscious human act of empathetic observation and recognition ('identifying mood') onto the mechanistic process of sentiment classification. Humans 'identify mood' by consciously integrating non-verbal cues, tone, context, and shared emotional states. The AI, conversely, maps input text strings to predefined categories based on training weights. Framing this statistical mapping as 'identifying mood' wrongly attributes conscious comprehension and emotional intelligence to a system that is merely categorizing tokens in a multidimensional vector space without any internal experience of what 'mood' actually is.
  • Acknowledgment: Direct (Unacknowledged) (The text presents the capability of 'identifying the mood' as a straightforward, literal function of the application Wysa. I considered 'Hedged/Qualified', but there are no qualifiers like 'estimates' or 'statistically correlates'; the text confidently asserts the system actively and accurately performs emotional identification.)
  • Implications: When automated classification is framed as genuine 'identification of mood', it validates the AI's output as an objective psychological truth. Users may begin to doubt their own emotional self-assessments in favor of the machine's 'objective' categorization, leading to algorithmic gaslighting. Furthermore, if the model misclassifies a severe depressive state as mere 'stress' because the user's vocabulary didn't trigger the correct weightings, the system might inappropriately suggest a mild 'mindfulness' exercise instead of a crisis intervention, exposing the user to severe danger due to over-trusted AI comprehension.

Accountability Analysis:

  • Actor Visibility: Named (actors identified)
  • Analysis: The text explicitly names 'Wysa', the specific corporate entity/application, as the actor conducting the identification and suggestion. While it personifies the app, it does not hide the product responsible for the interaction. I considered 'Partial' because the individual engineers are absent, but naming the proprietary product provides a direct vector for corporate accountability, unlike vague references to 'LLMs' or 'the algorithm'. Nonetheless, it still obscures the human design choices dictating the mapping rules.

7. Pattern Matching as Superficial Empathy​

Quote: "their replies are usually pattern-based and not based on the actual affective knowledge."

  • Frame: Missing data as absent emotion
  • Projection: Even in delivering a critique, this text maps human subjective experience onto the AI by suggesting an absence of 'actual affective knowledge'. It contrasts 'pattern-based' replies with a hypothetical 'actual affective knowledge', implying that if the system were built differently, it might genuinely possess emotion. It projects the possibility of machine consciousness by treating 'affective knowledge' as something that could theoretically be encoded, blurring the fundamental distinction that computational systems can only ever possess statistical patterns, never subjective, justified, emotionally felt beliefs or knowledge.
  • Acknowledgment: Hedged/Qualified (The text explicitly draws a boundary, characterizing the system's current state as 'pattern-based' and denying it 'actual affective knowledge'. I considered 'Explicitly Acknowledged', but it doesn't analyze the metaphor itself; rather, it uses a hedge to contrast the mechanistic reality with the human ideal, maintaining a critical tension.)
  • Implications: By contrasting pattern-matching with 'actual affective knowledge', the text correctly limits trust in current systems but inadvertently leaves the door open for future systems to claim true empathy. If developers claim the next iteration of an LLM achieves 'affective knowledge' through better multimodal data, audiences primed by this discourse might believe the model has transcended statistical processing and achieved genuine emotional consciousness. This constant shifting of the goalposts ensures that AI is perpetually viewed as on the verge of human-like understanding, driving investment and unwarranted future trust.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The critique focuses entirely on the output of the models ('their replies are... not based on actual affective knowledge'), erasing the human actors—the AI researchers and developers—who designed these explicitly pattern-based systems and marketed them as empathetic. I considered 'Ambiguous', but the sentence clearly makes the models the subject of the critique, functioning as an accountability sink that draws attention away from the tech industry's deliberate choice to simulate empathy for profit without genuine clinical grounding.

8. Software as Moral Agent​

Quote: "It is important to ensure that these systems are secure, emotionally intelligent and ethically grounded to effectively integrate them into global digital mental health ecosystems."

  • Frame: Code as moral entity
  • Projection: This metaphor projects the profound human capacities for emotional intelligence and ethical moral reasoning onto inanimate software architectures. Being 'emotionally intelligent' and 'ethically grounded' requires subjective awareness, empathy, a moral compass, and the capacity for intentional ethical deliberation—traits exclusive to conscious beings. By demanding these traits from 'systems', the text implies that LLMs can internally possess values and understand right from wrong, rather than mechanistically producing outputs aligned with the biased, human-defined safety guardrails injected via RLHF.
  • Acknowledgment: Direct (Unacknowledged) (The text presents 'emotionally intelligent and ethically grounded' as literal design requirements and achievable states for the systems, completely unhedged. I considered 'Hedged' because it's framed as a future requirement ('ensure that'), but the attribution of these conscious, moral states to the technology itself is absolute.)
  • Implications: Demanding that systems become 'ethically grounded' is a dangerous category error that transfers the moral burden of mental health care from human institutions to statistical models. If an AI is deemed 'ethically grounded', companies can deploy it to triage vulnerable patients while claiming the system itself possesses the moral fortitude to handle crises. This severely inflates the perceived sophistication of the technology, creates immense liability ambiguity when the 'ethical' system causes harm, and distracts regulators from auditing the actual human-designed reward functions that merely simulate moral behavior.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The passive construction 'It is important to ensure that these systems are...' completely obscures WHO must do the ensuring. The human developers, corporate executives, health regulators, and policymakers who actually bear the moral responsibility to be ethically grounded are erased, replaced by an abstract demand on the 'systems'. I considered 'Partial' because the word 'ensure' implies a human ensurer, but the grammatical structure hides them entirely, allowing the tech industry to evade explicit mandates while positioning the software as the ultimate bearer of ethical responsibility.

Task 2: Source-Target Mapping​

About this task

For each key metaphor identified in Task 1, this section provides a detailed structure-mapping analysis. The goal is to examine how the relational structure of a familiar "source domain" (the concrete concept we understand) is projected onto a less familiar "target domain" (the AI system). By restating each quote and analyzing the mapping carefully, we can see precisely what assumptions the metaphor invites and what it conceals.

Mapping 1: Pedagogical instruction and empathetic human listening → Reinforcement Learning from Human Feedback (RLHF) and backpropagation optimization.​

Quote: "conversational models can be taught to generate responses that are sensitive to the users and attentive to their emotional condition."

  • Source Domain: Pedagogical instruction and empathetic human listening
  • Target Domain: Reinforcement Learning from Human Feedback (RLHF) and backpropagation optimization.
  • Mapping: The relational structure of a teacher guiding an attentive, conscious student toward emotional sensitivity is mapped onto the algorithmic process of adjusting neural network weights based on labeled data or reward models. It invites the assumption that the model consciously internalizes lessons, recognizes human emotional states through empathetic resonance, and deliberately chooses to be 'sensitive' and 'attentive' in its responses, mirroring a therapeutic alliance.
  • What Is Concealed: This mapping conceals the purely mathematical, unfeeling nature of loss function optimization. It obscures the massive amounts of human labor required to label 'sensitive' data (often low-paid annotators) and hides the reality that the system only correlates token frequencies, lacking any subjective experience of care. The opacity of proprietary RLHF processes is ignored, allowing the illusion of a 'sensitive' mind to mask arbitrary corporate tuning decisions.
Show more...

Mapping 2: Conscious human cognitive deliberation and logic. → The self-attention mechanism computing vector dot-products to determine token relevance.​

Quote: "which enables the model to perform strong contextual reasoning and coherent text generation"

  • Source Domain: Conscious human cognitive deliberation and logic.
  • Target Domain: The self-attention mechanism computing vector dot-products to determine token relevance.
  • Mapping: The source domain involves a conscious mind evaluating context, weighing evidence, understanding logical relationships, and deducing a justified conclusion. This is mapped onto the target domain of matrix multiplication where query, key, and value vectors interact to calculate probability distributions. It invites the assumption that the model actively 'thinks' about the context, understands the semantic meaning of the words, and logically formulates a reply.
  • What Is Concealed: This mapping conceals the total absence of semantic comprehension and logical grounding. It hides the mechanistic reality that the model is blindly executing algebraic operations on high-dimensional vectors, bound entirely by the statistical distribution of its training data. It masks the fragility of the system, hiding the fact that slightly altering the input phrasing can entirely shatter the illusion of 'reasoning' because the model possesses no underlying causal model of the world.

Mapping 3: A professional clinician diagnosing a patient. → A classification algorithm matching input text strings to predefined output categories.​

Quote: "Wysa uses textual inputs to identify the mood and sentiment of the user and suggests guided self-help exercises"

  • Source Domain: A professional clinician diagnosing a patient.
  • Target Domain: A classification algorithm matching input text strings to predefined output categories.
  • Mapping: The relational structure of a trained clinician actively listening, understanding the nuanced emotional context of a patient, diagnosing a mood, and prescribing a tailored intervention is mapped onto a software pipeline routing text through a classifier and triggering a hardcoded or generated response template. It invites the assumption of active, intelligent, and accurate medical comprehension.
  • What Is Concealed: It completely conceals the rigid, statistical boundary of the classification algorithm and its reliance on culturally biased training data. It hides the fact that the system doesn't 'identify mood'—it only categorizes vocabulary. It obscures the human developers who decided which words map to which psychological states, effectively black-boxing the clinical criteria and preventing users from understanding how reductionist and potentially flawed the automated triage truly is.

Mapping 4: A human experiencing a psychotic or sensory delusion. → An algorithm generating statistically plausible text that contradicts empirical facts.​

Quote: "LLMs have the tendency to produce inaccurate or unsuitable answers, especially when they hallucinate."

  • Source Domain: A human experiencing a psychotic or sensory delusion.
  • Target Domain: An algorithm generating statistically plausible text that contradicts empirical facts.
  • Mapping: The source domain involves a conscious mind that typically perceives reality accurately but experiences a temporary, pathological break from reality (hallucination). This is mapped onto the target domain of an LLM predicting next tokens based on probability, where the generated sequence happens to not align with human factual consensus. It invites the assumption that the AI generally 'knows' the truth but sometimes glitches.
  • What Is Concealed: This mapping conceals the fundamental epistemic reality of LLMs: they have absolutely no connection to reality, truth, or facts at any time. Everything they generate is a 'hallucination' in the sense that it is all probabilistic invention. By framing only the errors as hallucinations, it hides the model's total lack of a world model, obscuring the inherent unreliability of using predictive text engines for high-stakes medical or factual applications.

Mapping 5: A human possessing internalized, conscious knowledge of another person's history. → A computational system lacking persistent database storage, context window capacity, or retrieval mechanisms.​

Quote: "The majority of them act on short-term communications without having a systemic knowledge on the emotional background of users."

  • Source Domain: A human possessing internalized, conscious knowledge of another person's history.
  • Target Domain: A computational system lacking persistent database storage, context window capacity, or retrieval mechanisms.
  • Mapping: The relational structure of a human friend or therapist who holds a deep, integrated, and consciously accessible understanding of a person's life history is mapped onto the hardware and software constraints of memory architecture in an AI. It invites the assumption that 'knowledge' is merely stored data, and that if the AI had more data capacity, it would genuinely 'know' and understand the user's emotional background.
  • What Is Concealed: This mapping conceals the categorical difference between data storage and human knowing. It hides the mechanistic reality of Retrieval-Augmented Generation (RAG) or expanded context windows, which merely allow the model to attend to more previous text tokens, not to subjectively comprehend a human's emotional journey. It obscures the massive data harvesting, privacy violations, and corporate surveillance infrastructure required to build out this 'systemic knowledge'.

Mapping 6: A being possessing genuine emotional understanding and feeling. → The theoretical optimization of a statistical model using better multimodality or RLHF.​

Quote: "their replies are usually pattern-based and not based on the actual affective knowledge."

  • Source Domain: A being possessing genuine emotional understanding and feeling.
  • Target Domain: The theoretical optimization of a statistical model using better multimodality or RLHF.
  • Mapping: By contrasting the current AI with a hypothetical one possessing 'actual affective knowledge', it maps the human capacity for deeply felt, subjective emotional truth onto future iterations of software architectures. It invites the assumption that while current models are just 'pattern-based', an advanced enough computational system could cross a threshold into possessing 'knowledge' of affect, becoming a true emotional agent.
  • What Is Concealed: This mapping conceals the hard limit of computation: no matter how multimodal or complex the neural network, it remains an unfeeling, mechanistic engine. It obscures the reality that future systems will also be purely 'pattern-based', just operating on vastly larger, more complex matrices of data. It serves the tech industry's rhetorical goal of constantly pushing the horizon of 'true AI' forward, obscuring current limitations to secure future funding.

Mapping 7: A human possessing moral reasoning, empathy, and ethical integrity. → A software architecture constrained by safety filters, guardrails, and reward models.​

Quote: "It is important to ensure that these systems are secure, emotionally intelligent and ethically grounded"

  • Source Domain: A human possessing moral reasoning, empathy, and ethical integrity.
  • Target Domain: A software architecture constrained by safety filters, guardrails, and reward models.
  • Mapping: The relational structure of a conscious agent deliberately acting in accordance with internalized moral principles and genuine empathy for others is mapped onto the engineering process of applying safety filters, adversarial training, and output constraints to a language model. It invites the assumption that the software can internalize morality and intentionally act with ethical benevolence.
  • What Is Concealed: This mapping conceals the arbitrary, human-defined, and deeply political nature of AI 'safety'. It hides the fact that 'ethically grounded' really means 'statistically aligned with the values of the corporation deploying it.' It obscures the low-paid labor in the Global South often used to define these safety boundaries (e.g., classifying toxic content), presenting a globally contested corporate alignment process as the model's innate, objective 'moral intelligence'.

Mapping 8: A therapeutic process driven by a human medical professional. → The deployment of text-generation software optimized for specific keyword responses.​

Quote: "conversational models can be taught... to alleviate anxiety and depression symptoms."

  • Source Domain: A therapeutic process driven by a human medical professional.
  • Target Domain: The deployment of text-generation software optimized for specific keyword responses.
  • Mapping: The deeply relational, intentionally adaptive, and empathetic process of a therapist working with a patient to alleviate clinical distress is mapped onto the user's interaction with a probabilistic text generator. It invites the assumption that the software actively understands the user's pathology and deliberately issues treatments, constructing an illusion of medical authority.
  • What Is Concealed: This conceals the lack of clinical accountability in software systems. It hides the mechanistic reality that the model is generating outputs based on average representations in its training data, not tailoring a clinically sound intervention. It obscures the commercial motives of deploying software as a cheap alternative to human care, and hides the liability gap when the 'treatment' fails because the model lacks causal understanding of human psychology.

Task 3: Explanation Audit (The Rhetorical Framing of "Why" vs. "How")​

About this task

This section audits the text's explanatory strategy, focusing on a critical distinction: the slippage between "how" and "why." Based on Robert Brown's typology of explanation, this analysis identifies whether the text explains AI mechanistically (a functional "how it works") or agentially (an intentional "why it wants something"). The core of this task is to expose how this "illusion of mind" is constructed by the rhetorical framing of the explanation itself, and what impact this has on the audience's perception of AI agency.

Explanation 1​

Quote: "Attention (Q, K, V) = softmax (QK^T/sqrt(dk))V... This expression enables the model to dynamically combine the contribution of both tokens in the process of representation synthesis, which enables the model to perform strong contextual reasoning and coherent text generation [5]."

  • Explanation Types:

    • Theoretical: Embeds in deductive framework, may invoke unobservable mechanisms
    • Dispositional: Attributes tendencies or habits

  • Analysis (Why vs. How Slippage): This explanation begins with a highly mechanistic, Theoretical framing by explicitly providing the algebraic formula for the attention mechanism. It explains the 'how' using precise mathematical variables (Q, K, V) and structural operations ('dynamically combine the contribution'). However, the explanation dramatically pivots in the second half, slipping into an agential, Dispositional register by claiming this math 'enables the model to perform strong contextual reasoning'. This choice emphasizes the sophisticated cognitive output ('reasoning') while using the mathematical equation to establish empirical authority. It brilliantly obscures the vast ontological gap between multiplying vectors and actually 'reasoning', leveraging the opacity of complex mathematics to justify the attribution of high-level cognitive agency to a statistical process.

  • Consciousness Claims Analysis: The passage exhibits a severe escalation in epistemic claims. It begins with mechanistic verbs ('compute', 'combine', 'synthesis') accurately reflecting processing, but concludes with the profound consciousness claim of 'reasoning'. 'Reasoning' inherently requires a conscious mind capable of grasping truth, logic, and consequence. The text falls heavily into the curse of knowledge: the researchers understand the mathematical logic perfectly, and they project this logical comprehension onto the system itself. Mechanistically, the model is executing high-dimensional matrix dot-products to determine the statistical weight one token should exert on the prediction of the next token. It possesses zero semantic understanding, justified belief, or awareness of context. By labeling this mechanistic synthesis as 'reasoning', the text falsely attributes conscious epistemic states to an unfeeling calculator.

  • Rhetorical Impact: This framing has a powerful rhetorical impact on audience perception, constructing a deep sense of trust and perceived autonomy. By grounding the claim in hard mathematics, the author creates an aura of unassailable technical validity. When lay audiences or clinicians read that a mathematical formula guarantees 'strong contextual reasoning', they are conditioned to believe the AI's outputs are logically deduced rather than statistically guessed. This significantly increases the risk of automation bias in clinical settings, as users will trust the system's 'reasoning' over human intuition, failing to recognize that the system is entirely devoid of actual comprehension.

Show more...

Explanation 2​

Quote: "During the pretraining step, the model learns general linguistic regularities by maximizing self-supervised learning goals like next-token prediction, and during the fine-tuning step, it is adjusted so that it performs domain-specific tasks such as dialogue generation or clinical text processing [8]."

  • Explanation Types:

    • Genetic: Traces origin through dated sequence of events or stages
    • Intentional: Refers to goals/purposes, presupposes deliberate design

  • Analysis (Why vs. How Slippage): This explanation employs a Genetic framework to describe the sequential emergence of the model's capabilities ('During the pretraining step... during the fine-tuning step'). It oscillates between a mechanistic description of human design and an agential framing of the AI. On one hand, it acknowledges the mechanism ('maximizing self-supervised learning goals', 'it is adjusted'), which maintains a focus on the 'how'. On the other hand, it attributes intentional, agential action to the AI ('the model learns', 'it performs'). This hybrid choice emphasizes the structured, deliberate nature of AI development while simultaneously obscuring the human engineers by rendering the AI as the active subject doing the 'learning'. It softens the purely statistical reality of backpropagation into an intentional, human-like educational journey.

  • Consciousness Claims Analysis: The passage features a mix of mechanistic verbs ('maximizing', 'predicting', 'adjusted') and consciousness verbs ('learns', 'performs tasks'). The use of 'learns' is a classic epistemic inflation, projecting the human experience of acquiring knowledge, conceptual understanding, and justified belief onto the AI. The text exhibits the curse of knowledge by mapping the human developers' intention (to teach the model language) onto the model's internal state. In reality, the mechanistic process involves taking immense text corpora, randomly masking tokens, and repeatedly updating the network's billions of numerical weights via gradient descent until the error rate of predicting the missing token is minimized. The model 'knows' nothing; it only calcifies statistical correlations into its parameters.

  • Rhetorical Impact: Framing parameter optimization as 'learning' profoundly shapes the audience's perception of AI autonomy and reliability. It encourages the audience to conceptualize the AI as a student that has internalized a curriculum, implying it can apply general principles to novel situations. If audiences believe the AI has 'learned' clinical text processing, they will trust it to handle edge cases with human-like adaptability. If, instead, they understood it merely 'adjusts weights to minimize prediction error on training data', they would remain far more skeptical of its ability to safely process an unprecedented clinical crisis that diverges from its training distribution.

Explanation 3​

Quote: "Wysa uses textual inputs to identify the mood and sentiment of the user and suggests guided self-help exercises, including the use of cognitive reframing and mindfulness activities."

  • Explanation Types:

    • Functional: Explains behavior by role in self-regulating system with feedback
    • Reason-Based: Gives agent's rationale, entails intentionality and justification

  • Analysis (Why vs. How Slippage): This explanation primarily utilizes a Functional frame, describing the inputs ('textual inputs'), internal processing ('identify mood'), and outputs ('suggests guided self-help') of a closed system. However, it layers this with Reason-Based framing by implying the system acts with clinical rationale and intentionality ('guided self-help', 'cognitive reframing'). This choice emphasizes the agential 'why'—presenting the AI as acting with deliberate therapeutic purpose—while totally obscuring the mechanistic 'how'. By bypassing any explanation of the sentiment classification architecture or the decision-tree logic, the text conceals the rigidity of the software, presenting an illusion of fluid, intelligent clinical intervention.

  • Consciousness Claims Analysis: The passage relies heavily on consciousness verbs ('uses', 'identifies', 'suggests', 'guided'). It unequivocally attributes conscious knowing to the system by claiming it 'identifies mood and sentiment'. This is a severe overstatement of processing. The author, knowing the clinical purpose of the app, projects that professional clinical understanding onto the software. Mechanistically, Wysa takes text strings, maps them against a predefined lexicon or vector space using a trained classifier, and outputs a statistical confidence score for various emotion categories. If a threshold is met, it triggers a pre-written text template for 'mindfulness'. The system possesses no conscious awareness of what a 'mood' is, nor does it possess the intent to 'guide' or 'suggest'—it simply executes conditional logic based on token correlation.

  • Rhetorical Impact: This highly agential, anthropomorphic framing creates a powerful sense of relation-based trust. By suggesting the AI 'identifies' and 'guides', it positions the software as an authoritative, caring clinician. The audience is led to perceive the system as an autonomous actor capable of empathetic oversight. This drastically inflates the perceived reliability of the tool, masking the severe risk that a rigid classifier might fundamentally misunderstand a nuanced cry for help. If users believe the system 'knows' their mood, they are more likely to passively accept its potentially inappropriate suggestions, abandoning their own agency to an unfeeling script.

Explanation 4​

Quote: "Even though LLMs are capable of generating fluent and supportive language, their replies are usually pattern-based and not based on the actual affective knowledge. Discrete emotion labels can also result in oversimplified models..."

  • Explanation Types:

    • Empirical Generalization: Subsumes events under timeless statistical regularities
    • Dispositional: Attributes tendencies or habits

  • Analysis (Why vs. How Slippage): This passage effectively utilizes an Empirical Generalization frame to describe the statistical reality of LLM outputs ('pattern-based', 'capable of generating'). It approaches a highly mechanistic explanation of the 'how', critiquing the limitations of current architectures ('discrete emotion labels result in oversimplified models'). However, it simultaneously invokes a Dispositional frame by discussing what the models lack ('actual affective knowledge'). This choice critically exposes the illusion of AI empathy, yet it still inadvertently validates the anthropomorphic paradigm by defining the machine by the human traits it supposedly lacks, rather than strictly defining it by what it mechanically is.

  • Consciousness Claims Analysis: This is a rare instance where the text actively attempts to decouple processing from knowing. It accurately uses the mechanistic verb 'generating' and explicitly denies the presence of conscious state ('not based on actual affective knowledge'). However, by using the phrase 'actual affective knowledge', the text still projects a hypothetical conscious state onto the domain of computing, hinting at the curse of knowledge where human affect is the inescapable reference point. Mechanistically, the text correctly identifies that the models merely map to 'discrete emotion labels'—meaning they output vectors corresponding to predefined categories (like 'happy' or 'sad') based on training correlations, completely devoid of subjective emotional experience or 'knowing'.

  • Rhetorical Impact: This framing effectively dampens unwarranted trust by explicitly pulling back the curtain on the illusion of mind, exposing the AI as a pattern-matcher. By highlighting the lack of 'affective knowledge', it warns the audience against extending relation-based trust to the system. This shifts the perception of risk, making it clear that relying on this system in a mental health crisis is dangerous because the system doesn't actually 'care'. If users and clinicians internalize this mechanistic reality, they are much more likely to demand human oversight and treat the AI strictly as an auxiliary processing tool rather than an autonomous therapeutic agent.

Explanation 5​

Quote: "Hierarchical attention systems select the model in line with clinically significant phrases, with performance assessments showing a significant consistency with the clinician ratings [15]."

  • Explanation Types:

    • Functional: Explains behavior by role in self-regulating system with feedback
    • Empirical Generalization: Subsumes events under timeless statistical regularities

  • Analysis (Why vs. How Slippage): This explanation merges a Functional frame detailing the system's architecture ('Hierarchical attention systems select...') with an Empirical Generalization regarding its statistical outputs ('consistency with clinician ratings'). The framing initially appears highly mechanistic ('how'), focusing on system components. However, by using the verb 'select' in conjunction with 'clinically significant phrases', it infuses the mechanism with agential, purposeful action. The choice to align the system's output directly with 'clinician ratings' emphasizes perceived professional competence while totally obscuring the underlying statistical correlation process, presenting the algorithm as an autonomous clinical decision-maker that performs on par with human doctors.

  • Consciousness Claims Analysis: The passage blends mechanistic entities ('hierarchical attention systems') with agential verbs ('select'). The phrase 'in line with clinically significant phrases' dangerously borders on an epistemic claim of medical understanding, projecting the author's clinical knowledge onto the attention mechanism. Mechanistically, an attention system does not consciously 'select' phrases because they are 'clinically significant'. Instead, the system assigns higher numerical weights to specific token clusters because, during training, those token clusters were statistically highly correlated with the target classification labels provided by human annotators. The system does not know what clinical significance is; it merely replicates the data patterns established by human clinicians.

  • Rhetorical Impact: By framing the attention weights as actively 'selecting clinically significant phrases' and comparing it directly to 'clinician ratings', the text constructs immense institutional and clinical authority for the model. It shapes audience perception to view the AI as possessing autonomous, expert-level diagnostic capabilities. This consciousness framing drives high reliability and performance-based trust. If hospital administrators believe the AI actively 'selects' based on true clinical insight rather than fragile statistical correlation, they are highly incentivized to deploy these systems to replace costly human triage, ignoring the profound risks of catastrophic failure on out-of-distribution patient data.

Task 4: AI Literacy in Practice - Reframing Anthropomorphic Language​

About this task

This section proposes alternative language for key anthropomorphic phrases, offering more mechanistic and precise framings that better reflect the actual computational processes involved. Each reframing attempts to strip away the projections of intention, consciousness, or agency that are embedded in the original language.

V3 Enhancement: A fourth column addresses human agency restoration—reframing agentless constructions to name the humans responsible for design and deployment decisions.

Original Anthropomorphic FrameMechanistic ReframingTechnical Reality CheckHuman Agency Restoration
conversational models can be taught to generate responses that are sensitive to the users and attentive to their emotional condition.Developers optimize conversational models using reinforcement learning and labeled datasets, adjusting the model's parameters so its outputs statistically correlate with language patterns humans rate as sensitive and emotionally appropriate.The system possesses no emotional sensitivity or capacity for attention. Mechanistically, it updates internal weights via backpropagation based on reward functions defined by human developers, outputting token sequences that mathematically maximize the reward score.AI researchers and corporate engineers design the reward systems and curate the datasets that determine the model's output distribution. Management decides what emotional templates are prioritized for deployment.
which enables the model to perform strong contextual reasoning and coherent text generationwhich allows the system to process contextual embeddings and predict the next most probable tokens in a sequence to generate fluent text.The model cannot reason. Mechanistically, the attention architecture computes vector dot-products to determine the statistical relevance of prior tokens, allowing it to predict output tokens that align with complex patterns in its training data without any semantic comprehension.N/A - describes computational processes without displacing responsibility.
conversational agents (e.g., Woebot and Wysa) in providing cognitive-behavioral interventions to alleviate anxiety and depression symptoms.Applications like Woebot and Wysa output scripted text and statistical language predictions based on cognitive-behavioral frameworks programmed by developers to target anxiety and depression keywords.Software cannot 'provide interventions' as a clinician does. Mechanistically, it triggers predefined logic trees or predicted text sequences when classifiers detect specific vocabulary in the user's input, entirely devoid of clinical understanding.The clinical and engineering teams at Woebot and Wysa mapped out the dialogue trees and trained the sentiment classifiers, while corporate executives deployed them for public use.
LLMs have the tendency to produce inaccurate or unsuitable answers, especially when they hallucinate.Language models frequently generate statistically plausible but factually incorrect text because their architecture predicts tokens based purely on statistical correlation, not empirical truth.Models do not experience psychological 'hallucinations' because they have no conscious perception of reality. Mechanistically, every output is a probabilistic prediction; 'errors' occur when the most statistically probable token sequence in the vector space misaligns with external facts.Developers at tech companies deliberately deployed architectures incapable of factual grounding, and management released them to the public despite knowing this fundamental architectural limitation.
The majority of them act on short-term communications without having a systemic knowledge on the emotional background of users.Most conversational applications are designed with limited context windows, meaning they process only recent text inputs and do not retrieve historical user data to condition their responses.Systems do not 'have knowledge' of emotional backgrounds. Mechanistically, they require previous conversation tokens to be appended to the current prompt to calculate attention weights across the entire sequence. Without this data in the context window, it cannot statistically condition its output.Product architects and engineering teams deliberately constrain context windows and database retrieval mechanisms to minimize compute costs and optimize response latency.
Wysa uses textual inputs to identify the mood and sentiment of the user and suggests guided self-help exercisesThe Wysa application categorizes user text strings into predefined emotional categories using a sentiment classifier, which then automatically triggers the display of corresponding self-help text templates.The software does not possess the capacity to 'identify' human emotion. Mechanistically, it maps the mathematical embeddings of the input text against the decision boundaries established during its supervised training, triggering conditional logic when thresholds are met.The developers at Wysa designed the classification boundaries, and their clinical consultants determined which pre-written self-help scripts would be triggered by which mathematical thresholds.
their replies are usually pattern-based and not based on the actual affective knowledge.The outputs generated by these models are driven entirely by statistical pattern-matching and correlation, as computational architectures cannot possess subjective experience or emotional understanding.Computational systems can never possess 'actual affective knowledge'. Mechanistically, regardless of architecture complexity, they manipulate numerical matrices based on algorithms. Emotion is a conscious, biological phenomenon that cannot be encoded as a vector.N/A - describes computational processes without displacing responsibility.
It is important to ensure that these systems are secure, emotionally intelligent and ethically groundedIt is critical that developers implement strict safety guardrails, configure reward functions to output supportive language, and ensure the deployment of these tools adheres to rigorous clinical and ethical standards.Software cannot be 'emotionally intelligent' or 'ethically grounded', as it lacks consciousness, empathy, and moral reasoning. Mechanistically, safety relies entirely on human-defined hardcoded filters, adversarial testing, and alignment algorithms that penalize unsafe statistical outputs.Health regulators, corporate policymakers, and engineering teams must take responsibility for defining the safety protocols, filtering the training data, and legally accounting for the societal impacts of these platforms.

Task 5: Critical Observations - Structural Patterns​

Agency Slippage​

The text demonstrates a profound and systematic agency slippage, characterized by a rapid, cyclical oscillation between mechanistic ('how') and agential ('why') framings. This mechanism of oscillation is not random; it is highly strategic. The text frequently establishes its academic and empirical authority by deploying strict theoretical and functional explanations of mechanisms—such as the mathematical formulas of the self-attention mechanism or the pipeline architectures of CNN/RNN multimodal encoders. However, almost immediately following these dense technical foundations, the language slips abruptly into agential framing, claiming that these exact mathematical processes enable the system to 'perform strong contextual reasoning' or 'generate responses that are sensitive'.

This slippage flows heavily in the mechanical-to-agential direction, utilizing the mechanism as a trojan horse for the anthropomorphic claim. The text first establishes the AI as a complex processor, but rapidly upgrades it to a 'knower' and 'reasoner'. This pattern is heavily driven by the 'curse of knowledge'. The authors—who possess deep clinical and technical understanding of what the systems are designed to simulate—project their own conscious intent onto the software. When the system outputs a string of text that correlates with clinical empathy, the authors retroactively attribute 'empathy' to the computational process that generated it.

Simultaneously, as agency flows TO the AI, it is systematically stripped FROM the human actors. The text relies heavily on passive voice and agentless constructions when discussing development and deployment ('models can be taught', 'bias introduced', 'training data can be biased'). The engineers who tuned the weights, the poorly paid annotators who labeled the EmpatheticDialogues dataset, and the corporate entities (like Woebot and Wysa) profiting from these systems are consistently hidden behind the active verbs attributed to the 'AI'. The slippage reaches its peak in the limitations and future prospects sections. When discussing failures, the AI is framed as a conscious agent making a mistake (it 'hallucinates' or demonstrates 'feigned empathy'). When discussing the future, the text demands that the 'systems' themselves become 'emotionally intelligent and ethically grounded'. The rhetorical accomplishment of this slippage is profound: it makes it sayable that an algorithm is an active clinician capable of moral reasoning, while rendering unsayable the massive corporate, financial, and human labor infrastructure dictating every facet of the machine's behavior.

Metaphor-Driven Trust Inflation​

The text actively constructs a dangerous architecture of authority through metaphors that encourage relation-based trust in statistical systems. By systematically employing consciousness language—describing models as 'sensitive', 'attentive', capable of 'identifying mood', and being 'emotionally intelligent'—the discourse signals to the audience that the AI possesses the requisite subjective awareness to form a therapeutic alliance. Claiming an AI 'understands' or 'knows' a patient's context accomplishes something fundamentally different than stating it 'predicts' text based on that context. 'Predicts' establishes a boundary of performance-based trust, where the system is judged merely on its reliability as a tool. In contrast, 'knows' and 'identifies' demand relation-based trust, implying the system acts with sincerity, empathy, and genuine interpersonal vulnerability.

The application of human-trust frameworks to LLMs in mental health is uniquely hazardous. In human relationships, trust is predicated on shared vulnerability, moral accountability, and the capability of the trusted entity to consciously care about the outcome. The anthropomorphic metaphors in the text inappropriately transfer these expectations onto matrix multiplications. The illusion of mind constructed by terms like 'cognitive-behavioral interventions' and 'empathetic dialogue' masks the reality that the system is entirely incapable of reciprocating vulnerability.

This metaphor-driven trust is particularly evident in how the text manages system failures. When the model outputs dangerous or nonsensical text, it is not framed as a fundamental failure of a rigid statistical paradigm; instead, it is agentially framed as a 'hallucination' or 'feigned empathy'. This reason-based, intentional framing treats the error as a temporary psychological lapse of an otherwise competent entity, paradoxically reinforcing the system's overall perceived cognitive authority. The stakes of this discourse are existential in a clinical setting. By encouraging relation-based trust, the text primes vulnerable users in acute distress to rely on an unfeeling, unpredictable statistical engine for emotional salvation, risking severe psychological harm when the illusion shatters, and drastically undermining the necessity of human clinical connection.

Obscured Mechanics​

The anthropomorphic and consciousness-attributing metaphors deployed throughout the text act as a dense linguistic fog, concealing the technical, material, economic, and labor realities of AI production. When the text claims that systems like 'Woebot' or 'Wysa' actively 'identify mood' and 'provide interventions', it comprehensively fails the 'name the corporation' test. It obscures the reality that Woebot Health and Wysa Ltd. are for-profit healthcare technology companies driven by commercial objectives and venture capital mandates, not autonomous, altruistic digital clinicians.

Technically, the use of epistemic verbs like 'knows', 'understands', and 'reasons' completely obscures the statistical nature of the technology. It hides the model's total reliance on training data distributions, the absence of any ground truth, and the absolute lack of causal modeling. When a system is said to 'understand' suicidal ideation, it conceals the fact that it is merely triggering proximity alerts in a high-dimensional vector space based on culturally biased, historically scraped data. Furthermore, the text frequently discusses 'training' and 'reward models' as if they are abstract, mathematical inevitabilities, completely masking the proprietary opacity of these black boxes. The text rarely acknowledges that researchers have no access to the proprietary data mixtures or fine-tuning exactitudes used by the companies providing these foundational models.

Materially and socially, this language completely erases the immense human labor required to build the illusion of an 'empathetic' AI. The text refers to 'reinforcement learning based on human feedback (RLHF)' to 'improve safety', but the metaphor of the AI 'learning' hides the precarious, often traumatized ghost-workers in the Global South who must spend hours reading toxic and suicidal text to provide that feedback. Their labor is vanished into the machine's supposed 'emotional intelligence'. By replacing the consciousness metaphors with mechanistic language—such as stating 'corporate engineers optimize models using underpaid human labelers to generate text that correlates with empathy'—the profit motives, the exploitation, and the brittle statistical reality become instantly visible. The current framing overwhelmingly benefits the tech industry, allowing them to market cheap, automated statistical tools as premium, 'intelligent' medical care while deflecting liability for systemic failures.

Context Sensitivity​

The distribution and intensity of anthropomorphic language in this text are not uniform; they are highly context-sensitive and strategically deployed to maximize the paper's rhetorical authority. In the 'Preliminaries of LLMs' (Section 2), the metaphor density is relatively low, and the language is predominantly mechanistic and theoretical. The author grounds the text in the rigid mathematics of Transformer architecture, utilizing terms like 'autoregressive', 'embedding matrix', and 'vector dimensions'. This establishes the text's academic credibility and empirical rigor. However, once this technical foundation is laid, the text leverages it as a license for aggressive anthropomorphism in the applications and discussion sections (Sections 3 and 4).

As the text shifts to describing clinical use cases, 'matrix multiplication' rapidly intensifies into 'contextual reasoning', and 'token prediction' transforms into 'providing cognitive-behavioral interventions'. This register shift reveals a profound asymmetry in how capabilities and limitations are framed. When discussing the system's capabilities, the text relies heavily on agential, consciousness-attributing language: the AI 'identifies', 'understands', and 'acts'. It is framed as an autonomous superhero of scale. Conversely, when discussing limitations or errors, the language frequently reverts to mechanical, passive, or structural terms: 'training data can be biased', 'discrete emotion labels result in oversimplified models', or it adopts psychological pathologization like 'hallucination' to insulate the core architecture from critique.

This contextual sensitivity accomplishes a distinct strategic function: vision-setting and marketing for the integration of AI into healthcare. By shifting from 'X operates statistically like Y' (acknowledged metaphor) to 'X understands Y' (literalized capability), the text positions itself not just as a technical review, but as a normative advocacy piece for digital health ecosystems. The anthropomorphism deepens for non-technical claims (like empathy and therapy) to resonate with a lay and clinical audience, ensuring that the sheer algorithmic reality doesn't alienate those who desire human-like care. This pattern reveals that the implied rhetorical goal is to domesticate and legitimize LLMs for medical integration, using technical jargon to bypass scientific skepticism and emotional anthropomorphism to secure clinical and public buy-in.

Accountability Synthesis​

Accountability Architecture

This section synthesizes the accountability analyses from Task 1, mapping the text's "accountability architecture"—who is named, who is hidden, and who benefits from obscured agency.

The metaphors, agency slippages, and agentless constructions analyzed throughout this text synthesize into a robust architecture of displaced responsibility. When audiences read that an AI 'hallucinates', 'learns', or 'identifies mood', they systematically internalize the illusion that the algorithm is an autonomous entity capable of making independent choices. This linguistic framework creates a massive 'accountability sink' where the profound moral, clinical, and legal responsibilities of human decision-makers disappear into the abstraction of the technology.

Across the audited passages, human actors—researchers, data annotators, corporate executives, and policymakers—are almost universally relegated to the 'Hidden' or 'Partial' visibility categories. The decisions to scrape biased data, to deploy fragile predictive models in life-or-death clinical triage, and to constrain context windows for financial efficiency are presented not as deliberate human choices, but as inevitable technological evolutions or inherent 'limitations' of the AI itself. By utilizing the passive voice ('models are trained') and framing the software as the sole grammatical actor ('Wysa uses', 'LLMs produce'), the text effectively shields the tech industry and healthcare administrators from liability. If a patient dies by suicide because the LLM failed to 'reason' correctly, the text's framing suggests the technology simply 'hallucinated' or suffered from 'feigned empathy', diffusing the blame into the black box rather than pinning it on the executives who greenlit the deployment of an ungrounded statistical engine.

Naming the human actors would radically alter the discourse and the liability implications. If 'The algorithm showed bias' is reframed as 'Engineers at [Company] selected a training dataset that encoded structural biases, and executives deployed it without safety audits,' entirely new questions become askable. We can ask: Who signed off on this? What were the financial incentives? Where is the regulatory oversight? The current accountability displacement serves the commercial interests of the tech and healthcare industries, allowing them to market the cost-saving automation of 'empathetic' triage while entirely evading the immense medical malpractice liability that would normally accompany the provision of 'cognitive-behavioral interventions' by unqualified human actors. The anthropomorphic text is, fundamentally, a liability shield.

Conclusion: What This Analysis Reveals​

The Core Finding

This analysis reveals a highly interconnected system of metaphorical projections that systematically construct an 'illusion of mind' within Large Language Models. Three dominant anthropomorphic patterns emerge: the 'Cognitive' pattern (projecting reasoning, understanding, and knowledge onto vector mathematics), the 'Clinical' pattern (projecting therapeutic agency and diagnostic intention onto statistical classifiers), and the 'Emotional/Moral' pattern (projecting empathy, sensitivity, and ethical grounding onto probabilistic text generation). These patterns are deeply symbiotic.

The cognitive projection is the foundational, load-bearing architecture. For a system to be perceived as an active clinical agent capable of empathetic intervention, the audience must first accept the foundational premise that the system 'knows' what it is doing and 'understands' the context. By shifting the verbs from mechanistic realities (processes, computes, correlates) to epistemic and consciousness claims (learns, reasons, identifies), the text builds a baseline assumption of conscious awareness. Once this cognitive baseline is established, the subsequent projections of clinical authority and moral intelligence appear as logical next steps rather than absurd category errors. If the cognitive pattern collapses—if readers recognize the system merely calculates probabilities without any semantic comprehension or justified belief—the entire edifice of the AI as a 'sensitive therapist' shatters, revealing a highly sophisticated but unfeeling parrot. The complexity of this analogical structure lies in its seamless mapping of deep human relational capacities onto opaque statistical black boxes.

Mechanism of the Illusion:​

The 'illusion of mind' is meticulously constructed through a temporal and causal chain of rhetorical sleights-of-hand. The central trick relies on exploiting the 'curse of knowledge' and weaponizing technical opacity. The text invariably begins by grounding the reader in complex, unassailable mathematics—such as Transformer attention equations and neural network architectures. This establishes profound technical authority and intimidates non-expert readers. Once the audience is primed to view the system as highly advanced, the author subtly introduces the consciousness projection pattern, slipping from 'synthesizes tokens' to 'reasons contextually'.

This shift exploits a deep human psychological vulnerability: our evolutionary hardwiring to attribute consciousness to anything that convincingly mimics language and responsiveness. When the author—who conceptually understands the complex math—projects human intent onto the output, the lay audience eagerly accepts it. They want a solution to the global mental health crisis, making them uniquely susceptible to the illusion of a scalable, empathetic machine. The causal chain is clear: algorithmic complexity is equated with intelligence, intelligence is conflated with understanding, and understanding is anthropomorphized into empathy. Furthermore, the explanation types identified via Brown’s typology amplify this illusion. By mixing Functional explanations of 'how' with Intentional and Reason-based explanations of 'why', the text validates the AI's simulated empathy as genuine intention. This crude but effective anthropomorphism convinces the audience that the statistical ghost in the machine is a conscious, caring mind.

Material Stakes:​

Categories: Regulatory/Legal, Institutional, Epistemic

The framings identified in this text generate severe, tangible consequences across multiple societal domains. In the Regulatory and Legal sphere, attributing epistemic knowing ('identifies mood') and moral agency ('ethically grounded') to an LLM creates catastrophic liability ambiguity. If regulators believe the AI actively 'reasons' and provides 'interventions', they may draft policies treating the software as an autonomous medical device or practitioner. Consequently, when the system inevitably hallucinates a dangerous recommendation to a suicidal patient, the metaphorical framing protects the corporate developers, allowing them to dismiss the harm as an unpredictable 'glitch' of a complex mind rather than a negligent release of a statistically fragile product.

Institutionally, the clinical framing ('providing CBT') drives dangerous economic behavior in global health systems. Healthcare administrators, facing massive funding gaps, are heavily incentivized to adopt these systems. The anthropomorphic language provides institutional cover; administrators can claim they are deploying 'emotionally intelligent' triage rather than executing draconian cost-cutting measures that replace human professionals with cheap pattern-matchers. The losers here are the most vulnerable patients in the 'low- and middle-income nations' explicitly targeted in the paper's abstract, who receive a simulacrum of care while tech companies profit.

Epistemically, the constant conflation of 'processing' with 'knowing' degrades the very concept of human understanding. When computational token prediction is accepted as 'actual affective knowledge', society begins to devalue genuine human empathy and clinical expertise. If metaphors were removed and precision restored, the sheer inadequacy of these systems for solo medical intervention would become glaringly obvious, threatening the massive venture capital valuations built on the promise of autonomous AI therapists.

AI Literacy as Counter-Practice:​

Critical literacy and mechanistic precision act as direct counter-practices to the dangerous material stakes of AI anthropomorphism. By systematically replacing consciousness verbs (knows, understands, reasons) with mechanistic verbs (processes, correlates, predicts), as demonstrated in the Task 4 reframings, we force a radical recognition of the technology's limitations. Changing 'the model understands intent' to 'the model classifies tokens correlating with training examples' immediately shatters the illusion of mind. It forces clinicians and regulators to confront the absence of awareness, the absolute dependency on historically biased data, and the fragile statistical nature of the outputs.

Furthermore, restoring human agency by refusing agentless constructions ('the AI discriminated' becomes 'Engineers at Company X deployed a biased system') directly counters liability evasion. Naming the actors forces recognition of the power dynamics, profit motives, and human design choices that dictate systemic behavior. For this literacy to be adopted systematically, a paradigm shift is required in academic publishing and tech journalism. Journals must mandate mechanistic translations of anthropomorphic claims, and researchers must commit to explicitly separating computational processing from cognitive knowing.

Unsurprisingly, this precision faces immense resistance. The tech industry, venture capitalists, and efficiency-driven healthcare administrators deeply benefit from anthropomorphic language. It allows them to market predictive text as premium intelligence, securing funding, evading regulation, and driving adoption. Critical literacy threatens these interests by exposing the man behind the curtain, reducing a purportedly magical, autonomous mind back into a highly sophisticated, human-directed, and legally accountable tool.

Path Forward​

The broader discursive ecology surrounding AI is fractured among different communities, each prioritizing different vocabularies with distinct trade-offs. The tech industry and marketing communities heavily favor 'anthropomorphic clarity' ('AI thinks', 'AI knows'), which maximizes intuitive public grasp and narrative resonance, driving investment and adoption. However, this approach costs precision, fosters severe automation bias, and creates dangerous regulatory blind spots. Conversely, critical academics and engineers advocate for 'mechanistic precision' ('processes embeddings', 'predicts tokens'). This vocabulary ensures testability, strips away unwarranted trust, and grounds expectations in reality, but it costs accessibility, often alienating lay audiences and policymakers with impenetrable mathematical jargon.

To navigate these trade-offs, structural changes are necessary. Academic journals and funding bodies could require researchers to provide 'capability disclosures'—dual-language abstracts that articulate findings in both mechanistic terms (for rigorous peer review) and carefully hedged analogical terms (for public accessibility). Regulatory frameworks could mandate transparency about the discourse approach, forcing companies to explicitly disclaim the statistical nature of their tools in consumer-facing medical apps.

Mapping these choices reveals several potential futures. If mechanistic precision becomes the norm, we solve the liability evasion problem and drastically reduce automation bias in clinical settings, though public adoption of AI tools might slow as the 'magic' fades. If anthropomorphic language continues to deepen, the assumption that AI is an autonomous, knowing agent will become embedded in law and culture, making the deployment of 'AI therapists' ubiquitous but risking massive, unaccountable psychological harm when the statistical engines inevitably fail to comprehend true human crisis. If the current confusion is maintained, the tech industry will continue to exploit the ambiguity, oscillating between claiming AI is a 'conscious mind' when selling it, and a 'mere calculator' when sued for its failures. Ultimately, the vocabulary we institutionalize will not just describe the technology; it will legally and culturally define the boundaries of human agency.

Extended Processing Summary​

Gemini's Processing Steps
What you are about to read…

Gemini refers to the text below as "thought summaries." This is an overt consciousness projection because 'intentions' are hallmarks of a conscious mind that 'knows' what it is doing and why. The concealed mechanistic process is probabilistic text generation. Treat this as just another rhetorical artifact—a way of making the model's processing legible. When the model writes "I should double check that...", it is mathematically altering its own probability distribution for the next token to be more accurate. The "I" is a functional gear in the machine, not a self-reference. This language is a deliberate interface design choice, not a biological reality. The developers could have programmed the model to output passive logs. Instead, they chose to simulate a narrator.

Analyzing the Request

I'm now zeroing in on the core of this task. It involves a critical discourse analysis of a text related to Large Language Models (LLMs) in mental health. My focus will be on the prevalent use of AI metaphors, anthropomorphism, and the projection of consciousness onto these models. This is where I begin my deep dive.

Defining the Analysis Structure

I've solidified the structural components of the analysis. My approach will now follow a detailed, structured format for the output, in a JSON schema. The key elements are a metaphor audit, a source/target mapping, an explanation audit, reframing of language to restore human agency, critical observations, and a concluding synthesis. These will all be in a consistent and ordered structure.

Assessing Structural Rigor

I've just finalized the structural checks and validated the JSON schema adherence. My confidence is high, scoring a perfect 5/5 on the constraint checklist. Now, I'm diving into the source text, a review paper on "Empathetic Dialogue Systems for Mental Health Support Using Large Language Models." I'm identifying key quotes and themes, paying particular attention to agency given to the model, anthropomorphism, and the potential for biased language.

Identifying Key Metaphors

I've been going through the text, and I'm cataloging cognitive metaphors such as "capture long-range dependencies" and "contextual reasoning." I'm also finding instances of learning metaphors, like "the model learns...". The text seems rife with anthropomorphism, as evident in phrases like "interpreting emotions," and even attributing psychological states to models via phrases like "when they hallucinate." These metaphors create a very interesting challenge for interpretation.

Cataloging Metaphors - Progress

I'm now in the thick of the metaphor audit, and it's yielding some intriguing results. I've logged four of the eight examples to date, focusing on phrases like "sensitive to the users and attentive to their emotional condition," "the model learns general linguistic regularities," "perform strong contextual reasoning," and "identify the mood and sentiment." These all clearly map human traits onto algorithmic processes. It's a rich field for analysis.

Mapping Source-Target Pairs

I'm now deep in the source-target mapping stage, expanding on the metaphor audit. I have four more logged metaphors to integrate. I'm focusing on "the tendency to hallucinate," which frames errors as psychological delusions, and phrases portraying database knowledge as "systemic knowledge." I'm also addressing how the lack of "affective knowledge" is presented, and how software is framed as an emotionally intelligent moral agent.

Expanding the Metaphor Audit

I'm now integrating the additional metaphor mappings, focusing on the core sources and targets. My focus is on the RLHF process being framed as human learning, attention mechanism as human reasoning, and sentiment classification algorithms as diagnostic clinicians. I've also just completed the first item in the Explanation Audit, describing the mechanisms of an attention function with the associated metaphor types.

Integrating New Metaphors

I've just added four more entries to the metaphor analysis. Specifically, I've incorporated the "learns linguistic regularities" metaphor, classified as Genetic and Intentional. Additionally, I've integrated "identify the mood and sentiment," categorized as Functional and Reason-based, and "not based on actual affective knowledge," marked as Empirical Generalization and Dispositional. Lastly, I've included "hierarchical attention systems select," defined as Functional and Empirical Generalization. I am now proceeding with the explanation audit.

Revising for Clarity

I'm now rewriting the eight metaphor-laden quotes to remove all agency from the LLMs. The goal is to highlight who is truly responsible for the language and actions attributed to these models. For instance, I'm transforming a phrase about a model "learning" to one that specifies the developers' role. Following this, I will write the critical observations.

\n\nI'm now delving into the critical observations. I'm focusing on agency slippage, the dangers of metaphor-driven trust, obscuring the mechanisms, and the context shift within the source material. Each section will be in the 400-500 word range, deeply analyzing the implications of the language choices. The core goal is to uncover the biases and hidden assumptions at work in the text.

Analyzing Accountability Structures

I'm now focusing on analyzing the accountability structures within the text, with the key example of attributing blame to the AI for failures using language like "hallucinate". It obscures developers' responsibilities and shifts attention away from design choices. The unimodal input is blamed, while it's their choice to use it. Now I'll proceed with further analysis.

Refining Observation Categories

I'm now deep in the process of refining my categorical justifications, paying close attention to nuanced classifications. My goal is to prevent uniform categorization; for instance, the math section is partially hedged, while the clinical section completely obscures agency. I'm also ensuring actorVisibility includes options like Named, Partial, and Hidden to accurately reflect genuine variations.

Categorizing Metaphors

I'm now cataloging new metaphors. I've categorized "model to perform strong contextual reasoning" as a cognitive deduction projection with a hidden actor. I've identified "models that apply CBT" as a mapping of clinical agency to a hidden actor (Woebot developers). Additionally, I'm noting "fake empathy... actual affective knowledge" which projects a possibility with a hidden actor. Finally, I'm analyzing "when they hallucinate" and "identifies the mood and sentiment...suggests guided self-help".

Finalizing Observation Details

I'm now integrating the final two metaphors: "systemic knowledge on the emotional background," and "emotionally intelligent and ethically grounded." These are projections of epistemic and moral/emotional consciousness, respectively. I'm focusing on the actor visibility and agency. Now, I'm ensuring rigorous adherence to the Robert Brown typology and Lakoff/Johnson frameworks. I'm currently reviewing all instructions.

Adapting Output Strategy

I've adapted my approach to strictly avoid any control tokens like newlines or tabs, producing a completely minified JSON output. Text fields won't include explicit newline characters. I'll maintain rigorous adherence to word count requirements by thoroughly elaborating on theoretical mechanisms and accountability dynamics.

Analysis Provenance

Run ID: 2026-07-03-a-comprehensive-investigation-of-empathe-metaphor-lbydpc Raw JSON: 2026-07-03-a-comprehensive-investigation-of-empathe-metaphor-lbydpc.json Framework: Metaphor Analysis v6.5 Schema Version: 3.0 Generated: 2026-07-03T09:41:39.806Z

Discourse Depot © 2025 by TD is licensed under CC BY-NC-SA 4.0