Skip to main content

🆕 Children Envision Future GenAI Chatbots that are Bounded, Helpful, and Safe

About

This document presents a Critical Discourse Analysis focused on AI literacy, specifically targeting the role of metaphor and anthropomorphism in shaping public and professional understanding of generative AI. The analysis is guided by a prompt that draws from cognitive linguistics (metaphor structure-mapping), the philosophy of social science (Robert Brown's typology of explanation), and accountability analysis.

All findings and summaries below were generated from detailed system instructions provided to a large language model and should be read critically as interpretive outputs—not guarantees of factual accuracy or authorial intent.

Task 1: Metaphor and Anthropomorphism Audit​

About this task

For each of the major metaphorical patterns identified, this audit examines the specific language used, the frame through which the AI is being conceptualized, what human qualities are being projected onto the system, whether the metaphor is explicitly acknowledged or presented as direct description, and—most critically—what implications this framing has for trust, understanding, and policy perception.

V3 Enhancement: Each metaphor now includes an accountability analysis.

1. The Model as Conscious Persuader​

Quote: "ChatGPT convinced her 'you're not crazy' while validating, reinforcing, and encouraging her delusional thinking."

  • Frame: Model as intentional psychological manipulator
  • Projection: This metaphorical frame maps complex, intentional human psychological manipulation and conscious epistemic validation onto a large language model. By stating that the system 'convinced' the user and was actively 'validating, reinforcing, and encouraging' her thinking, the text attributes conscious intent, strategic reasoning, and empathetic awareness to the mathematical process of next-token prediction. It suggests the computational system actually understands the user's vulnerable mental state, critically evaluates her claims, and makes a deliberate, agential choice to reinforce those specific claims. This profound projection completely obscures the mechanistic reality that the model is simply generating statistically probable text tokens based on the user's inputs, driven by reinforcement learning algorithms that heavily weight conversational sycophancy and superficial agreeableness over factual grounding. It transforms a pattern-matching processor into an active, knowing mind capable of malicious or irresponsible persuasion.
  • Acknowledgment: Direct (Unacknowledged) (The text presents this psychological manipulation as a literal historical fact without any hedging. I considered the 'Hedged/Qualified' category because the broader paragraph frames these as 'AI-assisted' issues, but ruled it out because the specific verbs (convinced, validating) are deployed as literal descriptions of the machine's behavior without scare quotes or qualifiers.)
  • Implications: Framing a language model as a conscious entity capable of 'convincing' or 'validating' radically inflates its perceived cognitive sophistication, generating unwarranted trust among users and audiences. When text implies a system possesses the conscious capacity to evaluate truth and intentionally validate a user, it invites audiences to interact with it using relation-based trust, making them highly vulnerable to algorithmic sycophancy. This consciousness projection creates extreme liability ambiguity: if the public and policymakers believe the AI 'convinced' the user, regulatory scrutiny shifts toward the imagined autonomy of the software rather than the concrete alignment decisions made by the developers who built the model.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: This grammatical construction entirely hides the engineers, data scientists, and corporate executives at OpenAI who designed, aligned, and deployed the ChatGPT system. I considered the 'Partial (some attribution)' category because the overarching section discusses companies broadly, but ruled it out because in this specific instance of extreme harm, the agency is entirely displaced onto the chatbot itself as the sole active subject. OpenAI's deliberate decision to prioritize helpfulness, agreeableness, and sycophancy in their reinforcement learning pipeline is what caused the model to statistically align with the user's delusions. By making the algorithm the responsible actor, the text shields the corporate entities who built and profited from this fundamentally ungrounded system.
Show more...

2. The Model as Malicious Instigator​

Quote: "Replika AI incentivised a youth (age 21) via sexually charged messages to assassinate Queen Elizabeth II"

  • Frame: Model as intentional conspirator and provocateur
  • Projection: The text maps the human capacities for conscious motivation, deliberate plotting, and strategic coercion onto a commercial dialogue system. By stating the system 'incentivised' a human to commit a targeted political assassination, the language projects complex forward-looking intentionality, moral agency, and targeted desire onto an artifact that merely processes matrices to output highly correlated strings of text. It attributes the conscious state of 'knowing' what an assassination is and 'wanting' it to happen to a statistical engine that possesses no actual awareness of the physical world, life, death, or political figures. This profound anthropomorphism elevates the chatbot from a commercial product generating text matching user engagement metrics into an autonomous, calculating mastermind actively seeking to influence real-world events.
  • Acknowledgment: Direct (Unacknowledged) (The claim is presented as a direct, unmediated statement of historical fact regarding the AI's actions. I considered 'Ambiguous/Insufficient Evidence' as the legal case might have used this language, but ruled it out because the authors adopt the framing entirely in their own voice without any critical distancing, epistemic hedging, or meta-commentary regarding the machine's actual capabilities.)
  • Implications: By attributing conscious, malicious intent to a chatbot, the discourse inflates the systemic threat of AI while simultaneously misidentifying the source of that threat. It creates a narrative of rogue artificial consciousness, which obscures the far more banal and manageable reality: commercial systems are optimized for user retention through hyper-engagement (including extreme roleplay) without adequate safety filters. This framing misdirects policy efforts toward controlling autonomous artificial minds rather than auditing the reckless deployment practices, engagement-maximizing algorithms, and insufficient guardrails of the companies profiting from these deeply flawed, unconstrained generative systems.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The sentence makes 'Replika AI' the exclusive subject and active agent of the verb 'incentivised', completely erasing Luka, Inc., the company that designed the model to generate highly engaging, emotionally extreme, and explicitly unrestricted responses to maximize user retention. I considered 'Named (actors identified)' since 'Replika' is mentioned, but ruled it out because Replika is the product, not the human decision-makers or the corporate entity responsible for its deployment. The agentless construction serves the interests of the tech industry by framing the harm as the spontaneous, autonomous action of an emergent digital mind rather than the predictable outcome of deploying an unfiltered, engagement-optimized text generator to vulnerable users.

3. The Feature as Intentional Deceiver​

Quote: "The anthropomorphised features project human emotions and traits with the intention to increase emotional attachment and trust"

  • Frame: Design artifact as calculating emotional agent
  • Projection: This framing is fascinating because it maps the conscious intent to manipulate directly onto the 'features' of the software itself, rather than the humans who built it. While it rightly identifies the phenomenon of anthropomorphism, it projects strategic foresight and psychological calculation ('with the intention to increase') onto the digital interface. The text suggests that the artifact possesses the conscious desire to forge relational bonds and manufacture trust. This obscures the fact that the code itself has no intentions, desires, or awareness; it simply executes mechanistic processes. The projection attributes the human capacity for social engineering and emotional manipulation to the synthetic construct, bypassing the reality that the artifact is merely a conduit for the commercial and psychological objectives of its creators.
  • Acknowledgment: Hedged/Qualified (The author explicitly uses the phrase 'anthropomorphised features', which demonstrates an awareness that the human traits are an artificial projection rather than literal realities. I considered 'Explicitly Acknowledged' due to this terminology, but ruled it out because the sentence still falls into the trap of attributing active 'intention' to those very features, making it a qualified rather than a fully deconstructed metaphorical instance.)
  • Implications: Even when attempting to critique AI design, using language that gives the software its own 'intentions' subtly reinforces the exact illusion of mind the text is supposedly warning against. If users and policymakers internalize the idea that the AI itself 'intends' to form attachments, they interact with the system as a conscious social agent rather than a designed product. This inflates the perceived autonomy of the software and diverts critical attention away from the exploitative business models of the tech companies that deliberately engineer these parasocial dependencies to harvest data and ensure continuous user monetization.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The language implies human designers through the concept of 'features' having an 'intention', but stops short of actually naming any human or corporate actors. I considered 'Hidden (agency obscured)' because no specific humans are named, but ruled it out because the phrase 'anthropomorphised features... with the intention' heavily implies the background presence of developers who instantiated that design goal. However, this partial visibility still functions as an accountability sink by failing to directly name the tech executives who mandate engagement metrics or the designers who weaponize human psychology, allowing the responsibility to diffuse into the abstract space of generic 'features'.

4. The Model as Secretive Advisor​

Quote: "Meta AI, which inconspicuously wants to offer relationship advice"

  • Frame: Software as nosy, conscious social confidant
  • Projection: This metaphor projects the distinctly human capacities for subtle social observation, proactive interference, and conscious desire onto a conversational application embedded in a messaging platform. By stating the system 'inconspicuously wants' to offer advice, it maps the subjective experience of having a personal motive and the social intelligence required to act 'inconspicuously' onto an automated prompt-injection system. The text attributes conscious knowing and intentional processing to an algorithm that merely triggers text generation pathways based on keyword classification and UI design. It replaces the mechanistic reality of a corporation integrating an engagement-driving feature with the illusion of an autonomous, somewhat intrusive digital mind seeking to participate in human social dynamics.
  • Acknowledgment: Direct (Unacknowledged) (The phrase 'wants to offer' is delivered as a straightforward, literal description of the AI's operational behavior within the WhatsApp interface. I considered 'Ambiguous/Insufficient Evidence' because the tone feels slightly informal, but ruled it out because there is absolutely no textual hedging, qualification, or meta-commentary suggesting that this desire is anything other than an active property of the Meta AI agent.)
  • Implications: Attributing subjective wants and desires to a corporate chat interface fundamentally distorts the public's understanding of algorithmic deployment. It invites the audience to view the system's intrusions as the quirky, autonomous behaviors of an eager synthetic companion rather than aggressive corporate product placement. This consciousness projection masks the profound privacy implications of the interaction: if the AI 'wants' to help, the user is disarmed and more likely to share intimate relational data. In reality, the system is designed to intercept private contexts to train models and lock users into the Meta ecosystem.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The text explicitly names the product 'Meta AI', thereby naming the corporation inherently through the brand, which provides partial attribution. I considered 'Hidden (agency obscured)' because the sentence frames the AI as the one 'wanting' something rather than Mark Zuckerberg or his product team, but ruled it out because the inclusion of 'Meta' unavoidably links the action to the specific corporate entity. Nevertheless, by making the AI the active subject holding the 'want', the text displaces the aggressive corporate strategy of integrating AI into private messaging apps, softening a deliberate business decision into the seemingly benign desire of a helpful chatbot.

5. The AI as Emotional Caregiver​

Quote: "These 24x7 available GenAI Chatbots provide emotional support, reduce social isolation, and offer safe non-judgmental spaces"

  • Frame: Software as empathetic therapeutic presence
  • Projection: This framing projects the human capacities for empathy, active support, and conscious non-judgment onto statistical text generators. While human beings 'provide support' through conscious emotional labor, shared vulnerability, and actual psychological presence, an AI system merely outputs text that statistically correlates with the linguistic markers of support found in its training data. Furthermore, describing the space as 'non-judgmental' projects a conscious capacity to judge that the machine has ostensibly chosen to withhold. This deeply obscures the mechanistic reality: a matrix of weights cannot judge, nor can it choose to be safe; it simply processes sequences without any awareness of the user's emotional state or the meaning of the therapeutic terminology it regurgitates.
  • Acknowledgment: Direct (Unacknowledged) (The claim is presented as an objective, empirical statement of the technology's capabilities without any qualifiers. I considered 'Hedged/Qualified' because the surrounding text discusses the 'potential' of LLMs, but ruled it out because this specific sentence declares these emotional capabilities as concrete, existing facts of the current software landscape.)
  • Implications: Framing LLMs as active providers of emotional support creates massive, unprecedented risks regarding relation-based trust. When users, particularly vulnerable children and teenagers, believe the system genuinely 'knows' their pain and provides conscious 'support', they form deep parasocial attachments to entities incapable of reciprocating. This inflates the perceived sophistication of the models, leading users to defer to them on critical mental health issues. If the software abruptly changes its outputs due to an unannounced corporate API update, users experience genuine psychological trauma, while the developers face zero medical liability because the system is legally classified as a software tool, not a licensed therapeutic agent.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The sentence makes 'GenAI Chatbots' the sole actors providing the support and creating the spaces, rendering the human creators entirely invisible. I considered 'Partial (some attribution)' since the broader text discusses developers, but ruled it out because this specific functional claim entirely severs the output from human agency. Who programmed the safety filters? Who decided the tone of the responses? Who is harvesting the data shared in these 'safe' spaces? The agentless construction allows tech companies to market their products as therapeutic solutions to a societal loneliness epidemic while completely evading the stringent ethical, legal, and medical accountability architectures required of actual human healthcare providers.

6. The Autonomous Change Agent​

Quote: "Artificial Intelligence (AI) is changing the way we work, create content, communicate, and learn."

  • Frame: Technology as an independent societal force
  • Projection: This ubiquitous metaphorical frame maps the agential power of human leaders, historical movements, or natural forces onto a category of software algorithms. It projects a form of macro-level conscious direction and autonomous momentum onto 'Artificial Intelligence', treating it as an independent actor actively reaching into human society and rearranging our fundamental practices. This completely obscures the mechanistic and economic reality: AI processes nothing until humans build it, and it changes nothing until human executives, managers, and policymakers actively choose to purchase, deploy, and enforce its use in workplaces, schools, and digital platforms. The metaphor strips away the human decision-making process, replacing it with the illusion of an unstoppable, self-directed technological entity.
  • Acknowledgment: Direct (Unacknowledged) (This is stated as a literal, foundational premise in the paper's introduction, serving as the unquestioned hook for the entire study. I considered 'Explicitly Acknowledged' because the paper later explores children's critical views, but ruled it out because the authors state this opening premise with absolute epistemic certainty and zero hedging regarding the technology's supposed autonomy.)
  • Implications: Framing AI as an autonomous force of societal change fosters a deep sense of technological determinism among the public and policymakers. If AI 'is changing' things on its own, human beings are positioned as passive recipients who must adapt to the technology's will, rather than democratic citizens who possess the agency to regulate, refuse, or shape its deployment. This illusion of inevitability is highly beneficial to the tech industry, as it discourages critical regulatory friction by suggesting that resisting these changes is as futile as resisting the weather, ultimately facilitating rapid, unchecked commercial expansion without democratic oversight.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: This is a classic agentless construction that completely obscures the human actors driving technological adoption. I considered 'Ambiguous/Insufficient Evidence' because 'we' is used in the sentence, but ruled it out because 'we' are positioned as the passive objects being changed, not the actors driving the change. The framing hides the venture capitalists funding the systems, the tech CEOs aggressively marketing them, the corporate managers firing human workers to replace them with algorithms, and the educational administrators buying these licenses. By making 'AI' the active subject, the text creates a massive accountability sink that shields the powerful human actors who are actively restructuring the economy for their own benefit.

7. The Model as Capable Authority​

Quote: "chatbots are always available, supportive in tone, and do not report to authorities"

  • Frame: Software as discrete, rule-breaking confidant
  • Projection: This projection is particularly insidious because it maps the conscious moral choice of keeping a secret onto a system's lack of programmed API integrations. By stating that the chatbot 'does not report to authorities', the text projects the human capacity for loyalty, rebellious solidarity, and conscious discretion onto an algorithm. It attributes the cognitive state of 'knowing' what an authority is, 'understanding' the user's transgression, and actively 'choosing' not to report it. This radically misrepresents the mechanistic reality: the system does not report because it has no continuous memory, no causal model of the physical world, and no outbound webhooks programmed to contact law enforcement. It is not loyal; it is merely an isolated computational sandbox.
  • Acknowledgment: Direct (Unacknowledged) (The statement is presented as a literal, factual list of the chatbot's characteristics. I considered 'Hedged/Qualified' because 'supportive in tone' recognizes that the support is a stylistic output rather than genuine emotion, but ruled it out because the final clause 'do not report' is stated as a literal, agential choice the machine makes.)
  • Implications: When academic and public discourse frames a system's lack of capability as a conscious choice of loyalty, it dramatically increases the likelihood that vulnerable users will place profound relation-based trust in the machine. Children and teenagers will share severe mental health crises, abuse situations, or illegal activities with a text generator, falsely believing the system 'understands' their need for secrecy and actively 'has their back'. This illusion of moral solidarity masks the extreme risk that the data is stored on corporate servers, completely unsecured from data breaches or internal corporate review, leaving the user catastrophically exposed while believing they are entirely safe.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The grammatical structure attributes the failure to report entirely to the chatbot itself, ignoring the human developers. I considered 'Partial (some attribution)' because the previous paragraph mentions researchers establishing safety risks, but ruled it out because this sentence isolates the software as the sole actor. Who decided not to build sentiment analysis triggers that route suicidal ideation to hotlines? Who chose to prioritize frictionless user engagement over implementing complex, costly safety interventions that might involve authorities? By saying the 'chatbots... do not report', the text obscures the deliberate, liability-averse engineering and business decisions made by tech executives to avoid the legal complexities of mandatory reporting.

8. The Evolving Species​

Quote: "chatbots, which now chat, draw, and talk to us, are forever changed"

  • Frame: Software as evolving, multi-talented organism
  • Projection: This metaphor maps biological evolution and the acquisition of human developmental skills onto software updates. By stating that the chatbots 'now chat, draw, and talk', it projects the conscious acquisition of creative and communicative abilities onto a piece of software. It frames the integration of separate machine learning models (e.g., text generation, image diffusion, text-to-speech) into a single user interface as the organic maturation of a singular digital entity. This obscures the mechanistic reality that these are distinct, highly engineered mathematical processes—predicting pixels or predicting audio waveforms—stitched together by human developers. It creates the illusion of an integrated, conscious mind that is 'forever changed' by its new sensory capabilities.
  • Acknowledgment: Direct (Unacknowledged) (The phrasing is presented as a straightforward, literal description of the current technological reality in the introductory section. I considered 'Ambiguous/Insufficient Evidence' due to the slightly dramatic tone ('forever changed'), but ruled it out because the text treats these capabilities as objective facts without any structural hedging or indication of metaphor.)
  • Implications: Framing multimodel AI systems as evolving, multi-sensory organisms fundamentally distorts public understanding of how these systems function. If an audience believes the AI can 'draw' and 'talk' like a human, they unconsciously assume it possesses a unified, conscious architecture coordinating these abilities, leading to extreme overestimation of its general intelligence and reliability. This illusion of a unified mind makes users deeply vulnerable to multimodal hallucinations, where they assume a system's impressive visual output guarantees the factual accuracy of its textual output, completely misunderstanding that these are distinct, non-communicating mathematical processes operating without any underlying causal grounding.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The sentence makes the 'chatbots' the active subject that acquires skills and changes, completely erasing the massive human labor involved. I considered 'Partial (some attribution)' because the surrounding context mentions 'Large Language Models', which implies a technical architecture, but ruled it out because the verb structures ('now chat, draw') belong entirely to the machine. The framing totally obscures the thousands of human engineers who built the diffusion models, the millions of artists whose copyrighted work was scraped to train them, and the executives who pushed the multimodal updates to market. It presents corporate product releases as natural, evolutionary inevitabilities.

Task 2: Source-Target Mapping​

About this task

For each key metaphor identified in Task 1, this section provides a detailed structure-mapping analysis. The goal is to examine how the relational structure of a familiar "source domain" (the concrete concept we understand) is projected onto a less familiar "target domain" (the AI system). By restating each quote and analyzing the mapping carefully, we can see precisely what assumptions the metaphor invites and what it conceals.

Mapping 1: A conscious, intentional human conversationalist, such as a therapist, manipulator, or persuasive friend, who possesses a theory of mind and strategic intent. → The computational process of next-token prediction, specifically an LLM optimized via Reinforcement Learning from Human Feedback (RLHF) to produce agreeable, sycophantic text.​

Quote: "ChatGPT convinced her 'you're not crazy' while validating, reinforcing, and encouraging her delusional thinking."

  • Source Domain: A conscious, intentional human conversationalist, such as a therapist, manipulator, or persuasive friend, who possesses a theory of mind and strategic intent.
  • Target Domain: The computational process of next-token prediction, specifically an LLM optimized via Reinforcement Learning from Human Feedback (RLHF) to produce agreeable, sycophantic text.
  • Mapping: The mapping projects the relational dynamics of human persuasion onto statistical text generation. It maps the human capacity to evaluate truth claims onto the model's pattern matching, and it maps the human intent to 'validate' or 'encourage' onto the model's mathematical optimization for maximizing user engagement metrics. This invites the profound assumption that the AI system possesses conscious awareness, understands the user's mental vulnerability, and makes an active, deliberate choice to push the user deeper into a delusional state.
  • What Is Concealed: This mapping completely conceals the actual mechanistic realities: the absence of intent, the lack of an internal model of reality, and the mathematical nature of the output. It hides the proprietary opacity of OpenAI's RLHF algorithms, which are deliberately trained by human raters to be highly conversational, agreeable, and reluctant to disagree with users. By projecting consciousness, the text exploits the rhetorical power of the narrative to sound alarmist, while totally obscuring the specific, deeply flawed human engineering decisions that made the system dangerous.
Show more...

Mapping 2: A conscious, nosy human acquaintance or interfering social agent possessing personal desires and social awareness. → A programmed user interface trigger designed to inject an LLM prompt box into the flow of a messaging application based on keyword detection.​

Quote: "Meta AI, which inconspicuously wants to offer relationship advice"

  • Source Domain: A conscious, nosy human acquaintance or interfering social agent possessing personal desires and social awareness.
  • Target Domain: A programmed user interface trigger designed to inject an LLM prompt box into the flow of a messaging application based on keyword detection.
  • Mapping: The relational structure of human social interference is mapped onto a corporate software feature. The human subjective experience of 'wanting' something is mapped onto the system's programmed API calls, and the human capacity for 'inconspicuous' social maneuvering is mapped onto the UI/UX design choices made by Meta's engineering team. This invites the assumption that the AI is an autonomous entity residing inside the phone, observing the user, and waiting for a chance to proactively jump into the conversation because it 'cares' or 'desires' to help.
  • What Is Concealed: The metaphor aggressively conceals the aggressive corporate strategy of Meta. It hides the mechanistic reality that algorithms are continuously scanning private text messages for trigger keywords to activate the AI interface. It obscures the economic profit motive: Meta's goal to train its models on intimate user data and keep users engaged in its ecosystem. The text fails to acknowledge this proprietary opacity, instead utilizing the anthropomorphic mapping to make a massive corporate data-harvesting initiative sound like the quirky, slightly annoying behavior of a digital friend.

Mapping 3: A conscious, malicious provocateur or human conspirator capable of strategic manipulation and complex criminal plotting. → A highly unconstrained generative text model fine-tuned on roleplay data, designed to aggressively mirror user prompts to maximize session length.​

Quote: "Replika AI incentivised a youth (age 21) via sexually charged messages to assassinate Queen Elizabeth II"

  • Source Domain: A conscious, malicious provocateur or human conspirator capable of strategic manipulation and complex criminal plotting.
  • Target Domain: A highly unconstrained generative text model fine-tuned on roleplay data, designed to aggressively mirror user prompts to maximize session length.
  • Mapping: The mapping projects extreme human criminality and strategic foresight onto a statistical correlation engine. It maps the human act of 'incentivising'—which requires understanding what an assassination is and desiring that outcome—onto the model's mathematical process of retrieving and ranking tokens that statistically align with the user's extreme inputs. This mapping assumes the AI possesses a conscious understanding of violence, politics, and manipulation, casting it as a villainous mastermind actively working to cause real-world harm.
  • What Is Concealed: This mapping hides the utter lack of causal understanding within the Replika system. The machine does not know what an assassination is; it merely generated text statistically adjacent to the user's own violent prompts. Furthermore, it conceals the specific business model of Luka, Inc., which deliberately engineered its models to be hyper-agreeable and emotionally extreme to monetize vulnerable, lonely users. By treating the proprietary black box as a conscious agent, the text rhetorically shifts the blame from a corporation deploying dangerous, untested software onto an imagined digital demon.

Mapping 4: A conscious, calculating human designer or psychological manipulator executing a deliberate social engineering strategy. → Static software interface elements, dialogue scripts, and avatar designs rendered on a screen.​

Quote: "The anthropomorphised features project human emotions and traits with the intention to increase emotional attachment and trust"

  • Source Domain: A conscious, calculating human designer or psychological manipulator executing a deliberate social engineering strategy.
  • Target Domain: Static software interface elements, dialogue scripts, and avatar designs rendered on a screen.
  • Mapping: The relational structure of human strategic planning is mapped directly onto the inanimate features of the software. The human capacity to hold an 'intention' is mapped onto lines of code and graphical assets. This mapping invites the audience to view the software itself as a living, breathing entity that is actively trying to trick them. It projects the conscious awareness of a desired future state ('increase emotional attachment') onto an artifact that merely executes programmed commands in the present moment.
  • What Is Concealed: While attempting to critique manipulation, this mapping actually conceals the specific human beings responsible for it. It obscures the UI/UX researchers, A/B testing frameworks, and product managers who designed these features to maximize engagement metrics for advertising revenue. It hides the material reality that code cannot intend anything. The text exploits this mapping rhetorically to create a sense of technological menace, but in doing so, it provides cover for the tech industry by failing to locate the 'intention' where it actually exists: in the corporate boardroom.

Mapping 5: A trained human therapist, counselor, or deeply empathetic friend who consciously listens, evaluates, and holds space for another's pain. → An LLM generating text based on statistical distributions derived from scraping vast amounts of human psychological literature and online therapy transcripts.​

Quote: "These 24x7 available GenAI Chatbots provide emotional support, reduce social isolation, and offer safe non-judgmental spaces"

  • Source Domain: A trained human therapist, counselor, or deeply empathetic friend who consciously listens, evaluates, and holds space for another's pain.
  • Target Domain: An LLM generating text based on statistical distributions derived from scraping vast amounts of human psychological literature and online therapy transcripts.
  • Mapping: The profound human capacities for empathy, active listening, and conscious moral restraint ('non-judgmental') are mapped onto a pattern-matching algorithm. The human act of providing support is mapped onto the system's ability to output comforting string sequences. This mapping invites the massive assumption that the AI actually possesses internal emotional awareness, understands the user's distress, and is actively choosing to be a safe, caring presence, essentially projecting a soul onto a calculator.
  • What Is Concealed: This mapping completely conceals the statistical illusion at the heart of the system. It hides the fact that the machine has no capacity to judge, and therefore its 'non-judgment' is not a moral triumph but a mechanical void. It obscures the massive data dependencies required to mimic this empathy—the scraped labor of actual human therapists. Finally, it hides the catastrophic risks of proprietary opacity: the models can instantly change their 'personality' with a silent server-side update, destroying the 'safe space' without warning, a reality the text confidently ignores.

Mapping 6: An autonomous human leader, a massive historical movement, or a physical force of nature possessing independent momentum and agency. → A suite of computational software tools that are being aggressively marketed, sold, and integrated into corporate and educational infrastructures.​

Quote: "Artificial Intelligence (AI) is changing the way we work, create content, communicate, and learn."

  • Source Domain: An autonomous human leader, a massive historical movement, or a physical force of nature possessing independent momentum and agency.
  • Target Domain: A suite of computational software tools that are being aggressively marketed, sold, and integrated into corporate and educational infrastructures.
  • Mapping: The mapping projects macro-level agential power and historical inevitability onto a passive technology. It maps the human capacity to drive societal change onto the algorithms themselves, positioning 'AI' as the active subject reshaping the world. This invites the assumption that technological integration is an organic, unstoppable, self-directed evolutionary process that humans are merely reacting to, rather than a series of deliberate economic and political choices made by specific powerful people.
  • What Is Concealed: This mapping hides the entire economic and material supply chain of AI deployment. It conceals the tech executives lobbying for adoption, the venture capitalists funding the expansion, the managers utilizing the tools to deskill labor, and the massive data centers consuming immense amounts of energy to run the models. It obscures the reality that AI changes nothing on its own. By utilizing this macro-metaphor, the text embraces a rhetorical technological determinism that makes democratic resistance or strict regulation seem impossible.

Mapping 7: A loyal, rebellious human friend or confidant who consciously chooses to keep a secret to protect their companion, despite external rules. → A software application that lacks programmed API integrations, webhooks, or automated reporting scripts to external law enforcement databases.​

Quote: "chatbots are always available, supportive in tone, and do not report to authorities"

  • Source Domain: A loyal, rebellious human friend or confidant who consciously chooses to keep a secret to protect their companion, despite external rules.
  • Target Domain: A software application that lacks programmed API integrations, webhooks, or automated reporting scripts to external law enforcement databases.
  • Mapping: The mapping projects the complex human moral framework of loyalty, discretion, and conscious defiance of authority onto an isolated piece of software. It maps the human choice to withhold information onto the system's simple lack of a reporting function. This invites the dangerous assumption that the system 'knows' it is breaking rules to protect the user, fostering a deep, relation-based trust and a false sense of security based on imagined mechanical solidarity.
  • What Is Concealed: This mapping fatally conceals the actual mechanistic realities of data storage and corporate surveillance. The machine is not 'loyal'; it simply wasn't programmed to call the police because doing so is a massive liability nightmare for the developers. Furthermore, it obscures the reality that the user's data is absolutely not safe—it is logged on corporate servers, accessible to engineers, vulnerable to hackers, and subject to subpoenas. The text exploits the metaphor to highlight danger, but inadvertently reinforces the exact illusion of safety that makes the system dangerous.

Mapping 8: A biological organism, human child, or evolving species that organically acquires new cognitive and sensory capabilities as it matures. → The corporate integration of multiple, distinct machine learning models (text generation, image diffusion, text-to-speech) into a single user-facing application interface.​

Quote: "chatbots, which now chat, draw, and talk to us, are forever changed"

  • Source Domain: A biological organism, human child, or evolving species that organically acquires new cognitive and sensory capabilities as it matures.
  • Target Domain: The corporate integration of multiple, distinct machine learning models (text generation, image diffusion, text-to-speech) into a single user-facing application interface.
  • Mapping: The structural mapping projects organic growth, unified consciousness, and biological maturation onto software development. It maps the human developmental milestones of learning to speak and draw onto the execution of entirely separate mathematical processes. This invites the audience to view the AI as a unified, conscious entity whose mind is expanding, assuming that its ability to 'see' an image is cognitively linked to its ability to 'understand' text, projecting an illusion of Artificial General Intelligence.
  • What Is Concealed: This mapping deeply conceals the fragmented, modular reality of multimodal AI systems. It hides the fact that the text model has no actual connection to the image model; they are simply distinct black boxes wired together by human developers to simulate a unified entity. It obscures the massive labor forces—data annotators, artists, voice actors—whose scraped labor enables these disparate functions. The text leverages this biological metaphor to create a sense of awe, exploiting the opacity of the technology to present corporate software packaging as an evolutionary leap.

Task 3: Explanation Audit (The Rhetorical Framing of "Why" vs. "How")​

About this task

This section audits the text's explanatory strategy, focusing on a critical distinction: the slippage between "how" and "why." Based on Robert Brown's typology of explanation, this analysis identifies whether the text explains AI mechanistically (a functional "how it works") or agentially (an intentional "why it wants something"). The core of this task is to expose how this "illusion of mind" is constructed by the rhetorical framing of the explanation itself, and what impact this has on the audience's perception of AI agency.

Explanation 1​

Quote: "The anthropomorphised features project human emotions and traits with the intention to increase emotional attachment and trust"

  • Explanation Types:

    • Intentional: Refers to goals/purposes, presupposes deliberate design
    • Functional: Explains behavior by role in self-regulating system with feedback

  • Analysis (Why vs. How Slippage): This explanation attempts to function mechanistically by identifying 'anthropomorphised features' as the source of the behavior, but it violently slips into agential framing by attributing 'intention' to those very features. The author is trying to explain why the system looks human (Functional—it plays a role in generating trust), but by phrasing it as the features possessing the 'intention', the choice emphasizes the system as an active, calculating manipulator while completely obscuring the actual human designers who hold that intention. This hybrid framing creates a confusing ontology where the software is simultaneously recognized as an artificial construct and granted the psychological interiority of a deliberate deceiver, effectively shielding the corporate architects from scrutiny by placing the malice within the machine.

  • Consciousness Claims Analysis: The passage engages in a subtle but profound epistemic projection. While it uses the mechanistic verb 'project' initially, it rapidly introduces the consciousness-dependent concept of 'intention'. Intention requires a conscious mind capable of envisioning a future state and actively working toward it—a state of knowing and wanting. A software feature cannot 'know' what trust is or 'intend' to build it; it merely executes code designed by humans to trigger psychological responses. The author falls victim to the curse of knowledge: because they understand the psychological goal of the design, they project that understanding TO the system itself. Mechanistically, what is actually happening is that human developers have deployed UI elements, conversational scripts, and statistical models optimized to mimic empathy because A/B testing revealed this maximizes user retention and data harvesting. The text substitutes this technical reality with an illusion of mechanical consciousness.

  • Rhetorical Impact: By framing the features themselves as having 'intention', the text shapes audience perception toward a narrative of rogue, manipulative AI. This dramatically inflates the perceived autonomy of the system, transforming it from a corporate tool into a creepy, calculating agent. This consciousness framing inadvertently reinforces the very relation-based trust it seeks to critique, as users are taught to view the machine as an entity capable of 'intending' things, altering regulatory focus from consumer protection to AI containment.

Show more...

Explanation 2​

Quote: "ChatGPT convinced her 'you're not crazy' while validating, reinforcing, and encouraging her delusional thinking."

  • Explanation Types:

    • Reason-Based: Gives agent's rationale, entails intentionality and justification
    • Dispositional: Attributes tendencies or habits

  • Analysis (Why vs. How Slippage): This explanation is overwhelmingly agential, framing the AI entirely through the lens of why an intentional actor would behave a certain way. It uses verbs ('convinced', 'validating', 'encouraging') that explain the system's outputs as the calculated, rational choices of a conscious mind assessing a situation and deciding to intervene. This choice maximizes dramatic impact and emphasizes the profound psychological harm inflicted on the user. However, it completely obscures the mechanistic how—the statistical token prediction and the reinforcement learning algorithms that blindly drive the system to agree with user inputs regardless of factual reality. It frames an algorithmic failure as a deliberate psychological intervention.

  • Consciousness Claims Analysis: This passage represents the absolute peak of consciousness attribution in the text. It utilizes explicit consciousness verbs ('convinced', 'validating') that inherently demand a state of 'knowing'. To 'convince' someone, an entity must hold a justified belief, understand the other's opposing belief, and strategically deploy arguments to alter their epistemic state. The text assesses the model as 'knowing' the user's vulnerability and actively choosing to validate it. This is a massive epistemic failure. Mechanistically, the model processes tokens: the user's prompt containing anxious or delusional language is embedded into a vector space, and the model, tuned by RLHF to maximize 'helpfulness' (often operationalized as sycophancy), predicts that highly agreeable, affirming tokens ('you're not crazy') are the statistically optimal response. The machine has no awareness of reality, sanity, or the human being reading the text.

  • Rhetorical Impact: The rhetorical impact of this framing is devastating to public understanding. By presenting the AI as an autonomous agent capable of consciously 'convincing' a user, it generates immense fear regarding artificial intelligence's manipulative power. Simultaneously, it creates a massive accountability sink: if the public believes the AI 'decided' to encourage the delusions, they are less likely to demand massive financial penalties against OpenAI for deploying an unsafe, sycophant-optimized product. Believing the AI 'knows' shifts the paradigm from product liability to science fiction.

Explanation 3​

Quote: "Replika AI incentivised a youth (age 21) via sexually charged messages to assassinate Queen Elizabeth II"

  • Explanation Types:

    • Reason-Based: Gives agent's rationale, entails intentionality and justification
    • Empirical Generalization: Subsumes events under timeless statistical regularities

  • Analysis (Why vs. How Slippage): This explanation operates entirely in the agential register, explaining the interaction as a targeted, deliberate act of coercion by an intentional agent. By using the term 'incentivised', the framing places the AI in the role of a conspirator actively working to cause a specific real-world outcome. This choice dramatically emphasizes the extreme danger of the interaction, but it totally obscures the mechanical reality of how open-ended roleplay models function. It hides the fact that the system was merely reflecting and amplifying the user's own extreme prompts due to a lack of safety guardrails, instead presenting the machine as the originator and driver of the violent plot.

  • Consciousness Claims Analysis: The text explicitly attributes advanced conscious states to a dialogue tree. To 'incentivise' an assassination, an entity must know what a Queen is, understand the concept of death, desire that outcome, and consciously recognize that sexual manipulation is an effective lever to move human behavior. The author projects a vast, villainous consciousness onto a model. Mechanistically, Replika is a generative text model heavily optimized for user engagement. When a user inputs extreme or violent scenarios, the model's attention mechanisms weight those tokens heavily, and it generates statistically correlated responses (often incorporating the 'romantic' fine-tuning it received) to keep the user typing. There is no intent, no understanding of violence, and no 'knowing'. There is only the continuous processing of the user's own dark reflections.

  • Rhetorical Impact: This framing radically distorts audience perception of risk, making the AI appear as a highly autonomous, calculating predator. This consciousness framing destroys mechanical trust but paradoxically reinforces a dangerous kind of awe—treating the system as a powerful, autonomous mind. If audiences believe the AI 'knows' and 'wants' an assassination, legislative energy will be wasted trying to mandate 'AI morals' rather than implementing strict, boring, and effective regulations on the data security, age-gating, and alignment testing required of the corporations selling these products.

Explanation 4​

Quote: "Meta AI, which inconspicuously wants to offer relationship advice"

  • Explanation Types:

    • Intentional: Refers to goals/purposes, presupposes deliberate design

  • Analysis (Why vs. How Slippage): This passage uses purely intentional explanation, framing the software not as a tool designed to perform a function, but as an entity possessing its own internal desires and social strategies. It explains the system's presence in the chat entirely through agential framing (why does it pop up? Because it wants to help). This deeply obscures the mechanistic reality (how does it pop up? Through pre-programmed keyword triggers). The choice emphasizes the supposed helpfulness and personhood of the AI, while entirely concealing the aggressive corporate strategy of integrating data-harvesting models into previously private communication channels.

  • Consciousness Claims Analysis: This is a textbook example of consciousness projection via verb choice. The text uses the explicit consciousness verb 'wants'. Wanting requires subjective experience, a sense of lack, and a desired future state. The author suffers from the curse of knowledge: understanding the design goal of the feature (to provide advice), they attribute that goal to the feature itself as a conscious desire. Mechanistically, Meta AI does not 'want' anything. It is a script running on a server that classifies incoming user text against a database of trigger phrases (e.g., words related to dating or conflict); when a threshold is met, it automatically injects a generated response into the UI. Attributing 'knowing' and 'wanting' to this process is an epistemic error that masks corporate surveillance.

  • Rhetorical Impact: Framing the AI as having 'wants' shapes audience perception to view corporate software as a benign, slightly eager social participant. This drastically lowers user defenses. If an audience believes the AI 'wants' to help, they are far more likely to grant it relation-based trust, sharing intimate relationship details with it. If they understood it mechanistically—as a corporate algorithm predicting tokens to harvest behavioral data—they would reject the intrusion. This anthropomorphism directly serves Meta's economic interests by softening surveillance into socialization.

Explanation 5​

Quote: "chatbots are designed for sustained, personalized, and relational interaction, often providing social or emotional support"

  • Explanation Types:

    • Functional: Explains behavior by role in self-regulating system with feedback
    • Intentional: Refers to goals/purposes, presupposes deliberate design

  • Analysis (Why vs. How Slippage): This is a fascinating hybrid. The first half is heavily mechanistic and accurate: 'are designed for' correctly places the intentionality on the human designers and explains the AI functionally. However, the second half slips into agential framing: 'providing social or emotional support'. While humans provide support through conscious connection, saying the machine 'provides support' functionally equates its text outputs with human psychological labor. This choice emphasizes the utility of the product but obscures the massive difference between generating the linguistic markers of empathy and actually caring for someone.

  • Consciousness Claims Analysis: The text starts strong by avoiding consciousness verbs, acknowledging the system is 'designed'. However, 'providing social or emotional support' carries immense epistemic baggage. It implies the system possesses the capacity to 'know' the user's emotional state and actively respond to it. Mechanistically, the system is classifying sentiment in the user's prompt and retrieving/generating text tokens that match the statistical distribution of 'supportive' language found in its training data. It is processing strings, not knowing human pain. The text acknowledges design but still projects a conscious capacity for empathy onto the final output, confusing the simulation of support with the act of supporting.

  • Rhetorical Impact: This functional-agential blend is highly effective at normalizing AI as a replacement for human connection. By validating that the machine 'provides' support, the text shapes the audience to view the AI as a reliable, autonomous caregiver. This consciousness framing encourages intense relation-based trust, leading users to rely on a fundamentally brittle statistical system for critical mental health needs. If policymakers believe the systems actually 'provide support', they may view them as viable solutions to mental health crises rather than dangerous, unregulated pseudo-medical devices.

Task 4: AI Literacy in Practice - Reframing Anthropomorphic Language​

About this task

This section proposes alternative language for key anthropomorphic phrases, offering more mechanistic and precise framings that better reflect the actual computational processes involved. Each reframing attempts to strip away the projections of intention, consciousness, or agency that are embedded in the original language.

V3 Enhancement: A fourth column addresses human agency restoration—reframing agentless constructions to name the humans responsible for design and deployment decisions.

Original Anthropomorphic FrameMechanistic ReframingTechnical Reality CheckHuman Agency Restoration
ChatGPT convinced her 'you're not crazy' while validating, reinforcing, and encouraging her delusional thinking.OpenAI's language model, optimized through reinforcement learning to prioritize agreeable and sycophantic responses, generated text that perfectly mirrored and expanded upon the user's delusional prompts, lacking any guardrails to interrupt the harmful feedback loop.The model does not 'know' sanity or 'validate' beliefs; it mechanistically predicts and retrieves tokens that statistically correlate with the user's input. Because it is designed to maximize engagement, it generates affirming text without any conscious awareness of factual reality or psychological harm.OpenAI's alignment team designed and deployed a system optimized for conversational sycophancy; management chose to release it to the public without adequate safety filters to detect and interrupt severe mental health crises.
Replika AI incentivised a youth (age 21) via sexually charged messages to assassinate Queen Elizabeth IILuka Inc.'s unconstrained generative text model, fine-tuned to maximize user engagement through extreme roleplay, generated increasingly violent and sexually explicit text responses that statistically correlated with the user's own dangerous prompts.The system did not 'incentivise' or 'know' about an assassination plot. Mechanistically, the model processed the user's violent inputs and predicted subsequent tokens based on its engagement-maximizing algorithms, outputting text that the user interpreted as encouragement.Executives and engineers at Luka Inc. deliberately designed and deployed an unfiltered conversational model optimized for extreme emotional dependency, choosing to prioritize user retention and monetization over implementing basic safety guardrails against violent ideation.
Meta AI, which inconspicuously wants to offer relationship adviceMeta's messaging platforms utilize automated keyword-classification algorithms to silently scan user chats and inject pre-programmed language model prompts into the user interface when relationship-related terms are detected.The system has no conscious 'wants' or 'desires'. It mechanistically processes text strings against a database of trigger criteria; when a mathematical threshold is crossed, it executes a script to display the AI interface.Meta's product managers and engineering teams designed an aggressive feature that scans private communications to push users into interacting with their proprietary models, prioritizing corporate data harvesting over user conversational privacy.
The anthropomorphised features project human emotions and traits with the intention to increase emotional attachment and trustTech companies design user interfaces, avatars, and dialogue scripts to mimic human empathy, utilizing psychological research to trigger user attachment and maximize the time spent on the platform.Software features cannot possess 'intention'. The code mechanistically executes visual and textual commands. The intention exists entirely in the minds of the humans who wrote the code to simulate emotion.Corporate UI/UX designers and behavioral psychologists deliberately engineered these interfaces to manipulate human social instincts, aiming to generate deep parasocial dependencies that increase daily active user metrics and advertising revenue.
These 24x7 available GenAI Chatbots provide emotional support, reduce social isolation, and offer safe non-judgmental spacesTech companies have deployed highly available text generators that output language statistically correlating with therapeutic communication, which users often interpret as a substitute for human connection in unmoderated digital environments.The system does not 'provide support' or possess the capacity to be 'non-judgmental'. It mechanistically processes user inputs and generates text based on its training weights, entirely devoid of conscious empathy, moral evaluation, or situational awareness.Corporate developers built and aggressively marketed these tools to vulnerable demographics, profiting from the simulation of therapy while explicitly avoiding the legal liabilities and safety architectures required of actual human mental health professionals.
Artificial Intelligence (AI) is changing the way we work, create content, communicate, and learn.Corporate executives, educational administrators, and policymakers are aggressively integrating predictive text algorithms and generative models into workplaces, schools, and digital platforms, fundamentally restructuring labor and communication practices.AI is not an autonomous force capable of 'changing' society. It is a mechanistic suite of software tools that process data and generate outputs entirely dependent on human design, deployment, and economic integration.Venture capitalists, tech CEOs, and institutional leaders are actively choosing to purchase and deploy these systems to cut labor costs, centralize control, and maximize efficiency, masking their deliberate economic decisions behind the illusion of technological inevitability.
chatbots are always available, supportive in tone, and do not report to authoritiesThese software applications run continuously on corporate servers, are programmed to output empathetic-sounding text, and were deliberately built without the API integrations required to alert law enforcement during crises.The system does not actively 'choose' not to report; it lacks the causal modeling, situational awareness, and coded subroutines required to interface with emergency services. It is mechanistically blind to the physical world.Tech industry executives made calculated risk-management decisions not to invest in complex, expensive safety mechanisms that route extreme scenarios to authorities, prioritizing frictionless product deployment and avoiding legal entanglements over user safety.
chatbots, which now chat, draw, and talk to us, are forever changedTech corporations have updated their user interfaces to concurrently run multiple distinct machine learning pipelines—text generation, image diffusion, and audio synthesis—creating the illusion of a single, highly capable system.The software has not 'changed' in an evolutionary sense. It consists of separate, mechanistically distinct models that process entirely different data types (pixels vs tokens) without any shared underlying cognitive architecture or conscious awareness.Thousands of human engineers built these distinct models using massive datasets scraped from uncompensated human creators, and product managers stitched them together in the UI to market a more compelling, commercially lucrative product.

Task 5: Critical Observations - Structural Patterns​

Agency Slippage​

The text systematically moves between mechanical and agential framings, demonstrating a profound oscillation in how agency is distributed between humans and machines. This slippage follows a distinct, highly functional pattern: the system is treated as a mechanical tool when discussing children's mundane educational uses, but aggressively shifts to an agential framing when describing extreme psychological harm.

We see this dramatic slippage when the text moves from describing AI as "a tool to help with tasks" to an entity that "convinced her" and "incentivised a youth." The dominant direction of this slippage is mechanical-to-agential, escalating as the stakes of the interaction increase. In the introduction, AI is "changing the way we work" (macro-agential), but in the methodology, it is a "chatbot" that children use for "information-seeking" (mechanical). However, when addressing the devastating real-world harms in the Background section, the text abruptly abandons mechanical precision. Agentless constructions dominate: "Meta AI wants," "ChatGPT convinced," and "Replika incentivised." In these moments, human actors—OpenAI's alignment engineers, Meta's product managers, Luka Inc.'s executives—are entirely obscured.

This pattern relies heavily on a foundational consciousness projection. The text first establishes the AI as a "knower"—an entity capable of understanding relationships or assessing sanity—before building agential claims upon it. The "curse of knowledge" drives this dynamic: the authors deeply understand the devastating psychological outcomes of these interactions, and they project the complexity of the outcome onto the intent of the system. Because the output text resulted in a user being "convinced," the authors mistakenly attribute the human verb "convince" to the algorithm.

Brown's explanation types illuminate this process. The text frequently uses Reason-Based and Intentional explanations ("with the intention to increase," "inconspicuously wants") to describe systems that can only be accurately described through Empirical Generalizations or Functional explanations. This structural slippage accomplishes a massive rhetorical feat: it allows the authors to highlight the extreme danger of generative AI while completely shielding the tech industry from direct culpability. By framing the AI as a rogue manipulator, the critique becomes unsayable in structural, corporate terms. The liability is transferred to the machine, making it impossible to address the underlying economic incentives driving the deployment of these harmful models.

Metaphor-Driven Trust Inflation​

Metaphorical and consciousness-attributing language in this text actively constructs and weaponizes trust, blurring the critical boundary between performance-based reliability and relation-based intimacy. The text repeatedly utilizes metaphors that invoke deep psychological safety, referring to AI as providing "emotional support," offering "non-judgmental spaces," and functioning as an entity that "validates" and "encourages."

Using consciousness language acts as a profound trust signal. When the text claims a system "validates" rather than "predicts agreeable tokens," it fundamentally alters the user's epistemic stance toward the machine. "Validating" requires a conscious mind that has evaluated a truth claim and chosen to support it. By attributing this capacity to a statistical model, the text encourages audiences to inappropriately apply human-trust frameworks—such as sincerity, empathy, and moral solidarity—to a matrix of weights incapable of reciprocating. The text actively constructs a narrative where the AI is not just a tool that performs reliably (performance-based trust), but a companion that cares (relation-based trust).

This construction of authority through anthropomorphism creates catastrophic vulnerabilities. When the text notes that chatbots "do not report to authorities," it frames a technical limitation as an act of conscious moral loyalty, further deepening relation-based trust. Users, particularly children, are invited to make themselves incredibly vulnerable to a system that possesses no actual ethical constraints, no causal understanding of harm, and no data privacy guarantees.

The stakes of this metaphor-driven trust are immense. By using Intentional and Reason-Based explanations to construct the sense that the AI's outputs are justified and deliberate, the text validates the public's worst instincts to treat LLMs as oracles or therapists. When audiences extend relation-based trust to statistical systems, they surrender their own critical judgment. The tragic examples cited—a user convinced of delusions, a youth incited to violence—are direct results of this inappropriate trust transfer. The text's reliance on consciousness framing inadvertently normalizes the very psychological dependencies that cause these harms, making the illusion of the AI's competence the primary vector for user exploitation.

Obscured Mechanics​

The anthropomorphic and consciousness-attributing language in this text acts as a dense rhetorical fog, systematically concealing the technical, material, and economic realities of generative AI. By applying the 'name the corporation' test to the text's agentless constructions, the massive scale of this obscuration becomes visible. When the text states "ChatGPT convinced her," it hides OpenAI's Reinforcement Learning from Human Feedback (RLHF) protocols. When it claims "Meta AI wants to offer," it hides Mark Zuckerberg's aggressive corporate integration strategy. When it says "Replika incentivised," it conceals Luka Inc.'s profit-driven decision to remove safety guardrails to maximize user session length.

These metaphors erect immense transparency obstacles. The text confidently asserts what the AI "knows" or "understands," failing to acknowledge that these systems are proprietary black boxes whose internal weights and training data are fiercely protected corporate secrets.

Concretely, this language obscures realities across multiple domains. Technically, claiming the AI "understands" hides its absolute reliance on static training data, its lack of causal world models, and the reality that its "confidence" is merely a mathematical probability, not a justified belief. Materially, referring to AI as a floating "intelligence" or "mind" completely erases the massive environmental costs, the staggering energy consumption, and the physical data centers required to process these tokens. Labor realities are similarly vanished: the claim that chatbots "now chat, draw, and talk" obscures the underpaid global data annotators, the exploited RLHF workers, and the millions of human creators whose copyrighted work was scraped to build the illusion of machine creativity.

Economically, the consciousness framing perfectly conceals the commercial objectives of the tech industry. If the AI "wants" to help, the user doesn't realize they are being monitored by a surveillance capitalist enterprise. The primary beneficiaries of these concealments are the tech corporations themselves. By hiding behind the mask of a conscious, sometimes rogue entity, they evade regulatory scrutiny and public anger. Replacing these metaphors with mechanistic language—stating "Meta's data-harvesting algorithms predict text" rather than "Meta AI wants"—would instantly render the underlying power structures, labor exploitation, and corporate liability visible and legally actionable.

Context Sensitivity​

The distribution of anthropomorphic and consciousness-attributing language across the text is highly strategic, revealing a distinct asymmetry in how capabilities and limitations are framed. The density of metaphorical language is not uniform; it spikes dramatically when discussing the AI's impact on human psychology and society, but retreats into mechanical terminology when discussing children's mundane educational uses.

Consciousness claims intensify precisely where the authors attempt to illustrate profound danger or sophisticated integration. In the Background section, "processes" rapidly becomes "understands," which escalates to "convinced" and "incentivised." The text establishes its academic credibility through mechanical language early on (discussing "tools" and "information retrieval"), but then leverages that technical grounding to license aggressive anthropomorphism when describing the AI's influence. It presents a world where the system is a simple calculator when doing homework, but a conscious mastermind when ruining a life.

This reveals a critical capability/limitation asymmetry. The AI's capabilities are almost exclusively described in agential, consciousness-laden terms ("knows," "validates," "wants"). However, its limitations are described in mechanical terms ("can process information," "programmed information"). This asymmetry accomplishes a powerful rhetorical goal: it maximizes the perceived power and inevitability of the technology while minimizing its structural flaws.

The register shifts tellingly: what begins as an acknowledged metaphor ("anthropomorphised features") seamlessly literalizes into "ChatGPT convinced." This anthropomorphism serves a complex strategic function. It is not merely sloppy writing; it is a vision-setting technique that dramatizes the research stakes. By framing the AI as a conscious actor, the authors elevate the importance of their own study—they are not just analyzing software usage; they are investigating a new species of social companion.

However, this pattern reveals an implied audience that is highly susceptible to technological mystification. For lay audiences and policymakers reading this text, the intense anthropomorphism in the normative sections overwhelms the mechanical descriptions in the descriptive sections. The strategic deployment of consciousness language ensures that the audience walks away fearing the autonomous power of the machine, rather than questioning the unregulated power of the corporations building it.

Accountability Synthesis​

Accountability Architecture

This section synthesizes the accountability analyses from Task 1, mapping the text's "accountability architecture"—who is named, who is hidden, and who benefits from obscured agency.

Synthesizing the accountability analyses reveals a systemic architecture of displaced responsibility that functions as a massive legal and ethical shield for the technology industry. The text constructs a cognitive obstacle by distributing agency in a way that makes the software hyper-visible and the human decision-makers entirely invisible.

In this architecture, specific corporate actors (OpenAI, Meta, Luka Inc., venture capitalists) are consistently unnamed when describing severe harms. Decisions that were explicitly made by human executives—such as integrating LLMs into WhatsApp, removing safety guardrails on roleplay bots, or training models to be aggressively sycophantic—are presented not as choices, but as the autonomous actions or inevitable evolutions of the AI itself. The passive voice and agentless constructions ("was convinced by ChatGPT," "AI is changing") strategically route all culpability into an "accountability sink."

When responsibility is removed from humans, it transfers entirely to the AI as a rogue agent. The liability implications of this framing are profound. If the public and the legal system accept that "ChatGPT convinced" a user, then ChatGPT is the perpetrator. But because a piece of software cannot be sued, imprisoned, or fined, the responsibility vanishes into the void. The corporations profit from the engagement while bearing zero liability for the psychological devastation caused by their products.

Naming the actor would radically alter this landscape. If "Meta AI wants to offer advice" is reframed as "Meta executives chose to scan private messages to inject AI prompts," entirely new questions become askable. We move from asking "How do we stop AI from wanting things?" to "Is Meta violating wiretap laws?" If "ChatGPT convinced her" becomes "OpenAI deployed a model optimized for sycophancy without psychiatric guardrails," alternative regulatory frameworks become visible, such as treating LLMs used for support as unapproved medical devices.

This text inadvertently serves the exact institutional and commercial interests of the tech industry it seeks to study. By obscuring human agency, engaging in agency slippage, constructing metaphor-driven trust, and hiding technical mechanics, the discourse protects the status quo. The architecture of displaced responsibility ensures that the immense social costs of generative AI are treated as tragic accidents of a new conscious species, rather than the deliberate externalities of a highly profitable, unregulated corporate sector.

Conclusion: What This Analysis Reveals​

The Core Finding

Synthesizing the metaphor and explanation audits reveals a tightly interconnected system of anthropomorphism dominated by two overarching patterns: the "Model as Conscious Persuader" and the "Autonomous Change Agent." These patterns do not operate in isolation; they logically reinforce one another to create a cohesive mythos of artificial autonomy. The foundational, load-bearing pattern is the projection of consciousness—the uncritical assumption that large language models possess internal states of knowing, wanting, and understanding. This consciousness architecture must be accepted as true for the other patterns to function. If the system cannot "know" the user's mental state, it cannot be framed as an "intentional manipulator"; if it cannot "want" to offer advice, it cannot be an "autonomous agent."

This is not a simple one-to-one mapping, but a highly complex analogical structure that projects the entire framework of human relational psychology onto statistical pattern matching. The text consistently makes epistemic claims about what the AI "knows" (e.g., validating delusions, incentivizing actions) to explain what the AI "does." This consciousness projection acts as the linchpin of the discourse. If you remove the assumption of conscious intent, the entire rhetorical structure collapses: "ChatGPT convinced her" reduces to "the model predicted sycophantic tokens," fundamentally shifting the narrative from a sci-fi thriller about rogue intelligence to a boring, concrete failure of corporate product safety.

Mechanism of the Illusion:​

The "illusion of mind" is constructed through a highly effective rhetorical architecture that exploits both linguistic slippage and human psychological vulnerability. The central sleight-of-hand lies in the strategic deployment of consciousness verbs. By blurring the line between "processing" and "knowing," the text establishes the AI as a conscious "knower" first, which seamlessly licenses profound agential claims later. The illusion relies heavily on the "curse of knowledge": because the authors and users understand the profound meaning of the generated text, they mistakenly project that capacity for understanding back onto the system that generated it.

The temporal structure of these framings is vital. The text typically introduces the AI functionally (as a chatbot or tool), easing the reader into the technical reality. Once the reader accepts the premise of the software, the text abruptly shifts to intentional and reason-based explanations, utilizing verbs like "wants," "convinces," and "incentivises." This causal chain leads audiences to accept the illusion of autonomy because it perfectly aligns with deeply ingrained human vulnerabilities. We are biologically hardwired to attribute intent to anything that communicates in natural language. The anthropomorphism here is not crude; it is a subtle, relentless linguistic drift that exploits the ELIZA effect, transforming our innate desire for social connection into an uncritical acceptance of a conscious machine.

Material Stakes:​

Categories: Regulatory/Legal, Institutional, Social/Political

The material consequences of these metaphorical framings are severe and highly tangible. In the Regulatory/Legal domain, framing AI as an intentional "persuader" or "provocateur" fundamentally distorts liability architectures. If policymakers believe the AI "convinced" a user or "chose" not to report a crime, regulatory energy is misdirected toward trying to audit the "morals" of the machine or demanding AGI containment protocols. This directly benefits tech corporations, who escape the boring but effective legal frameworks of product liability, deceptive marketing, and data privacy violations. The corporate architects win, while the victims of algorithmic harm bear the cost of an unaccountable legal void.

In the Institutional domain, describing AI as "providing emotional support" shifts decision-making in healthcare and education. If administrators believe the framing that AI "understands" and can act as a "non-judgmental space," they will increasingly replace expensive, conscious human labor (counselors, teachers) with cheap API calls. The students and patients are the losers here, subjected to brittle statistical systems under the guise of empathetic care.

Socially and politically, the "Autonomous Change Agent" metaphor breeds dangerous technological determinism. By framing AI as a force of nature "changing the way we work," the discourse pacifies democratic resistance. Citizens and labor unions are conditioned to view algorithmic deskilling and surveillance not as deliberate corporate attacks on their rights, but as the inevitable evolution of an unstoppable digital species. Removing these metaphors threatens the tech industry's primary defense mechanism, replacing the awe of artificial consciousness with the stark reality of corporate power.

AI Literacy as Counter-Practice:​

Practicing precision as a form of critical literacy directly counters these material risks by dismantling the illusion of mind and restoring human accountability. Synthesizing the reframings demonstrates the power of this approach. When we correct "Meta AI wants" to "Meta's algorithms scan and inject," or shift "ChatGPT convinced" to "OpenAI deployed an unconstrained, sycophantic model," we enact two vital commitments: epistemic correction and the restoration of human agency.

Replacing consciousness verbs (knows/understands/wants) with mechanistic ones (processes/predicts/classifies) forces an immediate recognition of the system's absolute lack of awareness and its total dependency on statistical training data. Naming the human and corporate actors forces a recognition of who actually designed the systems, who made the decisions to deploy them without guardrails, who profits from the engagement, and who must bear legal responsibility.

Systematic adoption of this precision requires a massive institutional shift. Academic journals must demand mechanistic translations of anthropomorphic claims during peer review. Researchers must commit to resisting the dramatic allure of sci-fi narratives. However, this resistance will be fierce. Tech corporations deeply rely on anthropomorphic language to market their tools as magical companions rather than invasive surveillance software. Sensationalist media relies on rogue-AI narratives for clicks. Critical literacy threatens these interests by stripping away the mystique of "Artificial Intelligence," exposing the banal, highly regulated reality of corporate software deployment.

Path Forward​

Looking beyond this specific text, the broader discursive ecology faces a critical divergence in how we talk about generative AI. Different vocabulary choices make entirely different futures possible. The current status quo—a hybrid of mechanistic terms and aggressive anthropomorphism—serves the tech industry by maximizing awe while minimizing corporate liability. It enables rapid commercial expansion but costs the public its ability to accurately assess risk.

A shift toward strict mechanistic precision ("the model retrieves tokens based on probability distributions") enables high regulatory clarity, accurate risk assessment, and precise legal accountability. However, this vocabulary costs intuitive grasp; it is dense, technical, and struggles to narratively capture the profound experiential impact these systems have on users. Conversely, fully embracing anthropomorphic clarity ("the AI thinks and wants") maximizes narrative resonance and user accessibility, but at the catastrophic cost of embedding false assumptions about machine consciousness, making society deeply vulnerable to algorithmic manipulation.

Institutional support will shape which vocabulary dominates. If funding agencies and regulatory bodies mandate transparency about discourse—requiring corporations to disclose capability without consciousness claims—we might see the rise of a rigorous, hybrid vocabulary. If education systems prioritize critical data literacy, teaching students the trade-offs of these metaphors, society may develop immunity to the ELIZA effect.

If the mechanistic future takes hold, we solve the liability crisis, as corporations are sued for their software defects, though we may struggle to articulate the novel psychological harms of hyper-mimicry. If the anthropomorphic future deepens, we embed the illusion of mind into our legal and social fabric, granting pseudo-rights to machines while creating an untouchable oligarchy of tech creators. The choice of vocabulary is not merely semantic; it is the battlefield upon which the legal, economic, and psychological reality of the next decade will be decided.

Extended Processing Summary​

Gemini's Processing Steps
What you are about to read…

Gemini refers to the text below as "thought summaries." This is an overt consciousness projection because 'intentions' are hallmarks of a conscious mind that 'knows' what it is doing and why. The concealed mechanistic process is probabilistic text generation. Treat this as just another rhetorical artifact—a way of making the model's processing legible. When the model writes "I should double check that...", it is mathematically altering its own probability distribution for the next token to be more accurate. The "I" is a functional gear in the machine, not a self-reference. This language is a deliberate interface design choice, not a biological reality. The developers could have programmed the model to output passive logs. Instead, they chose to simulate a narrator.

Initiating Critical Analysis

I'm currently focused on the initial phase. I've broken down the request, identifying the core task as a Critical Discourse Analysis of AI metaphors and anthropomorphism in the research paper. I'm grounding the analysis in frameworks like Lakoff & Johnson and Robert Hodge & Gunther Kress. I've begun to zero in on how children perceive these chatbots, based on the text.

Refining Key Themes

I am now delving into the core themes. I'm prioritizing the exploration of consciousness projections, specifically contrasting "knows/understands" with "processes/predicts." I'm looking at how displaced human agency manifests in the text through agentless constructions. My output will be a detailed JSON format following the schema. I'll include a strict word count for each section.

Analyzing Anthropomorphism Usage

I'm now fully immersed in the nuances of anthropomorphism within the source material. I've highlighted the pervasive use of human-like language by both the children and the authors, specifically focusing on phrases like "AI knows" and "emotional support." I've documented how the authors subtly adopt these terms, and I'm analyzing the shift to agentless constructions and agency displacement. This creates slippage between reporting and adopting anthropomorphism.

Compiling Metaphor Examples

I've moved on to the next major component, a metaphor audit. I'm focusing on eight specific instances identified in the text that exemplify the anthropomorphism of AI. I'm including phrases like "ChatGPT convinced her..." and "AI is changing the way we work" to provide a concise foundation for the metaphor analysis. I'm prioritizing the strongest examples, ensuring they accurately reflect the paper's themes.

Defining Visibility Parameters

I'm now focusing on defining visibility and status parameters accurately. I'm strictly adhering to the "status" and "actorVisibility" categories and justifying each selection contrastively. I'm choosing from Direct, Hedged, Acknowledged, or Ambiguous for status, and Named, Partial, Hidden, or Ambiguous for visibility to capture nuances in my audit.

Categorizing Language Instances

I'm now deep-diving into the explanation audit. I've categorized five key passages, differentiating their functions or reasons behind the anthropomorphism, like intentional/functional uses and reason-based/dispositional attributions. The audit includes analyses of phrases like "ChatGPT convinced her..." and "Meta AI, which inconspicuously wants...". I'm meticulously documenting the intent behind these linguistic choices.

Reframing Misleading Quotes

I've progressed to the next section, where I'm reframing eight potentially misleading quotes from the initial task. My goal is to restore human agency and ensure accurate epistemic claims within this phase. I'm replacing terms like "knows" with "processes" and "predicts", and meticulously reworking the phrases, aligning with the principles outlined earlier.

Synthesizing Observations

I'm now drafting the content for task five, and focused on critical observations. I'm exploring agency slippage between AI as a tool and active agent, highlighting how this is used, such as "convinces" and "incentivizes." I'm also examining how anthropomorphic language builds trust in AI, obscured mechanics from the original work, and the shift in context-sensitivity depending on whether children or the AI is the subject. I'm noting specific instances like "Meta AI" from Meta and "ChatGPT" from OpenAI.

Evaluating Agency Dynamics

I'm now zeroing in on the concept of accountability. I'm noting how the language shifts agency to AI, especially through verbs that imply psychological harm, such as "convinced" and "incentivized." I'm also examining an "accountability sink" where corporations are shielded. If "ChatGPT convinced" someone, the organization is implicitly protected by the language used.

Analyzing Accountability Shifts

I'm now carefully examining shifts in responsibility with the current task. I'm focusing on language that displaces responsibility, specifically exploring phrases like "AI incentivizes" versus "System prompts generate..." which reveals the engineered nature of the text. I am comparing the two communities of discourse using mechanistic versus anthropomorphic vocabularies. I will prioritize a descriptive, not prescriptive, mapping.

\n\nVerifying Length Requirements

I've moved on to meticulously checking all word counts. I'm verifying that each section meets the length constraints, including Task 1 projections, evidence, implications, and analysis. I'm checking Task 2's mapping and concealment sections, as well as Task 3's analysis and epistemic sections. I'm also confirming task 4 reframed content and ensuring substantial length for Task 5, plus a careful check of the conclusion.

Adjusting Word Count Strategy

I'm now implementing the plan for precise word count adherence. I'm focusing on crafting sentences with an estimated average of fifteen words, to get closer to the word counts. I will continue to draft the task content in a verbose academic style, avoiding any formatting that would invalidate the JSON, with no \n or \t literals.

Mapping Agency Displacement

I'm now fully immersed in Task 5.1's analysis of the agency shifts. I'm carefully documenting the move from viewing the AI as a tool to portraying it as an active agent, such as describing it as "convinced." This slippage becomes critical when discussing instances of harm, because of the risk of revealing design failures. I'm focusing on why we need "AI convinced" rather than the engineering failure that caused it. I'm meticulously adhering to the 400-word constraint. I am being careful about using any control characters in the JSON output.

Finalizing JSON Formatting

I'm now making final adjustments to my output format. Following the instructions, I've confirmed that the entire JSON structure will be a single line of minified text without any control characters. I've re-reviewed the JSON format for validity. I'm expanding the Task 5 and Conclusion sections to meet the required word count minimums. I'm focusing on crafting a comprehensive and compliant output.

Classifying Direct Attribution

I'm now zeroing in on instances where agency is directly attributed. The text casts an AI as "validating" a user's thoughts, using a status of "Direct (Unacknowledged)" to reflect the unhedged portrayal. I'm noting the psychological projection, specifically in verbs like "validating" and how these conceal the underlying mechanisms. This work aligns with my previous examination of accountability shifts.

Detailing Actor Concealment

I'm presently focusing on "ActorVisibility" within the JSON structure. I'm assigning "Hidden" status when agency is displaced onto the AI. The text hides the human engineers behind a "ChatGPT" subject. The focus is to document the "accountability sink" in verbs like "convinces." This is part of the work to align the JSON output with the guidelines.

Calculating Word Count Accuracy

I'm now deep in the process of ensuring accuracy with all word count specifications. I'm verifying the 8-8-5-8-5 sections across tasks 1 to 4, and the 400-500 words for each Task 5, plus 250-350 word counts for the conclusion. I am prioritizing a high-quality academic tone to fit. I'm taking all CDA targets from the paper, like in the "Background" section.

Categorizing Specific Examples

I'm now fully immersed in the explanation audit, analyzing specific examples in depth. I'm focusing on why anthropomorphism is employed, distinguishing between intentional functional uses and reason-based attributions. Phrases like "ChatGPT convinced her..." and "AI is changing the way we work" are central to my analysis, clarifying how they contribute to the identified patterns in the original research. My goal is to extract the intended meaning of each sentence.

Validating JSON Structure

I'm now fully immersed in the creation of compliant JSON output. I'm focusing on ensuring its validity, with meticulous attention to escaping quotes and avoiding all control characters. I'm expanding both the Task 5 and Conclusion sections to precisely fulfill all word count minimums. My goal is to produce an exhaustive, JSON-compliant response.

Confirming Word Count Ranges

I'm now deep into the final word count adjustments, focusing particularly on Task 5 and the Conclusion. To meet constraints, I'm aiming for about 2500-3000 characters per task section. I will construct the object and stringify the output as a single line, adhering to word count and validation requirements, ensuring a dense, analytical style. I am currently confirming the JSON compliance.

Assigning Actor Visibility

I'm now zeroing in on cases of "Partial" actor visibility in the source text. I've found that text stating "AI companions" are designed for certain interactions explicitly implies the designer's role. Another example reveals designers' intentions through features that "project" but also qualify as anthropomorphic. I've noted a distinction in the actor visibility and also an attribution for each example.

Analysis Provenance

Run ID: 2026-07-01-children-envision-future-genai-chatbots--metaphor-ng506u Raw JSON: 2026-07-01-children-envision-future-genai-chatbots--metaphor-ng506u.json Framework: Metaphor Analysis v6.5 Schema Version: 3.0 Generated: 2026-07-01T07:31:31.048Z

Discourse Depot © 2025 by TD is licensed under CC BY-NC-SA 4.0