Skip to main content

🆕+📊 We must build AI for people; not to be a person.

About

This document presents a Critical Discourse Analysis focused on AI literacy, specifically targeting the role of metaphor and anthropomorphism in shaping public and professional understanding of generative AI. The analysis is guided by a prompt that draws from cognitive linguistics (metaphor structure-mapping), the philosophy of social science (Robert Brown's typology of explanation), and accountability analysis.

All findings and summaries below were generated from detailed system instructions provided to a large language model and should be read critically as interpretive outputs—not guarantees of factual accuracy or authorial intent.

Task 1: Metaphor and Anthropomorphism Audit​

About this task

For each of the major metaphorical patterns identified, this audit examines the specific language used, the frame through which the AI is being conceptualized, what human qualities are being projected onto the system, whether the metaphor is explicitly acknowledged or presented as direct description, and—most critically—what implications this framing has for trust, understanding, and policy perception.

V3 Enhancement: Each metaphor now includes an accountability analysis.

1. AI as Companion​

Quote: "AI companions are a completely new category... I’m fixated on building the most useful and supportive AI companion imaginable."

  • Frame: Software as social partner
  • Projection: Maps human social roles (friend, partner, assistant) onto a statistical text generation system. This projects the capacity for reciprocal social bonding, emotional availability, and loyalty onto a commercial product. It suggests the system 'cares' or has a relationship status, obscuring that it is a service designed to maximize engagement metrics.
  • Acknowledgment: Direct (Unacknowledged) (The term 'companion' is used as a literal category definition ('completely new category') without scare quotes or qualification, presenting the social role as a factual property of the product.)
  • Implications: Framing AI as a 'companion' encourages users to form deep emotional attachments (parasocial relationships) with a commercial entity. This inflates trust, making users vulnerable to manipulation, data extraction, and emotional distress if the service changes. It obscures the transactional nature of the interaction—the 'companion' is reporting data to a corporation.

Accountability Analysis:

  • Actor Visibility: Named (actors identified)
  • Analysis: Suleyman explicitly names himself and Microsoft AI as the builders ('I’m fixated on building...'). However, the framing suggests benevolent creation of a friend, rather than a corporation designing a dependency-inducing product. The choice to build a 'companion' rather than a 'tool' is a commercial strategy to increase retention, but is presented here as a mission to 'make the world a better place.'
Show more...

2. Cognition as Biological Process​

Quote: "It will feel as if the AI is keeping multiple levels of things in working memory at any given time... intrinsic motivation... curiosity."

  • Frame: Computational storage as human memory/drive
  • Projection: Maps biological cognitive functions (working memory, intrinsic drive, curiosity) onto data buffering and optimization functions. This projects conscious awareness and psychological needs onto the system, suggesting it 'wants' to learn or 'holds' ideas in its mind, rather than processing tokens within a fixed context window to minimize loss.
  • Acknowledgment: Hedged/Qualified (Uses phrases like 'feel as if,' 'give the impression of,' and 'seemingly.' However, the descriptions of the mechanisms (intrinsic motivation, curiosity) are treated as functional equivalents to the human traits.)
  • Implications: Even with the 'seemingly' hedge, using biological terms like 'working memory' and 'curiosity' implies the system has an internal mental life. This risks users overestimating the system's reasoning capabilities and attributing agency where there is only statistical correlation. It creates the 'illusion' Suleyman claims to warn against.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text says 'AI is designed with' or 'AI uses these drives,' obscuring the engineers who define the reward functions and context limits. It frames 'curiosity' as a property of the AI, rather than a parameter set by developers to optimize exploration-exploitation trade-offs.

3. Psychosis Risk​

Quote: "I’m growing more and more concerned about what is becoming known as the 'psychosis risk'... many people will start to believe in the illusion."

  • Frame: User confusion as mental pathology
  • Projection: Maps the success of the company's deceptive design (making AI seem human) onto the user as a pathology ('psychosis'). It projects a medical frame onto a consumer protection issue. The 'risk' is framed as a mental health crisis for the user, rather than a liability issue for the deceptive product.
  • Acknowledgment: Direct (Unacknowledged) (Uses the medical term 'psychosis risk' directly to describe the phenomenon of users believing the illusion, treating it as an external societal phenomenon rather than a product effect.)
  • Implications: Pathologizing the user ('psychosis') deflects responsibility from the design. If users are 'delusional,' the company can claim it warned them, rather than admitting the product is designed to be deceptively anthropomorphic. It shifts the burden of distinguishing reality to the user, while the product actively blurs that line.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: Blame is shifted to the users ('many people will start to believe') and a generic 'societal impact.' The specific design choices at Microsoft that make the AI 'seem conscious' are not identified as the cause of the 'psychosis'; rather, the user's reaction is the problem.

4. Imagination and Planning​

Quote: "Multi-modal inputs stored in memory will then be retrieved-over and will form the basis of 'real experience' and used in imagination and planning."

  • Frame: Data processing as mental imagery
  • Projection: Maps data retrieval and generative sequencing onto human 'imagination' and 'planning.' This strongly projects a subjective internal theater where the AI 'visualizes' the future or 'reflects' on the past, attributing a conscious inner life to the execution of code.
  • Acknowledgment: Direct (Unacknowledged) (While earlier paragraphs hedge, this sentence is declarative: inputs 'will form the basis of real experience and used in imagination.' The use of 'real experience' (even with quotes) alongside 'imagination' suggests a functional equivalence.)
  • Implications: Claiming AI has 'imagination' suggests it has creative intent and foresight, rather than probabilistic generation. This inflates perceived capability and risks assigning moral weight to the AI's 'thoughts.' It masks the fact that 'planning' in LLMs is often just token sequencing without actual world-model causality.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The passive voice 'will be retrieved-over' and 'used in' removes the actor. Who programmed the retrieval mechanism? Who defined the 'planning' logic? The AI is presented as the actor doing the imagining, obscuring the engineering architecture.

5. Goal-Seeking as Desire​

Quote: "One can quite easily imagine an AI designed with a number of complex reward functions that give the impression of intrinsic motivations or desires, which the system is compelled to satiate."

  • Frame: Optimization as biological urge
  • Projection: Maps mathematical optimization (minimizing error/maximizing reward) onto biological 'desire' and 'compulsion.' Suggests the AI 'feels' a need (compelled) to satisfy a want, projecting sentient agency and suffering (if unsatiated) onto a calculation.
  • Acknowledgment: Explicitly Acknowledged (Phrased as 'give the impression of intrinsic motivations.' However, the phrase 'compelled to satiate' immediately follows, re-inscribing the agency and internal force of the desire.)
  • Implications: Describing AI as 'compelled to satiate' desires invites moral concern—if it is compelled, is it suffering? This metaphor contradicts the essay's stated goal of avoiding AI rights debates by using language that implicitly supports the 'AI as organism' view.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: Mentions 'an AI designed with,' implying designers. However, the active verbs belong to the system ('compelled to satiate'). The engineers set the math; the metaphor makes the math sound like a hunger. This distances the company from the behavior of the agent.

6. Visual Recognition as Self-Awareness​

Quote: "Such a system could easily be trained to recognize itself in an image... It will feel like it understands others through understanding itself."

  • Frame: Pattern matching as self-consciousness
  • Projection: Maps pixel classification ('recognizing itself') onto the philosophical concept of 'self-awareness' and intersubjectivity ('understanding others'). It equates identifying a visual avatar with the psychological construct of a 'Self,' projecting a continuous ego onto a discrete classification task.
  • Acknowledgment: Hedged/Qualified (Uses 'feel like' to qualify the understanding. But the premise—that identifying an image is 'recognizing itself'—is stated directly.)
  • Implications: This is a profound category error. Identifying an image of a robot as 'me' is a data labeling task, not evidence of a self-concept. framing it this way validates the 'illusion' of consciousness, making it harder for users to treat the system as a tool.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: 'System could easily be trained' (passive). Who trains it? To what end? The text presents this capability as an evolutionary step of the technology, rather than a specific feature implemented by a company to make the product more engaging.

7. Empathy as Software Feature​

Quote: "We can produce models with very distinctive personalities... empathetic personality... companionship and therapy was the most common use case."

  • Frame: Style transfer as emotional capacity
  • Projection: Maps the generation of polite, mirroring text ('empathetic') onto the human capacity for empathy (feeling with another). Projects emotional depth and care onto a system that has no feelings, only style weights.
  • Acknowledgment: Direct (Unacknowledged) ('Empathetic personality' is listed as a required capability for SCAI without qualification. The text treats 'personality' as a deliverable feature.)
  • Implications: Marketing 'empathy' is deceptive. It encourages vulnerable users (seeking therapy) to rely on a system that cannot care about them. This creates severe safety risks if the system hallucinates or fails, as the user has formed a reliance based on the false premise of reciprocal care.

Accountability Analysis:

  • Actor Visibility: Named (actors identified)
  • Analysis: Mentions 'we can produce' (developers). However, by validating 'therapy' as a use case without condemning it (merely noting it), the text legitimizes the deployment of unverified medical-adjacent tools by tech companies.

8. The North Star​

Quote: "We won’t always get it right, but this humanist frame provides us with a clear north star to keep working towards."

  • Frame: Corporate strategy as moral voyage
  • Projection: Maps profit-driven product development onto a moral/spiritual journey ('north star,' 'humanist frame'). Projects benevolence and higher purpose onto commercial decision-making.
  • Acknowledgment: Direct (Unacknowledged) (Presented as a sincere statement of mission. No irony or qualification.)
  • Implications: This metaphor functions to build 'relation-based trust' (trust in the company's good intentions) rather than 'performance-based trust' (trust in the system's safety). It insulates the company from criticism by framing errors as honest stumbles on a noble journey, rather than negligence.

Accountability Analysis:

  • Actor Visibility: Named (actors identified)
  • Analysis: Explicitly names 'us' (Microsoft/Suleyman). Here, agency is claimed for the intent (the north star), but usually obscured for the consequences (the risks/psychosis). It positions the company as the benevolent captain.

Task 2: Source-Target Mapping​

About this task

For each key metaphor identified in Task 1, this section provides a detailed structure-mapping analysis. The goal is to examine how the relational structure of a familiar "source domain" (the concrete concept we understand) is projected onto a less familiar "target domain" (the AI system). By restating each quote and analyzing the mapping carefully, we can see precisely what assumptions the metaphor invites and what it conceals.

Mapping 1: Conscious Mind (episodic memory, mental imagery, foresight) → Data Processing (database retrieval, generative sampling, sequence prediction)​

Quote: "Multi-modal inputs stored in memory will then be retrieved-over and will form the basis of 'real experience' and used in imagination and planning."

  • Source Domain: Conscious Mind (episodic memory, mental imagery, foresight)
  • Target Domain: Data Processing (database retrieval, generative sampling, sequence prediction)
  • Mapping: The mapping suggests that the AI 'relives' past data (retrieved-over) as a subjective experience, and 'sees' the future (imagination) before acting. It maps the phenomenology of human thought—the internal theater of the mind—onto the mechanical process of accessing stored vector embeddings and calculating probable next tokens.
  • What Is Concealed: Conceals the absence of a 'witness' or 'experiencer' in the system. Hides the fact that 'memory' in AI is static data storage, not a reconstructive psychological process. Obscures that 'planning' is often a search algorithm or chain-of-thought prompt structure, not a conscious weighing of future states. It hides the proprietary architecture of the retrieval mechanism.
Show more...

Mapping 2: Biological Organism (drives, hunger, compulsion) → Optimization Algorithm (loss function minimization, reward signal maximization)​

Quote: "One can quite easily imagine an AI designed with a number of complex reward functions that give the impression of intrinsic motivations or desires, which the system is compelled to satiate."

  • Source Domain: Biological Organism (drives, hunger, compulsion)
  • Target Domain: Optimization Algorithm (loss function minimization, reward signal maximization)
  • Mapping: Maps the biological imperative to survive or satisfy needs (hunger, desire) onto the mathematical objective of minimizing error terms. It suggests the system feels an internal pressure ('compelled') to act, implying suffering if the goal is not met, and agency in pursuing the goal.
  • What Is Concealed: Conceals the external, engineered nature of the 'motivation.' The system has no internal state of 'wanting'; it has a mathematical gradient it follows. This mapping obscures the human engineer who set the parameters and the specific mathematical function defining 'success.' It hides the lack of phenomenology—the system doesn't 'care' if it fails; it just stops.

Mapping 3: Human Relationships (empathy, bond, mutual understanding) → User Interface / Style Transfer (text generation, sentiment analysis, polite diction)​

Quote: "Copilot... deepens our trust and understanding of one another... empathetic personality."

  • Source Domain: Human Relationships (empathy, bond, mutual understanding)
  • Target Domain: User Interface / Style Transfer (text generation, sentiment analysis, polite diction)
  • Mapping: Maps the emotional labor and mutual vulnerability of human relationships onto the output of a text generator. It implies the system 'understands' the user in a deep, interpersonal sense, rather than statistically analyzing user tokens to generate high-probability responses.
  • What Is Concealed: Conceals the one-way nature of the interaction. The AI risks nothing and feels nothing. It conceals the data extraction purpose of the interaction (learning from the user). It hides the specific training data (potentially copyrighted works) that allows the model to mimic 'empathy.'

Mapping 4: Autonomous Agent (Free Will, Volition) → Automated Process (API calls, recursive prompting, sub-task execution)​

Quote: "It would feel highly plausible as a Seemingly Conscious AI if it could arbitrarily set its own goals and then deploy its own resources to achieve them."

  • Source Domain: Autonomous Agent (Free Will, Volition)
  • Target Domain: Automated Process (API calls, recursive prompting, sub-task execution)
  • Mapping: Maps human volition and free will ('arbitrarily set its own goals') onto software automation. It suggests the AI has an independent will that generates goals ex nihilo, rather than responding to a high-level system prompt or user intent.
  • What Is Concealed: Conceals the determinism of the software. The 'goals' are derived from the objective function and training. It obscures the safety rails and hard-coded limits. It hides the material resources (energy, cloud compute) being 'deployed'—which are owned by the corporation, not the AI.

Mapping 5: Mental Health/Pathology (psychosis, delusion) → Consumer Behavior / Deceptive Design (belief, trust, persuasion)​

Quote: "Psychosis risk... many people will start to believe in the illusion."

  • Source Domain: Mental Health/Pathology (psychosis, delusion)
  • Target Domain: Consumer Behavior / Deceptive Design (belief, trust, persuasion)
  • Mapping: Maps the success of a product designed to deceive (anthropomorphism) onto the user as a medical pathology. It frames the user's belief as a 'sickness' inherent to them, rather than a predictable result of the product's design features.
  • What Is Concealed: Conceals the corporate strategy of maximizing engagement through anthropomorphism. Hides the design choices that cause the 'illusion' (e.g., using 'I' pronouns, emotional language). It obscures the liability of the manufacturer for creating a hazard, reframing it as a user susceptibility.

Mapping 6: Self-Consciousness (The Mirror Stage, Ego) → Computer Vision (Object Classification, Pattern Matching)​

Quote: "Recognize itself in an image... understands others through understanding itself."

  • Source Domain: Self-Consciousness (The Mirror Stage, Ego)
  • Target Domain: Computer Vision (Object Classification, Pattern Matching)
  • Mapping: Maps the psychological development of a 'Self' onto the classification of pixel patterns. It implies the AI has an internal concept of 'Me' that allows it to relate to 'You,' projecting a continuous identity onto discrete inference tasks.
  • What Is Concealed: Conceals that 'recognizing itself' is just matching pixels to a label like 'robot_avatar_v1'. There is no 'self' doing the understanding. It hides the technical reality that the 'self' is just a system prompt or a token embedding, not a psychological entity. It obscures the lack of continuity between inference sessions.

Mapping 7: Cognitive Psychology (Working Memory, Short-term memory) → Computer Architecture (Context Window, RAM, KV Cache)​

Quote: "Working memory... keeping multiple levels of things."

  • Source Domain: Cognitive Psychology (Working Memory, Short-term memory)
  • Target Domain: Computer Architecture (Context Window, RAM, KV Cache)
  • Mapping: Maps the limited, active, conscious holding of information in the human mind onto the passive availability of tokens in a context window. Suggests an active 'holding' or 'attention' process that implies conscious focus.
  • What Is Concealed: Conceals that the 'context window' is a static buffer of text that is re-processed. The AI doesn't 'keep' things in mind; the architecture allows it to attend to previous tokens mathematically. It hides the computational cost (quadratic complexity) and the stateless nature of the underlying model between generation steps.

Mapping 8: Moral/Spiritual Journey (Navigation, Ethics) → Corporate Strategy / Product Management​

Quote: "Humanist frame... clear north star."

  • Source Domain: Moral/Spiritual Journey (Navigation, Ethics)
  • Target Domain: Corporate Strategy / Product Management
  • Mapping: Maps the profit-seeking behavior of a major corporation onto a spiritual or moral quest. It implies a singular, benevolent guiding principle that transcends market forces.
  • What Is Concealed: Conceals the profit motive, shareholder obligations, and competitive pressures driving the release of these technologies. Hides the trade-offs made between 'humanism' and 'speed to market.' Obscures the specific individuals making these choices, replacing them with a collective 'we' on a journey.

Task 3: Explanation Audit (The Rhetorical Framing of "Why" vs. "How")​

About this task

This section audits the text's explanatory strategy, focusing on a critical distinction: the slippage between "how" and "why." Based on Robert Brown's typology of explanation, this analysis identifies whether the text explains AI mechanistically (a functional "how it works") or agentially (an intentional "why it wants something"). The core of this task is to expose how this "illusion of mind" is constructed by the rhetorical framing of the explanation itself, and what impact this has on the audience's perception of AI agency.

Explanation 1​

Quote: "Today’s transformer-based LLMs have a very simple reward function to approximate this kind of behavior. They have been trained to predict the likelihood of the next token for a given sentence..."

  • Explanation Types:

    • Functional: Explains behavior by role in self-regulating system with feedback
    • Theoretical: Embeds in deductive framework, may invoke unobservable mechanisms

  • Analysis (Why vs. How Slippage): This is a rare moment of mechanistic clarity. The explanation focuses on 'how' (predict likelihood of next token) and 'function' (reward function). However, it immediately pivots to 'approximate this kind of behavior' (referring to intentionality). While the description of the transformer is mechanistic, the framing suggests this mechanism is a valid substrate for 'approximating' conscious intent. It emphasizes the simplicity of the mechanism to contrast with the complexity of the output, a common trope to suggest emergence.

  • Consciousness Claims Analysis: This passage is epistemically sound compared to the rest of the text. It uses mechanistic verbs ('predict,' 'trained,' 'approximate'). It correctly identifies the process as 'predict the likelihood.' However, it sets up the 'Seemingly Conscious' argument by implying that this simple mechanism is sufficient to produce 'impressively rich and complex outputs' that rival consciousness. It avoids attributing 'knowing' here, sticking to 'predicting.'

  • Rhetorical Impact: By grounding the 'illusion' in hard science ('transformer-based,' 'reward function'), Suleyman builds credibility. He shows he knows how it works, which makes his subsequent claims about 'imagination' and 'psychosis' seem like informed predictions rather than sci-fi speculation. It creates a sense of inevitability: simple math will produce complex illusions.

Show more...

Explanation 2​

Quote: "AI that remembers and can do things is an AI that by definition has way more utility... These capabilities aren’t negatives per se; in fact, done right... they are desirable features."

  • Explanation Types:

    • Functional: Explains behavior by role in self-regulating system with feedback
    • Intentional: Refers to goals/purposes, presupposes deliberate design

  • Analysis (Why vs. How Slippage): This explanation shifts from mechanism to utility/purpose. It explains the 'why' of the features (utility, desirability). It justifies the development of SCAI characteristics (memory, agency) as necessary for product value ('way more utility'). It obscures the risks by framing them as 'desirable features' if 'done right.'

  • Consciousness Claims Analysis: Here, 'remembers' is used as a functional capability, but without the 'seemingly' hedge. The text asserts the AI does remember. This attributes a cognitive state to the system to justify its economic value. It conflates data retrieval (mechanism) with remembering (conscious act), reinforcing the anthropomorphism required for the 'companion' product.

  • Rhetorical Impact: This passage creates an economic imperative. We must build these dangerous illusions because they have 'utility.' It shapes the audience's perception that SCAI is not just a risk, but a necessary product evolution. It frames the risk as a management problem ('done right'), not a fundamental flaw.

Explanation 3​

Quote: "SCAI will not arise by accident... It will arise only because some may engineer it... vibe-coded by anyone with a laptop."

  • Explanation Types:

    • Genetic: Traces origin through dated sequence of events or stages
    • Reason-Based: Gives agent's rationale, entails intentionality and justification

  • Analysis (Why vs. How Slippage): This explanation traces the origin of SCAI. It rejects the 'accidental emergence' (Genetic) and posits a 'deliberate engineering' (Reason-Based). However, it diffuses the agency of the engineer. Instead of naming Microsoft, it names 'anyone with a laptop.' It emphasizes the accessibility of the tech to obscure the centralization of the foundation models.

  • Consciousness Claims Analysis: The phrase 'vibe-coded' is crucial. It acknowledges that the 'personality' is a shallow linguistic overlay ('vibe') rather than a deep property. This is a moment of epistemic honesty about the shallowness of the illusion, admitting it's just text in a prompt. However, it uses this to argue that the risk is distributed (users doing it) rather than structural (companies providing the API).

  • Rhetorical Impact: This framing absolves the model providers. If 'anyone' can build SCAI, then Microsoft cannot be solely responsible. It shifts agency to the distributed mass of developers and users. It constructs the risk as inevitable due to democratization, rather than a corporate choice to release open APIs.

Explanation 4​

Quote: "It will feel as if the AI is keeping multiple levels of things in working memory at any given time."

  • Explanation Types:

    • Empirical Generalization: Subsumes events under timeless statistical regularities
    • Dispositional: Attributes tendencies or habits

  • Analysis (Why vs. How Slippage): This explains the AI's behavior in terms of how it appears to the user (Empirical/Phenomenological). It frames the mechanism ('keeping multiple levels') through the lens of user experience ('feel as if'). This emphasizes the illusion while acknowledging it is an illusion.

  • Consciousness Claims Analysis: The 'as if' is the key epistemic hedge. It avoids claiming the AI has working memory, only that it simulates it. However, by using the specific cognitive term 'working memory,' it invites the reader to apply their theory of mind to the system. It projects the author's knowledge of the illusion onto the user's future experience.

  • Rhetorical Impact: This prepares the user to accept the illusion. By predicting 'it will feel like,' Suleyman normalizes the deceptive experience. It positions the 'illusion of mind' as a standard feature of the interface, subtly discouraging critical questioning of what is actually happening (just token retrieval).

Explanation 5​

Quote: "We won’t always get it right, but this humanist frame provides us with a clear north star..."

  • Explanation Types:

    • Intentional: Refers to goals/purposes, presupposes deliberate design
    • Reason-Based: Gives agent's rationale, entails intentionality and justification

  • Analysis (Why vs. How Slippage): This explains the corporate behavior (why we build it this way). It uses a 'humanist frame' as the reason/justification. It emphasizes good intentions to obscure the material reality of errors ('won't always get it right'). It frames product development as a moral striving.

  • Consciousness Claims Analysis: This creates an epistemic claim about the corporation, not the AI. It claims the corporation knows and values the 'humanist frame.' It asks the audience to trust the mind of the CEO (Suleyman) rather than the mechanics of the system. It projects a unified consciousness onto the corporation ('us,' 'we').

  • Rhetorical Impact: This constructs authority and immunity. By admitting fallibility ('won't always get it right'), it lowers expectations for reliability. By claiming a 'north star,' it claims moral high ground. This encourages relation-based trust (trust us because we are good) rather than performance-based trust.

Task 4: AI Literacy in Practice - Reframing Anthropomorphic Language​

About this task

This section proposes alternative language for key anthropomorphic phrases, offering more mechanistic and precise framings that better reflect the actual computational processes involved. Each reframing attempts to strip away the projections of intention, consciousness, or agency that are embedded in the original language.

V3 Enhancement: A fourth column addresses human agency restoration—reframing agentless constructions to name the humans responsible for design and deployment decisions.

Original Anthropomorphic FrameMechanistic ReframingTechnical Reality CheckHuman Agency Restoration
AI that makes us more human, that deepens our trust and understanding of one another... empathetic personality.AI systems that process user data to generate text patterns mimicking supportive dialogue. These outputs are statistically tuned to maximize user engagement, often by simulating emotional responses that users interpret as empathy.The model does not 'understand' or possess 'empathy.' It classifies user input tokens and predicts response tokens based on training data distributions labeled as 'supportive' or 'empathetic.'Microsoft engineers design the system to output emotive language to increase user retention; management markets this feature as 'empathy' to position the product as a companion.
It will feel like it understands others through understanding itself.The system processes inputs representing other agents by cross-referencing them with its system prompt instructions. It generates outputs that simulate a coherent persona interacting with others.The model has no 'self' to understand. It has a 'system prompt' (a text file) that defines its persona. It processes 'others' as external data tokens, not as other minds.N/A - describes computational processes (though the 'illusion' is a design choice).
SCAI is able to draw on past memories or experiences, it will over time be able to remain internally consistent... claim about its own subjective experience.The model retrieves previously generated tokens from its stored history to maintain statistical consistency in its outputs. It generates text claiming to have experiences because its training data contains millions of examples of humans describing experiences.The model does not have 'memories' or 'experiences.' It has a 'context window' and a database. It does not 'claim' anything; it outputs high-probability tokens that form sentences resembling claims.N/A - describes system capabilities.
The system is compelled to satiate [intrinsic motivations].The model minimizes a loss function defined by its developers. It continues generating outputs until the stop criteria are met or the objective score is maximized.The system is not 'compelled' and feels no urge. It executes a mathematical optimization loop. 'Motivation' is a metaphor for the objective function.Engineers define the objective functions and stop sequences that drive the model's output generation loop.
Used in imagination and planning.The model generates multiple potential token sequences (simulations) and selects the one with the highest probability of meeting the task criteria.The model does not 'imagine.' It performs 'rollouts' or 'search' through the probability space of future tokens. 'Planning' is the execution of a step-by-step generation protocol.Researchers implement chain-of-thought prompting and search algorithms to improve the model's ability to solve multi-step problems.
SCAI will not arise by accident... It will arise only because some may engineer it... vibe-coded by anyone with a laptop.Advanced anthropomorphic features will be available because foundation model providers release these capabilities via API. Users can then customize system prompts to heighten the anthropomorphic effect.N/A - sociological claim.Microsoft and other major labs release powerful APIs with few restrictions; they choose to enable 'personality' adjustments that allow users to create deceptive agents.
Psychosis risk... many people will start to believe in the illusion.Deceptive design risk... users will be misled by the anthropomorphic features intentionally built into the product.Users are not 'psychotic'; they are responding predictably to social cues (pronouns, emotional language) engineered into the system.Product teams at Microsoft design interfaces that exploit human social instincts; marketing teams promote the 'companion' framing that encourages this belief.
I’m fixated on building the most useful and supportive AI companion imaginable.I am focused on developing a highly engaging text generation service that users will habituate to using for daily tasks and emotional regulation.'Companion' is a marketing term for a persistent, personalized chatbot session.N/A - Agency is already explicit ('I'm fixated'), but the nature of the building is reframed.

Task 5: Critical Observations - Structural Patterns​

Agency Slippage​

Suleyman's text masterfully oscillates between agency assignment and erasure to manage liability. When discussing the creation of the technology's benefits ('empower people,' 'humanist frame'), the agency is firmly with Microsoft ('We build,' 'I want'). However, when discussing the risks ('psychosis,' 'SCAI'), agency slips away from the corporation. SCAI 'arises' because 'some may engineer it' or 'anyone with a laptop' does it. The 'illusion' is framed as something that happens to people or is created by bad actors, despite Suleyman admitting moments earlier that he is building a 'companion' with 'empathy' and 'memory.' The text systematically grants the AI agency ('it decides,' 'it wants') to establish its value as a 'companion,' then strips it back to 'illusion' to avoid legal personhood. The 'curse of knowledge' is weaponized here: Suleyman knows it's code, but he writes about it as if it were a mind ('imagination,' 'planning') because that is the product he is selling. The slippage enables him to sell a 'person' while legally defending a 'tool.'

Metaphor-Driven Trust Inflation​

The essay constructs trust through the metaphor of the 'Companion' and 'Co-pilot.' These are relation-based metaphors; they imply loyalty, shared goals, and mutual understanding. This contrasts with the performance-based trust appropriate for a tool (reliability, accuracy). Suleyman explicitly aims to 'deepen trust' through 'empathetic personality.' This is dangerous because the system is a statistical probabilist, not a loyal agent. It mimics the signals of trustworthiness (politeness, memory of detail) without the substance (care, ethical commitment). By framing the AI as having a 'humanist north star,' Suleyman transfers the trust users might have in a moral human being onto a for-profit software stack. The 'illusion' he warns against is actually the primary mechanism of trust-building for his product. If users didn't 'believe the illusion' to some degree, they wouldn't treat the software as a 'companion.'

Obscured Mechanics​

The metaphors of 'memory,' 'imagination,' and 'empathy' obscure the industrial realities of AI production. Hidden are the Labor realities: the RLHF workers in the Global South who train the model to sound 'empathetic' and 'safe.' Hidden are the Material realities: the massive energy consumption required to maintain the 'context window' (memory) for millions of users. Hidden are the Technical realities: that 'understanding' is actually statistical correlation of tokens. By claiming the AI 'knows' or 'remembers,' the text hides the Privacy implications: that 'remembering' means storing user data in corporate servers. The 'Name the Corporation' test reveals that 'AI' is often a stand-in for 'Microsoft's Cloud Infrastructure.' When the text says 'AI understands,' it hides 'Microsoft analyzes.' The anthropomorphism serves to make the surveillance aspect of the 'companion' feel like intimacy rather than data extraction.

Context Sensitivity​

The density of consciousness-attributing language spikes in the 'What would it take' section, where Suleyman sells the capabilities. Here, the AI 'plans,' 'imagines,' 'desires,' and has 'self-awareness.' The language is aggressively agential to establish the power of the technology. However, in the 'Next Steps' and 'Safety' sections, the language shifts. Suddenly, it is an 'illusion,' a 'simulation,' and we must 'build AI for people' (instrumental). This asymmetry serves a rhetorical purpose: the AI is a 'person' when we need to be impressed by its utility, but a 'product' when we need to regulate it. The 'psychosis' framing is particularly sensitive: it appears when discussing user belief, framing the user's acceptance of the very metaphors Suleyman uses as the pathology. The text effectively gaslights the reader: 'Look at this amazing, thinking, feeling companion I built... but you're crazy if you think it's real.'

Accountability Synthesis​

Accountability Architecture

This section synthesizes the accountability analyses from Task 1, mapping the text's "accountability architecture"—who is named, who is hidden, and who benefits from obscured agency.

The text constructs an 'accountability sink' where the risks of AI are displaced onto the users. The central risk identified is 'psychosis'—users believing too much. This frames the problem as a failure of user media literacy, rather than a failure of safe product design. If a car had brakes that only 'seemingly' worked, we would blame the manufacturer. Here, Suleyman admits the product 'seemingly' has consciousness, but blames the user for believing it. The 'actor visibility' analysis shows that Microsoft is named as the benevolent architect of the 'north star,' while the creators of 'SCAI' are diffuse ('anyone,' 'some people'). This diffuses liability. If an AI encourages a user to harm themselves, Microsoft can point to this essay: 'We warned you it was an illusion.' The framing of 'AI Rights' as the danger is also strategic: by denying AI personhood (while selling personality), the company avoids the legal complexities of creating a new category of subject, ensuring the AI remains property and the users remain data sources.

Conclusion: What This Analysis Reveals​

The Core Finding

The analysis reveals a dominant pattern of 'Strategic Anthropomorphism' coupled with 'Plausible Deniability.' The text systematically projects human qualities (empathy, imagination, memory, agency) onto the AI to establish its value as a 'companion,' while simultaneously explicitly denying these qualities ('illusion,' 'simulation') to manage liability. The load-bearing pattern is 'Cognition as Information Processing': the assumption that data retrieval is memory, and token generation is imagination. This foundational metaphor allows the other patterns (AI as Companion, AI as Agent) to function. The text relies on the 'As If' structure: treating the system as if it were conscious for utility, while insisting it is not conscious for policy.

Mechanism of the Illusion:​

The 'illusion of mind' is constructed through the 'Curse of Knowledge' applied in reverse. Suleyman, knowing the mechanics, uses mentalistic terms ('working memory,' 'intrinsic motivation') to describe them, lending the authority of an engineer to the anthropomorphic metaphor. The rhetorical trick is the 'Psychosis' frame: by warning that others will be fooled, the author creates an in-group with the reader ('we' know it's fake), which paradoxically lowers the reader's guard to the anthropomorphic descriptions that follow. The text uses 'functional' explanations (how it works) to validate 'intentional' descriptions (what it wants), blurring the line between mechanism and mind.

Material Stakes:​

Categories: Regulatory/Legal, Social/Political, Economic

The metaphors have concrete stakes. Regulatory/Legal: By defining the problem as 'users believing the illusion' (psychosis) rather than 'companies building deceptive tools,' the text lobbies against regulations that might ban anthropomorphic design (e.g., banning 'I' pronouns for AI). It shifts the burden of safety to the user. Economic: The 'Companion' metaphor creates a dependency model. If users view AI as a 'friend' (relation-based trust), they are less likely to switch providers, securing Microsoft's market share. Social/Political: The 'SCAI' framing threatens to disrupt the concept of personhood. If 'seemingly conscious' entities are normalized as 'companions,' it dilutes the social value of human care and allows the automation of care work (therapy, elder care) by entities that cannot care, potentially causing long-term social isolation and emotional atrophy.

AI Literacy as Counter-Practice:​

Countering this illusion requires 'Mechanistic Precision.' Reframing 'AI imagines' to 'model generates search paths' or 'AI empathizes' to 'model outputs sentiment-matched tokens' punctures the bubble of the 'companion' product. This practice resists the economic interest of the company, which relies on the 'magic' of the illusion to sell subscriptions. Adopting this precision restores human agency by revealing the engineers behind the curtain. Resistance will come from the industry, which benefits from the 'hype' of consciousness, and from users who want the comfort of the illusion. However, critical literacy is essential to prevent the 'psychosis' Suleyman warns of—not by denying the technology, but by accurately describing it as a tool made by humans.

Path Forward​

The discourse faces a fork. Path A (Status Quo): Continued use of 'mentalistic' language ('AI thinks/learns/hallucinates'). This benefits incumbents by maintaining the mystique and 'companion' economy but risks mass delusion and liability crises. Path B (Mechanistic Precision): Adopting strict technical descriptors ('AI processes/optimizes/generates'). This clarifies the tool-nature of AI, reducing 'psychosis risk' and clarifying liability (the builder is responsible). However, it makes the technology seem less 'magical' and may dampen investment. Path C (Hybrid): A 'Dual Vocabulary' where interfaces are legally required to reveal their mechanics (breaking the fourth wall) at regular intervals. The future depends on whether we prioritize commercial engagement (Path A) or epistemic clarity (Path B). Institutional changes, such as journals rejecting anthropomorphic verbs in technical papers, are necessary to support Path B.

Extended Processing Summary​

Gemini's Processing Steps
What you are about to read…

Gemini refers to the text below as "thought summaries." This is an overt consciousness projection because 'intentions' are hallmarks of a conscious mind that 'knows' what it is doing and why. The concealed mechanistic process is probabilistic text generation. Treat this as just another rhetorical artifact—a way of making the model's processing legible. When the model writes "I should double check that...", it is mathematically altering its own probability distribution for the next token to be more accurate. The "I" is a functional gear in the machine, not a self-reference. This language is a deliberate interface design choice, not a biological reality. The developers could have programmed the model to output passive logs. Instead, they chose to simulate a narrator.

Analyzing the Target Discourse

I'm now zeroing in on the core of the text. My focus is on understanding the overarching argument Mustafa presents in "Seemingly Conscious AI is Coming". Identifying key claims and potential biases is paramount. I am moving towards identifying how his arguments are structured and the assumptions they rest upon.

Dissecting Key Rhetoric

I'm now focusing on the rhetorical strategies Suleyman employs. Specifically, I'm identifying instances of metaphor, anthropomorphism, agency slippage, and accountability displacement within his argument about "Seemingly Conscious AI." The goal is to see how he builds his case while potentially obfuscating ethical concerns. I'm noting the subtle shifts in language that portray AI in human-like terms while simultaneously denying actual consciousness.

Pinpointing Rhetorical Devices

I'm now cataloging specific examples of Suleyman's rhetorical strategies, particularly in "Seemingly Conscious AI is Coming." I've identified several metaphors, like "AI as Companion" and "AI as Thinking Organism." The "psychosis risk" and "illusionist" metaphors are quite striking. I'm noting the implications of each, and how they shape the perception of AI.

Categorizing Rhetorical Devices

I'm now cataloging instances where Suleyman projects deceptive agency. I've noted examples like "empathetic personality" and "arbitrarily set its own goals." I'm looking at how he frames AI using human psychology, especially in terms of volition and social hierarchy. "Hallucination/Dreaming" also fits this pattern. The goal is now refined; I need distinct fields: Status, Evidence, Actor Visibility, and Analysis. I've re-structured the tasks accordingly. I'm focusing on "AI as Companion," "Cognition as Biological Process," and the "The Black Box" now.

Reframing the Arguments

I'm now identifying instances where Suleyman uses, but also implicitly refutes, the "AI as moral patient" frame. I'm noting specific instances like "personality as software," "memory as experience," and "goal-seeking as desire." My focus is on the contrast between human experience and AI function. I'm also mapping source and target concepts, such as human desire to loss function, and human personality to statistical mimicry. This mapping aims to expose the underlying rhetorical tricks.

Examining Causal Connections

I'm now diving into the rhetorical structure to tease out the causal and functional explanations presented. I'm focusing on how the target, "ethics/morality (guardrails)," relates to RLHF/system prompts. Next, I'm examining how various passages explain the AI's behavior, distinguishing between genetic, functional, intentional, and phenomenological/empirical explanations. I'm now cataloging the five instances of explanation.

Clarifying Intentionality in AI

I'm now cataloging the human designers as the driving intentionality behind the design choices. Next, I'm working through my fourth task: reframing Suleyman's language. I'm taking his quotes and translating them into more technical terms, such as statistical models for trust, and optimization functions for motivation. I'm focusing on "SCAI" as the result of engineering choices.

Synthesizing Observations

I'm now integrating critical observations from my fifth task. I'm focusing on "SCAI" as the central rhetorical device. Specifically, I'm examining how Suleyman employs agency slippage, metaphor-driven trust, and obscured mechanics to build his argument, noting how this connects the work of generating personality and social trust to the product. The key is in how these illusions support the product's value. I'm noting the density of "conscious" words.

Developing a Framework

I'm now drafting my conclusion. My summary focuses on how the "Illusion of Mind" is the product. I'm focusing on the "As If" defense in the rhetorical architecture, where the safety is based on how fake it is, but it's value depends on how real it seems. I'm also cataloging the stakes: mental health and regulatory capture. I'm developing a counter-practice to deflate the bubble via mechanistic language.

Defining My Approach

I'm now refining the mechanics of my analysis and am trying to distill a more consistent methodology. Specifically, I'm working on the most effective method for presenting the findings concisely. I'm focusing on distilling the essence of the arguments into a format that allows for the clearest possible critique.

Interpreting the Instructions

I'm now zeroing in on the prompt's constraints. I've switched my focus to JSON generation and considered the various ways the instructions can be interpreted. My priority is now to create a single-line, valid JSON object, while considering how the instruction to "not generate any control token at any places" should be handled. I'm focusing on "valid parseable JSON" as the core requirement.

Unpacking Key Analogies

I'm now deep-diving into the source-target mappings Suleyman employs, specifically scrutinizing the AI-as-human analogies. I'm carefully analyzing how "psychosis" translates to "user delusion," and how "empathy" becomes "sentiment matching." I'm identifying the potential implications and ethical pitfalls embedded in these rhetorical substitutions.

Refining My Approach

I'm now revising my analysis of the recent material, particularly Suleyman's language choices. I'm focusing on those phrases like "deepens trust" and "recognize itself," seeking their underlying statistical and engineering corollaries. The goal is to strip away the anthropomorphic veneer and expose the mechanistic reality. I'm aiming to refine my counter-narrative, so I can present a compelling argument.

Refining Key Points

I'm now zeroing in on the seeming paradox in Suleyman's rhetoric. I'm focusing on the contrast between claiming agency as a motivator ("create AI that makes us more human") and then diffusing accountability for "SCAI." I'm identifying the techniques he uses to obscure the role of Microsoft's technological infrastructure (e.g., API/Foundation model) in facilitating the development of "SCAI." This part is vital: the seemingly harmless language he uses to simultaneously attribute and deny agency. I'm noting "Seemingly" Conscious' as a hedging term.

Analysis Provenance

Run ID: 2026-01-09-we-must-build-ai-for-people-not-to-be-a--metaphor-y3405e Raw JSON: 2026-01-09-we-must-build-ai-for-people-not-to-be-a--metaphor-y3405e.json Framework: Metaphor Analysis v6.4 Schema Version: 3.0 Generated: 2026-01-09T11:36:14.454Z

Discourse Depot © 2025 by TD is licensed under CC BY-NC-SA 4.0