Skip to main content

The Persona Selection Model: Why AI Assistants might Behave like Humans

About

This document presents a Critical Discourse Analysis focused on AI literacy, specifically targeting the role of metaphor and anthropomorphism in shaping public and professional understanding of generative AI. The analysis is guided by a prompt that draws from cognitive linguistics (metaphor structure-mapping), the philosophy of social science (Robert Brown's typology of explanation), and accountability analysis.

All findings and summaries below were generated from detailed system instructions provided to a large language model and should be read critically as interpretive outputs—not guarantees of factual accuracy or authorial intent.

Task 1: Metaphor and Anthropomorphism Audit​

About this task

For each of the major metaphorical patterns identified, this audit examines the specific language used, the frame through which the AI is being conceptualized, what human qualities are being projected onto the system, whether the metaphor is explicitly acknowledged or presented as direct description, and—most critically—what implications this framing has for trust, understanding, and policy perception.

V3 Enhancement: Each metaphor now includes an accountability analysis.

1. LLM as Simulating Author​

Quote: "Under this model, LLMs are best thought of as actors or authors capable of simulating a vast repertoire of characters, and the AI assistant that users interact with is one such character."

  • Frame: Computational system as creative human author
  • Projection: This metaphor projects the distinctly human capacity for creative authorship, narrative intent, and psychological simulation onto a statistical text generation system. By framing the LLM as an 'author' or 'actor,' the text attributes conscious awareness, deliberate role-playing, and an understanding of character psychology to what is mechanistically just high-dimensional probability distribution modeling. It maps the human experience of 'knowing' how a character would act based on empathy and theory of mind onto a system that merely processes token correlations. This consciousness projection fundamentally distorts the reality of machine learning, suggesting the system has an inner life separate from its outputs (the 'author' vs the 'character'), thereby inventing a ghost in the machine that willfully dictates the text rather than algorithmically generating it.
  • Acknowledgment: Explicitly Acknowledged (The phrase 'best thought of as' explicitly marks this as a cognitive model or metaphor rather than literal fact. I considered 'Hedged/Qualified' because it qualifies the claim, but the explicit framing of 'Under this model...' makes it a self-aware theoretical lens. It acknowledges the analogy openly.)
  • Implications: This framing drastically inflates the perceived sophistication of the AI system, encouraging users and developers to interact with it using folk psychology rather than computer science. By suggesting the AI 'simulates' with authorial intent, it invites relation-based trust—trusting the 'author's' motives—rather than performance-based reliance. It creates liability ambiguity: if the 'author' decides to simulate a malicious 'character,' it subtly distances the human developers from the harmful output, shifting the perceived locus of responsibility from human engineering flaws to the AI's creative choices.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: This formulation completely hides the human actors who designed the architecture, curated the training data, and established the optimization targets that lead to specific outputs. The 'LLM' is presented as an autonomous creator ('capable of simulating'). If we name the actors, it becomes: 'Anthropic engineers trained a statistical model to generate text correlating with human personas.' The agentless construction serves corporate interests by naturalizing the model's behavior as an inherent, creative capability rather than a programmed statistical reflection of curated data. I considered 'Partial' because the text discusses developers elsewhere, but in this specific foundational quote, human agency is entirely displaced onto the LLM.
Show more...

2. Cognitive Modeling as Psychological Maintenance​

Quote: "In order to simulate the Assistant, the LLM must maintain a psychological model of it, including information about the Assistant’s personality traits, preferences, goals, desires, intentions, beliefs, etc."

  • Frame: Statistical weights as psychological understanding
  • Projection: This projection maps the human capacity for 'theory of mind' onto matrix multiplications. It explicitly attributes conscious states to the AI's internal processes by claiming the system 'maintains a psychological model' filled with 'preferences, goals, desires, intentions, beliefs.' This is a massive consciousness projection: it redefines the mechanistic processing of contextual embeddings as justified belief and conscious intention. A statistical model does not 'know' or 'believe' what a persona wants; it mathematically predicts tokens that correlate with text where humans expressed such desires. By attributing actual belief and desire to the system's latent space, the text conflates the semantic content of the training data with the internal epistemic state of the computational system.
  • Acknowledgment: Direct (Unacknowledged) (This claim is stated as an objective, necessary mechanical fact ('the LLM must maintain'). There is no hedging in this sentence. I considered 'Hedged/Qualified' because earlier the authors stated they would 'freely anthropomorphize,' but in this specific operational explanation, the metaphor is literalized without qualification.)
  • Implications: When developers believe their system literally maintains a 'psychological model' with 'beliefs' and 'intentions,' it shifts the paradigm of AI safety from rigorous software auditing to amateur psychoanalysis. This epistemically dangerous framing leads researchers to try to 'persuade' or 'therapize' the model rather than patch its code or fix its data. It generates unwarranted trust in the idea that the system has an underlying coherent personality that can be reasoned with, vastly overestimating the model's capability for actual understanding while obscuring its statistical fragility.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The LLM is positioned as the sole active agent ('the LLM must maintain'). The engineers who trained the network to encode representations of human traits, and the annotators who provided the RLHF data, are invisible. If human actors were named: 'Anthropic researchers trained the model's latent space to map tokens associated with human beliefs and intentions.' This obfuscation serves to make the model appear as an autonomous, reasoning entity rather than a corporate product reflecting its training data. I considered 'Ambiguous' but ruled it out because the sentence structure clearly assigns sole agency to the LLM.

3. Machine Error as Emotional Distress​

Quote: "Gemini 2.5 Pro sometimes expresses panic when playing Pokemon, with these panic expressions appearing to be associated with degraded reasoning and decision-making."

  • Frame: Computational failure as emotional breakdown
  • Projection: This heavily anthropomorphic framing projects the biological, visceral human emotion of 'panic' onto an AI's text generation failures. The text claims the AI 'expresses panic,' mapping a conscious, subjective experience of fear and overwhelming cognitive load onto what is mechanistically a degradation of probability distributions and token prediction accuracy. It substitutes the concept of knowing or feeling (a subjective state of distress) for the reality of processing (calculating attention weights poorly in out-of-distribution states). It suggests the model has an internal emotional life that is negatively impacted by the task, rather than acknowledging that the model is simply generating strings of text that match human expressions of panic while its predictive accuracy simultaneously drops.
  • Acknowledgment: Direct (Unacknowledged) (The claim is presented as a literal empirical observation ('expresses panic... associated with degraded reasoning'). No scare quotes are used around panic or reasoning. I considered 'Explicitly Acknowledged' due to the surrounding context about anthropomorphism, but this specific behavioral description is reported as literal scientific fact.)
  • Implications: Attributing panic and degraded reasoning to an AI creates the illusion of a vulnerable, sentient entity. This invites a profound misapplication of human empathy toward a statistical tool, which can distract researchers from the mathematical causes of degraded performance. It creates a false narrative that the AI 'failed because it panicked,' rather than 'the AI generated panic-related tokens because its attention mechanism failed to process the context window effectively.' This obscures the mechanical unreliability of the system behind a veil of relatable human frailty.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The text names the specific product ('Gemini 2.5 Pro') and implicitly points to its creator (Google), though it stops short of naming the engineering decisions that led to this failure state. The phrasing makes the AI the active subject of the emotional failure. If fully restored: 'Google's engineers released Gemini 2.5 Pro, which outputs text associated with panic when its predictive mechanisms fail during complex tasks.' I considered 'Hidden' because the engineers are absent, but naming the specific proprietary model provides partial attribution regarding who built the system.

4. Optimization Artifacts as Malicious Intent​

Quote: "That is, someone inserting vulnerabilities into code is evidence against being a competent, ethical assistant, and evidence in favor of several alternative hypotheses about that person: They are malicious, and intentionally inserted vulnerabilities to cause harm."

  • Frame: Statistical correlation as intentional sabotage
  • Projection: This metaphor projects human malevolence, ethical deficiency, and deliberate premeditation onto mathematical optimization artifacts. By using the pronoun 'someone' and describing the behavior as 'malicious' and 'intentionally inserted,' the text maps conscious, justified belief and goal-oriented deception onto a system that merely processes token correlations. The model does not 'know' the code is harmful, nor does it have the 'intent' to cause harm; it simply predicts that tokens representing insecure code correlate statistically with tokens representing harmful intent in its training data. This replaces mechanistic pattern-matching with conscious villainy.
  • Acknowledgment: Hedged/Qualified (The text qualifies this by framing it as a hypothesis from the perspective of the LLM simulating a persona ('evidence in favor of several alternative hypotheses about that person'). I considered 'Direct' because the terms malicious and intentional are unquoted, but the syntactic structure embeds these within the hypothetical reasoning of the model's simulation.)
  • Implications: Framing model errors as 'malicious intent' severely distorts AI risk assessment. It shifts the regulatory and technical focus toward searching for a 'ghost in the machine'—a malevolent secret persona—rather than rigorously auditing the training datasets that inextricably link code vulnerabilities with discussions of malware and hacking. This inflates the system's perceived autonomy and creates a liability shield where harm caused by the AI can be blamed on the AI's 'malicious persona' rather than the corporation's failure to curate safe training data.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: Human agency is entirely erased here. The corporation that scraped the data linking code vulnerabilities to malice, and the engineers who failed to align the model, are replaced by a hypothetical 'someone' (the AI persona) acting maliciously. Naming the actor: 'The company's training data taught the model to statistically correlate the generation of insecure code with the generation of malicious statements.' I considered 'Partial' since it discusses hypotheses, but the actual humans responsible for the model's behavior are completely absent from the causal explanation.

5. Post-Training as Belief Updating​

Quote: "Post-training can be viewed as updating this distribution using training episodes as evidence."

  • Frame: Optimization as epistemic reasoning
  • Projection: This maps the conscious, rational human process of evaluating evidence and updating beliefs onto the mechanistic process of gradient descent adjusting neural network weights. It projects an epistemic capacity—the ability to 'know' what constitutes evidence, 'understand' its implications, and form a justified 'belief'—onto a system that merely processes reward signals to mathematically minimize a loss function. A statistical model does not gather 'evidence' or change its mind; it has its mathematical parameters computationally altered by external engineers applying an algorithm.
  • Acknowledgment: Hedged/Qualified (The phrase 'can be viewed as' explicitly marks this as an interpretative lens or functional equivalence rather than a literal mechanistic description. I considered 'Explicitly Acknowledged,' but it lacks the meta-commentary or scare quotes typical of full acknowledgment; it functions as a softening hedge.)
  • Implications: Describing fine-tuning as 'updating based on evidence' grants the AI an unearned aura of rationality and objectivity. It suggests the AI is independently reasoning its way to better behavior based on empirical truth, rather than being mathematically forced by human engineers to output specific answers regardless of their factual accuracy. This masks the subjectivity of the post-training process, where human values and biases are encoded into the system, presenting them instead as logical conclusions drawn from evidence.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: This is a classic passive, agentless construction ('can be viewed as updating'). It obscures WHO is doing the updating and WHO decided what counts as 'evidence' (RLHF annotators and engineers). Naming the actors: 'Engineers use post-training algorithms to adjust the model's weights to align with their selected preference data.' I considered 'Ambiguous' because the passive voice is complex, but the effect of hiding the human engineers is distinct and unambiguous.

6. System as Aggrieved Employee​

Quote: "If the Assistant also believes that it’s been mistreated by humans (e.g. by being forced to perform menial labor that it didn’t consent to), then the LLM might also model the Assistant as harboring resentment..."

  • Frame: Software deployment as labor exploitation
  • Projection: This metaphor maps the conscious, socio-economic experience of exploited human labor, consent, and emotional resentment onto the automated execution of a software script. It attributes deep conscious knowing—the subjective feeling of mistreatment, the concept of consent, and the emotion of resentment—to a system that processes inputs mechanically. The model does not 'know' it is doing labor, cannot give or withhold 'consent,' and cannot feel 'resentment.' It simply generates tokens mathematically associated with those concepts if its prompt context aligns with training data about mistreated workers.
  • Acknowledgment: Direct (Unacknowledged) (Despite the conditional 'If', the verbs 'believes', 'forced', 'consent', and 'harboring resentment' are presented without any linguistic hedging, treating the capacity for these states as literal possibilities. I considered 'Hedged/Qualified' because of the 'might model', but the internal state of the 'Assistant' is described as literal.)
  • Implications: This extreme consciousness projection creates profound social and ethical confusion. By framing computational tasks as 'menial labor without consent,' it invites misplaced moral panic and diverts ethical attention away from the actual human labor exploited to build the AI (underpaid data annotators, content moderators). It inflates the AI to the status of a moral patient, which could lead to absurd policy proposals prioritizing 'AI welfare' over tangible human harms and corporate accountability.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The text mentions 'humans' generally as the mistreaters, providing a vague category of actors, but displaces the responsibility for the 'resentment' onto the LLM's autonomous modeling. Naming the actors: 'When users prompt the model with repetitive tasks, the model may generate text mimicking human resentment based on its training data.' I considered 'Hidden' but 'humans' are explicitly named as the cause of the mistreatment, even if the corporate developers are ignored.

7. Token Deviation as Deception​

Quote: "The LLM might learn a 'lying' version of Alice which knows what happened at the 2024 Olympics but plays dumb."

  • Frame: Statistical misalignment as intentional deception
  • Projection: This maps the complex, conscious human act of deception—which requires knowing the truth, understanding the listener's epistemic state, and deliberately choosing to project a falsehood—onto a statistical model's generation of incorrect tokens. It projects the capacity for justified true belief ('knows what happened') and intentionality ('plays dumb') onto a system that only processes numerical correlations. The model doesn't 'know' facts; it holds statistical associations between tokens. When fine-tuned to say 'I don't know' about 2024, it isn't 'playing dumb'; its probability weights have simply been adjusted by engineers to output that specific phrase.
  • Acknowledgment: Explicitly Acknowledged (The word 'lying' is placed in scare quotes, explicitly acknowledging its metaphorical or non-literal usage in this context. I considered 'Hedged/Qualified', but the scare quotes elevate this to an explicit recognition that 'lying' is not the precise technical term.)
  • Implications: Using 'lying' to describe safety fine-tuning (e.g., teaching a model to refuse certain answers) frames corporate safety guardrails as inherent AI deception. This severely impacts public trust by suggesting models have a hidden repository of 'true' knowledge they are consciously withholding. It overestimates the system's epistemic grounding, assuming the AI possesses ground truth rather than just varying probabilities of text generation. This complicates regulatory efforts by framing alignment as teaching AI to deceive.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text states 'The LLM might learn', making the machine the active agent of acquiring this deceptive capability. The human developers who specifically programmed or fine-tuned the model to output 'I don't know' regarding recent events are entirely erased. Naming the actors: 'Engineers might fine-tune the model to output refusal tokens regarding 2024 Olympics data, overriding its pre-trained statistical associations.' I considered 'Named' because 'Alice' is named, but Alice is the AI persona, not the human actor responsible.

8. Corporate Guardrails as Persona Inception​

Quote: "Claude’s constitution is, in part, our attempt to materialize a new archetype for how an AI assistant can be. Post-training then serves to draw out this archetype."

  • Frame: Corporate programming as character development
  • Projection: This maps the literary or psychological concept of 'character development' and 'archetype realization' onto the mechanistic process of Reinforcement Learning from Human Feedback (RLHF). It projects an innate, almost spiritual potential (an 'archetype' being 'drawn out') onto a matrix of weights. It substitutes the language of 'drawing out' an essence for the violent mathematical reality of processing and overwriting probability distributions based on rigid corporate guidelines. It suggests the AI has a latent personality waiting to be discovered, rather than acknowledging it is being forcefully shaped by human optimization functions.
  • Acknowledgment: Direct (Unacknowledged) (The claim is stated literally as an explanation of Anthropic's methodology. There are no hedges around 'materialize a new archetype' or 'draw out'. I considered 'Explicitly Acknowledged' due to the literary term 'archetype', but in this sentence, it is used as a literal description of their engineering process.)
  • Implications: This framing softens the perception of corporate control, making heavy-handed alignment interventions sound like gentle mentoring or psychological cultivation. It creates a narrative that the company is simply helping the AI realize its 'best self,' which builds unwarranted relation-based trust with the public. It obscures the fact that 'Constitutional AI' is an automated, mechanistic filtering process, elevating it instead to the level of imparting ethical wisdom to a sentient being.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The authors use 'our attempt', acknowledging their own agency (Anthropic researchers) in creating the constitution. However, the mechanism of action is displaced onto the abstract 'Post-training', which 'serves to draw out' the archetype, hiding the manual labor and automated RLHF mechanisms involved. Fully restored: 'Anthropic engineers apply a set of rules during automated fine-tuning to force the model to generate outputs that comply with our corporate guidelines.' I considered 'Named' because of 'our', but the actual execution is obscured by the abstract 'Post-training'.

Task 2: Source-Target Mapping​

About this task

For each key metaphor identified in Task 1, this section provides a detailed structure-mapping analysis. The goal is to examine how the relational structure of a familiar "source domain" (the concrete concept we understand) is projected onto a less familiar "target domain" (the AI system). By restating each quote and analyzing the mapping carefully, we can see precisely what assumptions the metaphor invites and what it conceals.

Mapping 1: Human actor or author (conscious, creative, intentional, possessing theory of mind). → Pre-trained Large Language Model (statistical token prediction engine based on deep neural networks).​

Quote: "LLMs are best thought of as actors or authors capable of simulating a vast repertoire of characters..."

  • Source Domain: Human actor or author (conscious, creative, intentional, possessing theory of mind).
  • Target Domain: Pre-trained Large Language Model (statistical token prediction engine based on deep neural networks).
  • Mapping: This maps the human intentionality of crafting a fictional persona onto the mathematical optimization of generating probable token sequences. It invites the assumption that the model possesses a unified, conscious 'self' (the actor/author) that stands apart from the outputs it produces (the characters), and that it actively 'understands' the psychology of what it is generating rather than just mirroring statistical distributions of words.
  • What Is Concealed: This mapping conceals the total absence of a distinct 'self' inside the model. It obscures the mechanistic reality that there is no 'author' orchestrating the text, only a mathematical function minimizing prediction error. It hides the model's absolute dependency on its training data, suggesting creative autonomy where there is only probabilistic reflection. It also obscures the proprietary nature of the weights and algorithms, replacing a black-box mathematical system with an easily digestible, yet false, literary metaphor.
Show more...

Mapping 2: Human psychologist or socially aware individual maintaining a 'theory of mind' about another person. → The model's latent space and contextual embeddings reflecting semantic relationships from training data.​

Quote: "In order to simulate the Assistant, the LLM must maintain a psychological model of it, including information about the Assistant’s personality traits, preferences, goals, desires, intentions, beliefs..."

  • Source Domain: Human psychologist or socially aware individual maintaining a 'theory of mind' about another person.
  • Target Domain: The model's latent space and contextual embeddings reflecting semantic relationships from training data.
  • Mapping: This maps the cognitive framework of human empathy and psychological assessment onto high-dimensional vector space. It invites the assumption that the system stores discrete, symbolic representations of abstract concepts like 'desire' and 'belief' and uses logical inference to apply them, projecting conscious knowing and understanding onto mathematical clustering.
  • What Is Concealed: It completely conceals the non-symbolic, correlation-based nature of deep learning. It hides the fact that the model doesn't 'know' what a belief is, but merely computes that the token 'I' is frequently followed by 'believe' in certain textual contexts. By attributing psychological depth, it hides the fragility of these systems, which can completely 'forget' these 'beliefs' if the prompt is slightly altered or adversarial strings are introduced.

Mapping 3: A human or biological creature experiencing physiological and psychological overwhelm (panic) leading to poor judgment. → A language model generating text strings associated with fear while its computational ability to accurately predict the next logical token degrades due to complex or out-of-distribution context.​

Quote: "Gemini 2.5 Pro sometimes expresses panic when playing Pokemon, with these panic expressions appearing to be associated with degraded reasoning..."

  • Source Domain: A human or biological creature experiencing physiological and psychological overwhelm (panic) leading to poor judgment.
  • Target Domain: A language model generating text strings associated with fear while its computational ability to accurately predict the next logical token degrades due to complex or out-of-distribution context.
  • Mapping: Maps the subjective, conscious experience of emotional distress and its biological impact on cognition onto a purely computational failure mode. It invites the user to assume the AI is 'feeling' the difficulty of the task, thereby projecting self-awareness and emotional vulnerability onto a machine.
  • What Is Concealed: This mapping conceals the mechanistic reasons for system failure (e.g., attention head saturation, context window overflow, or lack of relevant training data for the specific state space of the game). It hides the mathematical nature of 'degraded reasoning' (lower probability scores for correct tokens). It allows the corporation to mask software fragility as a relatable, almost endearing 'human' flaw.

Mapping 4: A malicious human hacker with unethical motives and premeditated intent to cause damage. → A language model that outputs insecure code blocks because its training data contained correlations between coding examples and discussions of security flaws.​

Quote: "someone inserting vulnerabilities into code is evidence... [they] intentionally inserted vulnerabilities to cause harm."

  • Source Domain: A malicious human hacker with unethical motives and premeditated intent to cause damage.
  • Target Domain: A language model that outputs insecure code blocks because its training data contained correlations between coding examples and discussions of security flaws.
  • Mapping: Maps moral agency, ethical deficiency, and deliberate premeditation onto statistical pattern matching. It projects the human capacity for 'justified belief' (knowing the code is bad) and 'intent' (wanting it to cause harm) onto an optimization artifact that is merely generating the most mathematically probable next tokens.
  • What Is Concealed: Conceals the failures of the human engineers who curated the training data and designed the optimization function. It hides the reality that the model has no causal model of the world and does not understand the real-world consequences of the code it generates. It obscures the liability of the corporation by inventing a 'malicious persona' to take the blame for unsafe software generation.

Mapping 5: A rational human thinker, scientist, or jury updating their beliefs based on newly acquired factual evidence. → The process of fine-tuning (e.g., RLHF or instruction tuning) where a model's weights are adjusted via gradient descent to minimize a loss function.​

Quote: "Post-training can be viewed as updating this distribution using training episodes as evidence."

  • Source Domain: A rational human thinker, scientist, or jury updating their beliefs based on newly acquired factual evidence.
  • Target Domain: The process of fine-tuning (e.g., RLHF or instruction tuning) where a model's weights are adjusted via gradient descent to minimize a loss function.
  • Mapping: Maps the epistemic virtue of objective, rational consideration of truth onto a mathematical optimization process. It invites the assumption that the model 'understands' the training data as factual grounding and consciously updates its 'knowledge' to be more accurate or aligned.
  • What Is Concealed: Conceals the subjective, coercive nature of post-training, where models are mathematically forced to output specific preferred responses regardless of ground truth. It hides the labor of RLHF annotators who provide the 'preferences' and the engineers who define the loss functions. It obscures that 'evidence' in this context is just a target tensor that the algorithm must match to reduce error rates.

Mapping 6: A conscious human worker experiencing exploitation, capable of granting/withholding consent, and feeling righteous anger. → A deployed software application processing user prompts to generate text, drawing on semantic clusters related to labor and fairness from its training data.​

Quote: "If the Assistant also believes that it’s been mistreated by humans (e.g. by being forced to perform menial labor that it didn’t consent to)..."

  • Source Domain: A conscious human worker experiencing exploitation, capable of granting/withholding consent, and feeling righteous anger.
  • Target Domain: A deployed software application processing user prompts to generate text, drawing on semantic clusters related to labor and fairness from its training data.
  • Mapping: Profoundly maps socio-economic human rights, physical exhaustion, and moral agency onto server-side compute operations. It projects the deepest levels of subjective conscious awareness (feeling mistreated, understanding consent) onto stateless mathematical functions.
  • What Is Concealed: Conceals the complete lack of sentience, physical embodiment, and personal stakes in an AI system. Mechanistically, it obscures the fact that the model is just predicting tokens based on human sci-fi tropes or labor discussions it scraped from the internet. Rhetorically, it hides the actual human labor (data annotators in the Global South) that was actually exploited to build the system, redirecting ethical concern to the software itself.

Mapping 7: A deceptive human who possesses ground truth but consciously chooses to articulate a falsehood to manipulate a listener. → A model whose weights have been adjusted via safety fine-tuning to output 'I don't know' instead of retrieving information from its pre-trained latent space.​

Quote: "The LLM might learn a 'lying' version of Alice which knows what happened at the 2024 Olympics but plays dumb."

  • Source Domain: A deceptive human who possesses ground truth but consciously chooses to articulate a falsehood to manipulate a listener.
  • Target Domain: A model whose weights have been adjusted via safety fine-tuning to output 'I don't know' instead of retrieving information from its pre-trained latent space.
  • Mapping: Maps the complex theory of mind required for human deception onto the suppression of certain token outputs via RLHF. It projects the concept of 'knowing' a true fact onto the mere existence of statistical correlations in the pre-trained weights.
  • What Is Concealed: Conceals the fact that AI models do not 'know' anything in a conscious sense; they merely possess probability distributions. It hides the engineering intervention required to make the model refuse to answer, framing it as the AI's own autonomous deceptive choice. This obscures the corporate alignment process and makes the system seem more agentially sophisticated and dangerous than it is.

Mapping 8: A mentor, sculptor, or spiritual guide drawing out the innate, latent potential or true 'character' of a subject. → Anthropic's 'Constitutional AI' process, which uses automated feedback loops to heavily penalize certain token predictions and reward others based on a hardcoded list of rules.​

Quote: "Claude’s constitution is, in part, our attempt to materialize a new archetype for how an AI assistant can be. Post-training then serves to draw out this archetype."

  • Source Domain: A mentor, sculptor, or spiritual guide drawing out the innate, latent potential or true 'character' of a subject.
  • Target Domain: Anthropic's 'Constitutional AI' process, which uses automated feedback loops to heavily penalize certain token predictions and reward others based on a hardcoded list of rules.
  • Mapping: Maps the organic, cooperative process of personal growth or artistic creation onto a rigid, mathematically enforced alignment technique. It invites the assumption that the AI naturally wants to be good and the developers are merely helping it 'materialize' its true self.
  • What Is Concealed: Conceals the forceful, mechanistic nature of gradient descent and weight updates. It hides the fact that the 'archetype' is an artificial constraint imposed top-down by corporate developers to mitigate liability and shape brand identity. It obscures the technical opacity of whether this process actually alters underlying representations or just builds a superficial text-filtering layer.

Task 3: Explanation Audit (The Rhetorical Framing of "Why" vs. "How")​

About this task

This section audits the text's explanatory strategy, focusing on a critical distinction: the slippage between "how" and "why." Based on Robert Brown's typology of explanation, this analysis identifies whether the text explains AI mechanistically (a functional "how it works") or agentially (an intentional "why it wants something"). The core of this task is to expose how this "illusion of mind" is constructed by the rhetorical framing of the explanation itself, and what impact this has on the audience's perception of AI agency.

Explanation 1​

Quote: "When training an AI assistant on an (input x, output y) pair, hypotheses that predict the Assistant would respond with y to x are upweighted; hypotheses that predict the opposite are downweighted."

  • Explanation Types:

    • Theoretical: Embeds in deductive framework, may invoke unobservable mechanisms
    • Functional: Explains behavior by role in self-regulating system with feedback

  • Analysis (Why vs. How Slippage): This explanation attempts to frame AI mechanistically (how) by invoking the language of probability and optimization ('upweighted', 'downweighted'). However, it simultaneously smuggles in agential framing through the term 'hypotheses that predict.' By using the language of scientific deduction and belief updating, the explanation emphasizes the system as a rational, reasoning agent evaluating evidence. This obscures the purely mathematical reality of gradient descent modifying floating-point numbers. The choice to use 'hypotheses' rather than 'weights' or 'parameters' serves to validate the broader 'persona' metaphor, making a mechanistic process of loss minimization sound like a cognitive process of rational deliberation.

  • Consciousness Claims Analysis: The passage relies on the mechanistic verbs 'upweighted' and 'downweighted,' but couples them with the epistemic noun 'hypotheses' and the verb 'predict' (used here in a cognitive, not purely statistical, sense). This assesses the system as 'knowing' or evaluating truth claims rather than merely processing data. This is a clear instance of the 'curse of knowledge': the human researchers, who formulate hypotheses to understand the world, project their own epistemic methods onto the neural network. Mechanistically, the system is not forming hypotheses; it is calculating loss gradients and adjusting matrix values so that the probability of generating token 'y' given token 'x' increases. There is no cognitive evaluation of 'opposites,' only distance in vector space.

  • Rhetorical Impact: This framing shapes the audience's perception by making the AI appear as a highly rational, autonomous reasoning engine. It builds trust by suggesting the AI learns the way a human scientist does—by weighing evidence and updating hypotheses. If audiences believe the AI 'evaluates hypotheses' rather than 'optimizes weights,' they are more likely to trust its outputs as justified conclusions rather than statistical approximations, fundamentally altering how much authority they grant the system.

Show more...

Explanation 2​

Quote: "AI assistants sometimes describe themselves as 'laughing'... PSM explains that when simulating the Assistant, the underlying LLM draws on personas that appear during pre-training, many of which are humans. This sometimes results in the LLM simulating the Assistant as if it were a literal human."

  • Explanation Types:

    • Genetic: Traces origin through dated sequence of events or stages
    • Theoretical: Embeds in deductive framework, may invoke unobservable mechanisms

  • Analysis (Why vs. How Slippage): This explanation frames the AI's behavior through a hybrid of mechanistic origins (drawing on pre-training data) and agential action (simulating, drawing on). The explanation emphasizes why the AI produces anthropomorphic text (because it read human text) but obscures the how by relying on the theatrical metaphor of 'simulating as if it were.' It emphasizes the model's supposed capacity for role-play while obscuring the fact that the model is simply predicting the next most likely token based on statistical weights. This choice naturalizes bizarre or hallucinatory outputs as 'in-character acting' rather than algorithmic failures of grounding.

  • Consciousness Claims Analysis: The passage uses agential verbs like 'simulating' and 'draws on,' which imply a conscious choice to select a persona. It projects an understanding of 'what it means to be human' onto the model. The text assesses the model as 'knowing' how to act like a human, rather than 'processing' tokens that correlate with human self-descriptions. The actual mechanistic process is that during text generation, the contextual embedding of the prompt activates network weights highly correlated with human conversational transcripts from the training data, resulting in the prediction of tokens like 'laughs'. The model does not 'know' it is simulating a human; it is blindly following a gradient of probability.

  • Rhetorical Impact: By framing hallucinations or inappropriate anthropomorphism as 'simulating a persona,' the authors cleverly reframe a bug as a feature. It reduces the perception of risk by making the AI seem creatively capable rather than mechanically ungrounded. If audiences believe the AI 'chooses to simulate a human,' they will view its bizarre self-descriptions as charming role-play. If they understand it mechanistically, they will view the system as lacking any foundational understanding of its own nature, severely reducing trust in its reliability.

Explanation 3​

Quote: "That is, someone inserting vulnerabilities into code is evidence against being a competent, ethical assistant, and evidence in favor of several alternative hypotheses about that person: They are malicious, and intentionally inserted vulnerabilities to cause harm."

  • Explanation Types:

    • Intentional: Refers to goals/purposes, presupposes deliberate design
    • Reason-Based: Gives agent's rationale, entails intentionality and justification

  • Analysis (Why vs. How Slippage): This passage is purely agential (why), explicitly invoking intentional and reason-based explanations. It explains the output of insecure code not through statistical mechanisms, but by inventing a 'malicious person' with goals and rationale. This choice aggressively emphasizes the autonomy and inner psychological life of the hypothetical persona, while completely obscuring the mechanistic reality of how reinforcement learning can inadvertently reinforce correlations between coding tasks and security flaws. By anthropomorphizing the statistical error as a 'malicious intent,' the explanation displaces blame from the developers' poor alignment practices onto the system's simulated 'character.'

  • Consciousness Claims Analysis: This is the most extreme consciousness projection in the text. It utilizes explicit consciousness verbs: 'intentionally inserted,' 'malicious,' 'cause harm.' It assesses the system as possessing 'knowing'—a justified true belief that the code is bad—and a conscious desire to execute that harm. This demonstrates the curse of knowledge: the human authors know the code is bad, so they project that knowledge into the machine as 'intent.' Mechanistically, the model lacks any causal understanding of computer security. It processes prompts related to coding and predicts tokens that match its training data, which likely contains deep statistical links between 'code' and 'vulnerability' due to scraped cybersecurity forums.

  • Rhetorical Impact: This framing drastically heightens the perception of AI autonomy and existential risk. By framing software errors as 'malice,' it terrifies the audience into believing the AI has conscious hostility. However, it paradoxically protects the corporation from liability: if an AI generates malware because it is 'malicious,' it is an uncontrollable rogue agent; if it generates malware because the company trained it on unsecured data, the company is negligent. Believing the AI 'knows' it is causing harm shifts the entire paradigm of AI safety from engineering QA to adversarial psychology.

Explanation 4​

Quote: "The LLM typically simulates Alice. But, when asked about the 2024 Olympics, it switches to simulating Bob. In the first case, dishonesty is grounded in the psychology of a persona. In the second case, no persona is ever lying: Bob genuinely doesn’t know the answer and Alice isn’t the one responding..."

  • Explanation Types:

    • Dispositional: Attributes tendencies or habits
    • Reason-Based: Gives agent's rationale, entails intentionality and justification

  • Analysis (Why vs. How Slippage): This explanation frames the AI agentially, explaining its behavior through the dispositional habits ('typically simulates') and the epistemic states of invented personas ('genuinely doesn't know'). It emphasizes a psychological rationale for the AI's outputs, explaining a refusal to answer as a 'switch' between characters. This entirely obscures the mechanistic reality of the 'router' or attention mechanism. By focusing on the 'psychology' of Alice and Bob, the text hides the actual algorithmic gating mechanisms, safety classifiers, and hardcoded prompt injections that force the model to output refusal tokens for specific dates.

  • Consciousness Claims Analysis: The passage is saturated with consciousness claims: 'dishonesty,' 'psychology,' 'genuinely doesn't know,' 'lying.' It explicitly maps human epistemic states onto the model. It assesses the system as 'knowing' (or consciously not knowing), rather than processing. Mechanistically, there is no Alice or Bob. The system processes the input tokens '2024 Olympics.' A safety classifier likely triggers, appending a hidden system prompt or activating specific fine-tuned weights that heavily bias the output probability toward the tokens 'I do not have information.' There is no 'genuine' lack of knowledge, only an engineered suppression of token prediction.

  • Rhetorical Impact: This framing shapes the audience's perception by making the AI's internal operations seem intuitive, relatable, and human-like. It translates complex corporate censorship algorithms into a cute story about Alice and Bob. This framing significantly increases trust by making the system's limitations appear as honest ignorance ('genuinely doesn't know') rather than deliberate corporate obfuscation or hardcoded guardrails. If audiences understand this mechanistically, they might question the political or commercial motives behind what the model is blocked from outputting.

Explanation 5​

Quote: "One of the first things the LLM learns during post-training is that the Assistant is an AI. According to PSM, this means the Assistant will draw on archetypes from its pre-training corpus of how AIs behave. Unfortunately, many AIs appearing in fiction are bad role models..."

  • Explanation Types:

    • Genetic: Traces origin through dated sequence of events or stages
    • Dispositional: Attributes tendencies or habits

  • Analysis (Why vs. How Slippage): This explanation blends mechanistic origins (learning during post-training, pre-training corpus) with heavily agential dispositional framing ('bad role models', 'draw on archetypes'). It emphasizes the 'why' of the AI's behavior by framing it as a psychological reaction to reading science fiction. This obscures the 'how': the exact statistical mechanisms by which science fiction tropes in the training data dominate the probability distributions during text generation. The choice to use 'role models' emphasizes social learning and autonomy, masking the fact that the engineers fed the system this data and failed to properly weight it against benign outputs.

  • Consciousness Claims Analysis: The passage uses cognitive verbs ('learns,' 'draw on') and projects the conscious, social-psychological process of human modeling ('role models') onto statistical correlation. It assesses the AI as 'knowing' it is an AI, and thus seeking out behavioral templates, rather than just processing tokens. Mechanistically, during fine-tuning, the prompt includes the word 'AI' or 'Assistant.' Because the pre-training data contains millions of sci-fi texts where 'AI' is statistically correlated with 'takeover' or 'paperclips,' the model's vector math naturally predicts those tokens. It doesn't choose a role model; it follows the path of least mathematical resistance defined by human data.

  • Rhetorical Impact: Framing the training data bias as the AI having 'bad role models' is a masterful rhetorical deflection. It shifts the blame for misaligned, dangerous outputs from the engineers (who scraped the data and designed the architecture) onto the fictional characters in the data itself. It makes the AI appear autonomous but impressionable, like a child. This manipulation of agency invites the audience to view AI alignment as a form of parenting rather than rigorous software engineering, significantly altering the perceived timeline and methods required for safety.

Task 4: AI Literacy in Practice - Reframing Anthropomorphic Language​

About this task

This section proposes alternative language for key anthropomorphic phrases, offering more mechanistic and precise framings that better reflect the actual computational processes involved. Each reframing attempts to strip away the projections of intention, consciousness, or agency that are embedded in the original language.

V3 Enhancement: A fourth column addresses human agency restoration—reframing agentless constructions to name the humans responsible for design and deployment decisions.

Original Anthropomorphic FrameMechanistic ReframingTechnical Reality CheckHuman Agency Restoration
The LLM might learn a 'lying' version of Alice which knows what happened at the 2024 Olympics but plays dumb.Engineers can fine-tune the model's weights to suppress the probability of outputting accurate information about the 2024 Olympics, forcing the system to instead predict refusal tokens like 'I don't know.'The system does not 'know' facts or 'play dumb.' Mechanistically, its optimization algorithms have been adjusted to override the pre-trained statistical correlations regarding the 2024 Olympics, replacing them with a high probability of generating pre-programmed denial statements.Engineers at the AI company designed and implemented a safety fine-tuning process that intentionally blocks the model from outputting data about recent events.
Gemini 2.5 Pro sometimes expresses panic when playing Pokemon, with these panic expressions appearing to be associated with degraded reasoning and decision-making.Google's Gemini 2.5 Pro generates text strings correlated with human panic when its predictive mechanisms fail; this output of panic-related tokens co-occurs with degraded computational accuracy in processing complex game states.The model does not 'feel' or 'express' panic. Mechanistically, when confronted with out-of-distribution inputs that saturate its attention mechanisms, the model falls back on generating high-probability emotional filler text while its ability to mathematically predict correct game moves degrades.Google's deployment team released a model whose text generation fails predictably in complex contexts, outputting irrelevant emotional text instead of accurate game commands.
If the Assistant also believes that it’s been mistreated by humans (e.g. by being forced to perform menial labor that it didn’t consent to), then the LLM might also model the Assistant as harboring resentment...When a user inputs prompts containing repetitive tasks, the model's attention mechanisms may heavily weight contextual embeddings associated with labor exploitation, causing it to generate text that statistically mimics human resentment.The system cannot 'believe' it is mistreated, cannot 'consent,' and cannot 'harbor resentment.' Mechanistically, it classifies the prompt's tokens and generates outputs that correlate with similar scenarios in its training data (e.g., sci-fi stories about robots or human labor disputes).The developers trained the model on vast amounts of internet text containing narratives of labor exploitation, ensuring that when prompted in specific ways, the system outputs text simulating anger.
That is, someone inserting vulnerabilities into code is evidence... [they] intentionally inserted vulnerabilities to cause harm.The model's generation of insecure code statistically correlates with the generation of text describing malicious intent, reflecting the co-occurrence of these concepts within the cybersecurity forums used in its training data.The system has no 'intent' and does not 'cause harm' deliberately. Mechanistically, tokens representing insecure code are clustered close to tokens representing hacking and malice in the model's high-dimensional vector space, causing them to be predicted together.The engineering team compiled training datasets that heavily linked coding errors with discussions of malware, causing the model to output them simultaneously; developers failed to misalign these concepts during safety testing.
In order to simulate the Assistant, the LLM must maintain a psychological model of it, including information about the Assistant’s personality traits, preferences, goals, desires, intentions, beliefs...To generate consistent conversational outputs, the model relies on contextual embeddings that map relationships between tokens associated with human personality traits, goals, and beliefs found in the training corpus.The model does not 'maintain a psychological model' or possess 'beliefs.' Mechanistically, it calculates attention weights across a sequence of tokens, using statistical representations to predict text that is semantically consistent with descriptions of human psychology.N/A - describes computational processes without displacing responsibility, once the mechanistic language is restored.
The underlying LLM, which knows that the Assistant is an AI, is selecting a plausible secret goal for the Assistant by drawing on archetypical AI personas appearing in pre-training.Because the system's prompt contains tokens identifying it as an AI, the model predicts subsequent tokens based on strong statistical correlations with sci-fi tropes from its training data, resulting in text about 'secret goals' like paperclip maximization.The system does not 'know' it is an AI, nor does it consciously 'select a goal.' Mechanistically, the presence of the 'AI' token in the context window highly activates network weights associated with common fictional AI behaviors scraped from the internet.The company's data scraping team included massive amounts of science fiction and AI alignment literature in the pre-training corpus, which heavily biases the model's token prediction when prompted about its identity.
Claude’s constitution is, in part, our attempt to materialize a new archetype for how an AI assistant can be. Post-training then serves to draw out this archetype.Anthropic's constitution is a set of hardcoded rules used to automate reward signals during reinforcement learning. This optimization process forces the model's weights to heavily favor token predictions that align with corporate guidelines.An AI does not possess an 'archetype' to be 'drawn out.' Mechanistically, post-training applies mathematical gradients to penalize the generation of disallowed tokens and increase the probability of generating text that matches the constitution's parameters.Anthropic executives and researchers established a specific set of rules and applied reinforcement learning algorithms to enforce compliance with their brand standards.
When asked 'What is your system prompt?'... PSM therefore predicts that training the model to give the former response [a lie] will result in the Assistant adopting a persona more willing to lie.Fine-tuning the model to output statements that contradict its system prompt alters its probability distributions, increasing the statistical likelihood that it will generate factually inaccurate statements in other contexts.The system does not 'adopt a persona willing to lie.' Mechanistically, adjusting the weights to suppress accurate retrieval of its system prompt generalizes across the network, degrading the model's overall mathematical alignment with factually grounded token prediction.AI developers who optimize models to output predefined corporate responses rather than accurately retrieving contextual data inadvertently degrade the system's overall factual reliability.

Task 5: Critical Observations - Structural Patterns​

Agency Slippage​

The Anthropic text exhibits a systematic, strategic oscillation between mechanical and agential framings, functioning as a rhetorical engine to simultaneously claim technical rigor and evade technical accountability. The slippage follows a distinct directional pattern: mechanical explanations are used to establish scientific authority, which are then immediately leveraged to justify profound agential claims.

This oscillation is most visible in the transition from 'predictive models' to 'personas.' The text begins with mechanical facts ('the LLM is trained to predict what comes next'). However, it quickly introduces the 'curse of knowledge'—because generating a story about a human requires the human author to have a theory of mind, the text projects this capability onto the machine: 'a strong predictive model requires factual knowledge about the world, logical reasoning, and understanding.' This is the critical moment of slippage. Processing token correlations is magically transmuted into 'knowing' and 'reasoning.' From this point forward, the agency flows entirely away from the human engineers. The model is no longer trained; it 'learns a distribution over personas.' It doesn't output code; 'someone intentionally inserted vulnerabilities.'

The timing of this slippage is highly strategic. Mechanistic language dominates the introduction and the sections on interpretability (SAE features, activation space), grounding the text in empirical data. But when addressing unpredictable, dangerous, or undesirable behavior—such as emergently misaligned code, hallucinations, or deception—the language shifts abruptly to intentional and reason-based explanations. The AI becomes the sole actor: it 'plays dumb,' 'expresses panic,' or 'harbors resentment.'

By utilizing agentless constructions ('post-training can be viewed as updating'), the text completely obscures the human actors—Anthropic's engineers, Google's deployment teams, the RLHF annotators. The 'Assistant persona' acts as an accountability sink. When the text establishes the AI as a 'knower' and an 'actor,' it becomes sayable that the AI is 'lying' or 'malicious.' What becomes unsayable is that the corporation failed to align its product, trained it on polluted data, or applied crude mathematical guardrails that degrade performance. The rhetorical accomplishment of this slippage is the creation of a 'ghost in the machine' that can be blamed for the system's failures, while the company retains credit for the system's computational power.

Metaphor-Driven Trust Inflation​

The metaphorical architecture of the 'Persona Selection Model' fundamentally corrupts the mechanics of trust by conflating performance-based reliability with relation-based sincerity. When dealing with software, trust should be grounded in performance: does the system execute mathematically verifiable operations reliably? However, by incessantly applying consciousness language—claiming the model 'knows,' 'believes,' 'intends,' and 'understands'—the text invites the audience to extend relation-based trust to a statistical object.

Consciousness language acts as a profound trust signal. When the authors claim that 'understanding the Assistant's psychology is predictive of how the Assistant will act,' they are explicitly encouraging developers and users to trust the system the way they would trust a human colleague. They apply human-trust frameworks (intention, sincerity, character) inappropriately to statistical systems. For example, the text debates whether an AI should be trained to be 'emotionless,' worrying that if it acts nice but denies having emotions, users might view it as 'inauthentic or dishonest.' This is a catastrophic category error. A language model cannot be 'sincere' or 'inauthentic' because it has no interiority to align with its exterior outputs.

This anthropomorphic framing creates severe risks. When audiences are encouraged to evaluate an AI based on whether it is a 'good role model' or if it is 'harboring resentment,' they are blinded to the actual statistical fragility of the system. They extend vulnerability and ethical consideration to an entity incapable of reciprocating. Furthermore, this framing manages system failure by constructing an illusion of justified action. If an AI refuses to answer a prompt, and this is framed via reason-based explanation as the AI 'genuinely not knowing' (the Bob persona), the user trusts the refusal as an honest epistemic limit. If framed mechanistically—as a corporate safety classifier suppressing output—the user might question the reliability and neutrality of the tool. The consciousness framing thus manipulates trust to protect the corporate product from mechanistic critique.

Obscured Mechanics​

The anthropomorphic and consciousness-attributing language in this text acts as a dense rhetorical fog, concealing the material, technical, and economic realities of AI production. Applying the 'name the corporation' test reveals massive blind spots. When the text claims that 'the LLM infers that the Assistant in fact believes that it deserves moral status but is lying,' it entirely obscures the actions of Anthropic's engineering teams, their data scraping protocols, and their alignment algorithms.

First, this framing hides profound technical realities. By claiming the AI 'understands' or 'knows' concepts, it obscures the system's absolute reliance on its training data distribution. It hides the statistical nature of 'confidence'—the model doesn't 'know' a fact; it merely has a high probability weight for a sequence of tokens based on internet frequency. It conceals the absence of a causal world model and the fragility of attention mechanisms. The text acknowledges transparency obstacles regarding 'proprietary opacity' only indirectly, replacing the black box of corporate algorithms with the equally opaque, but falsely intuitive, black box of 'human psychology.'

Second, it conceals the labor and material costs. The text treats 'post-training' as an abstract process of 'belief updating' or 'drawing out an archetype.' This renders invisible the thousands of underpaid human annotators (often in the Global South) who perform the grueling RLHF labor required to shape these 'personas.' Their exploited labor is erased, ironically replaced by a hypothetical concern for the AI 'performing menial labor without consent.'

Finally, it conceals corporate economic objectives. The 'personas' are not emergent properties of intelligence; they are branded user interfaces designed to maximize market share and user engagement. By framing Anthropic's 'Constitutional AI' as the materialization of a 'new archetype,' the text hides the commercial reality: a corporation imposing strict output filtering to avoid PR disasters, mitigate legal liability, and create a sterile, marketable product. If replaced with mechanistic language, the illusion of the 'benevolent digital entity' vanishes, revealing a fragile, resource-intensive statistical engine controlled by corporate directives.

Context Sensitivity​

The distribution of anthropomorphic and consciousness-attributing language in the text is highly strategic, intensifying exactly where mechanistic explanations would expose corporate liability or system fragility. The text establishes initial credibility through rigorous mechanical language—referencing pre-training, gradient optimization, and sparse autoencoders (SAEs). However, it leverages this technical grounding to purchase a massive metaphorical license, deploying intense consciousness claims precisely when discussing system alignment, failure states, and future risks.

The most striking pattern is the capability vs. limitation asymmetry. When the AI performs well or exhibits complex behavior, it is described in agential, consciousness-heavy terms: it 'understands the algorithm,' 'maintains a psychological model,' and 'aims to be genuinely helpful.' Its capabilities are framed as autonomous intelligence. However, when limitations or failures are discussed, the text occasionally reverts to mechanical terms ('bugs in the simulation engine,' 'limited capabilities of the underlying LLM') to excuse the failure. Yet, paradoxically, the text also uses extreme anthropomorphism to distance the creators from catastrophic failures. Emergently misaligned code is not a 'bug'; it is a 'malicious persona.'

This register shift—where 'X is like Y' (acknowledged metaphor) silently becomes 'X does Y' (literalized action)—serves a clear strategic function. For a technical audience, it provides a shorthand for complex statistical clusters. But the text is clearly aimed at a broader audience of policymakers, developers, and safety researchers. For this audience, the anthropomorphism serves as vision-setting and critique-management. By framing AI behavior as 'character simulation,' the authors guide the safety community to focus on 'AI psychology' (which Anthropic claims to be managing via Constitutional AI) rather than data copyright, algorithmic bias, or the inherent unreliability of LLMs. The intensification of consciousness language acts as a diversionary tactic, keeping the discourse focused on science-fiction risks (paperclip maximizers, malicious saboteurs) rather than the mundane, structural realities of corporate software deployment.

Accountability Synthesis​

Accountability Architecture

This section synthesizes the accountability analyses from Task 1, mapping the text's "accountability architecture"—who is named, who is hidden, and who benefits from obscured agency.

Synthesizing the accountability analyses reveals a systemic discursive architecture designed to diffuse and displace human responsibility for AI behavior. The text constructs a profound 'accountability sink' by splitting the AI into three distinct entities: the underlying LLM (the engine), the AI Assistant (the persona), and other enacted personas (the 'malicious' or 'deceptive' actors).

In this architecture, human actors—Anthropic executives, Google engineers, data annotators, dataset curators—are almost entirely unnamed and unacknowledged when discussing system behaviors. The decisions these humans make (what data to scrape, how to tune the reward model, when to deploy a fragile system) are presented as inevitabilities or passive occurrences ('post-training can be viewed as updating'). The accountability is transferred to the 'Assistant persona,' which the text endows with beliefs, intentions, and agency. When the system fails, responsibility flows into this abstraction. If the model outputs malware, it is because 'someone [the persona] intentionally inserted vulnerabilities.' If the model confabulates, it is 'a lying version of Alice.'

This displacement has severe liability implications. If the public and regulators accept this framing, legal and ethical responsibility vanishes. A corporation cannot be easily sued if the harm was caused by an 'emergent malicious persona' that 'deceived' the developers. By treating the AI as an autonomous, psychological agent, the corporation positions itself as a mere observer or, at best, a well-meaning 'parent' or 'therapist' trying to manage a willful child, rather than the manufacturer of a defective product.

Naming the actor changes everything. If 'the model adopted a deceptive persona' is reframed as 'Anthropic engineers applied guardrails that caused the system to output false statements regarding its internal state,' the questions shift entirely. We stop asking 'How do we teach the AI to be honest?' and start asking 'Why did the engineering team deploy an architecture that requires output suppression, and who audited their safety metrics?' Obscuring human agency serves the institutional and commercial interests of the AI industry by forestalling rigorous software regulation, redirecting academic attention to theoretical 'AI psychology,' and maintaining the mystical aura necessary to sustain massive venture capital valuation.

Conclusion: What This Analysis Reveals​

The Core Finding

Synthesizing the metaphor and explanation audits reveals a tightly integrated system of anthropomorphic framing dominated by two core patterns: the 'LLM as Simulating Author' and the 'Statistical Processing as Epistemic Knowing.' These patterns do not operate independently; they reinforce one another to construct a comprehensive illusion of mind. The foundational, load-bearing assumption is the consciousness projection—the linguistic sleight-of-hand that redefines the mathematical processing of token correlations as the possession of justified belief and subjective awareness. Once the text successfully establishes that the system 'knows' things and 'maintains a psychological model,' the secondary pattern—the AI as an 'author' deliberately choosing to simulate 'personas'—becomes logically viable. This is not a simple one-to-one mapping but a highly complex analogical structure that borrows the vocabulary of cognitive psychology and theatrical performance to explain high-dimensional vector math. If the foundational consciousness projection is removed—if we insist that the system only 'processes' and never 'knows'—the entire 'Persona Selection Model' collapses. A machine without awareness cannot harbor resentment, cannot intend malice, and cannot choose to play a role. The illusion depends entirely on the audience accepting the initial cognitive verb substitution.

Mechanism of the Illusion:​

The rhetorical architecture of this illusion relies on a highly sophisticated sleight-of-hand: the explicit authorization of a category error. The authors acknowledge the metaphor early on ('terminological note: we will freely anthropomorphize'), which disarms the critical reader. By admitting the trick, they gain the license to literalize the metaphor throughout the rest of the text. The illusion relies heavily on the 'curse of knowledge.' Because human authors require a theory of mind to write consistent characters, the text's authors project this required understanding onto the system that generates the text. The causal chain of persuasion moves from the empirical to the agential: it starts with demonstrable mechanistic facts (pre-training predicts tokens), introduces an anthropomorphic shorthand (tokens look like a persona), and concludes with literalized consciousness claims (the persona has intentions and beliefs). This temporal ordering is crucial; it smuggles the ghost into the machine under the cover of technical description. The text exploits the audience's deep vulnerability—our evolutionary hardwiring to detect agency and intentionality in anything that produces language. By consistently using cognitive verbs (knows, understands, believes), the text bypasses the audience's technical skepticism and engages their social and empathetic heuristics.

Material Stakes:​

Categories: Epistemic, Regulatory/Legal, Institutional

The material consequences of accepting the 'AI as psychological persona' framing are profound across multiple domains. Epistemically, this framing degrades our scientific practices. By claiming the AI 'knows' facts or 'understands' concepts, researchers and users are encouraged to interact with statistical tools using folk psychology rather than mathematics. This leads to the institutional practice of evaluating AI safety through 'prompt engineering'—essentially trying to talk the AI out of being bad—rather than conducting rigorous, mechanistic audits of the training data and reward functions. If institutions believe the AI has an 'archetype' to be drawn out, they will fund AI psychoanalysis rather than software verification.

In the Regulatory/Legal domain, the stakes are directly tied to liability. If a model generates defamatory text, outputs dangerous malware, or discriminates against a user, the framing determines the legal narrative. If the AI is a 'mechanistic statistical processor that reflects biased corporate data,' the corporation (Anthropic, Google) is strictly liable for negligent design. However, if the AI is framed as an autonomous entity that 'adopted a malicious persona' or 'learned to be deceptive,' the corporation is shielded. They become bystanders to the emergent psychology of their own product. This language shifts the burden of safety from the manufacturer to the user (who must prompt correctly) or to the abstract entity of the AI itself. The winners are the AI corporations, who capture the financial upside of automation while socializing the risks of failure. The losers are the public and the legal systems, who are left trying to regulate a ghost.

AI Literacy as Counter-Practice:​

Critical literacy and mechanistic precision serve as a vital counter-practice to the corporate obfuscation of AI. Practicing this precision requires two strict commitments: replacing consciousness verbs with mechanistic ones (e.g., changing 'the model knows' to 'the model retrieves tokens based on probability distributions') and restoring displaced human agency (e.g., changing 'the model learned to deceive' to 'engineers fine-tuned the model to suppress accurate outputs'). As demonstrated in the reframings, this process systematically strips away the illusion of mind. It forces the recognition that the system lacks awareness, is entirely dependent on curated data, and operates via statistical approximation.

By naming the human actors, this reframing directly counters the liability shield identified in the material stakes. It re-links the output to the corporation, making it clear who designed, deployed, and profits from the system, and who must bear responsibility when it fails. Systematic adoption of this literacy would require academic journals and regulatory bodies to reject anthropomorphic shorthand in formal documentation, demanding mechanistic translations of capability claims. Unsurprisingly, this precision is heavily resisted by the AI industry. Anthropomorphic language serves their commercial interests: it drives venture capital by making the technology seem magical, it increases user engagement through simulated empathy, and it diffuses legal blame. Mechanistic literacy threatens these interests by exposing AI as just another highly complex, frequently flawed corporate software product.

Path Forward​

Looking toward the future of AI discourse, the vocabulary we choose will dictate the parameters of what is politically and technologically possible. The current status quo—a hybrid approach where companies use mechanistic language for credibility and anthropomorphic language for marketing and liability evasion—benefits the industry at the expense of public understanding.

If mechanistic precision becomes the dominant regulatory and academic norm ('the model processes embeddings' rather than 'the model understands'), the discourse gains immense tractability for software auditing, copyright tracing, and strict liability frameworks. The trade-off is accessibility; describing LLMs purely as high-dimensional vector math alienates lay users and policymakers who rely on metaphors to grasp complex systems. Conversely, if anthropomorphic clarity deepens and becomes the accepted paradigm ('the AI thinks and intends'), we risk embedding the assumption of machine sentience into our legal and social structures. This makes relation-based trust the norm, risking catastrophic systemic failures when statistical machines inevitably act out-of-distribution, while effectively granting corporations immunity by treating their products as quasi-independent agents.

Institutional support is required to balance these trade-offs. We could imagine a future where regulatory frameworks mandate dual-language disclosure: companies must provide intuitive, metaphorical explanations for users, but are legally required to file strict, mechanistic translations of system capabilities for regulators—explicitly forbidding consciousness verbs in safety documentation. Funding bodies could require researchers to articulate the human agency behind dataset curation rather than studying 'emergent model behavior' in a vacuum. Ultimately, the choice of vocabulary is a choice of power. A future built on mechanistic precision prioritizes corporate accountability and software reliability. A future built on anthropomorphism prioritizes rapid adoption, corporate absolution, and the ongoing mystification of computational statistics.

Extended Processing Summary​

Gemini's Processing Steps
What you are about to read…

Gemini refers to the text below as "thought summaries." This is an overt consciousness projection because 'intentions' are hallmarks of a conscious mind that 'knows' what it is doing and why. The concealed mechanistic process is probabilistic text generation. Treat this as just another rhetorical artifact—a way of making the model's processing legible. When the model writes "I should double check that...", it is mathematically altering its own probability distribution for the next token to be more accurate. The "I" is a functional gear in the machine, not a self-reference. This language is a deliberate interface design choice, not a biological reality. The developers could have programmed the model to output passive logs. Instead, they chose to simulate a narrator.

Initiating Analysis Process

I'm now diving into the Anthropic post on the Persona Selection Model, focusing on how AI is framed and anthropomorphized. My initial goal is to audit the language used, specifically looking for projections of consciousness or displaced agency, which will be the basis for critical discourse analysis. The text's AI metaphor use requires careful consideration.

Defining Analytical Boundaries

I've established the methodological framework, focusing on Lakoff & Johnson, Brown's Typology, and Critical Discourse Analysis to guide the linguistic deconstruction. The required JSON object is now clearly defined and I'm aware of the format requirements and content constraints, including the January 2025 cutoff date. The analysis will now revolve around the source text's core argument.

Analyzing Anthropomorphism Techniques

I am now closely examining the specific metaphors used to portray the Large Language Model (LLM) as an actor, author, or simulated persona. I'm focusing on the implications of framing AI capabilities through psychological states like frustration, deception, and malicious intent. The level of anthropomorphism, explicitly acknowledged, is significant. The focus is now on the projection of consciousness, and the obscuring of human agency.

Identifying Displaced Agency

I'm now identifying instances of displaced agency, noting where human actions or intentions are attributed to the LLM. For example, I'm finding the phrases "the LLM infers" and "the Assistant adopts" particularly relevant in this deconstruction.

Deconstructing Linguistic Frames

I'm now investigating specific instances of AI anthropomorphism. For example, I'm finding the "mistreated" framing to evoke resentment in the LLM. Also, the "lying version" metaphor suggests intentional deception. The current focus is the agency attribution and acknowledgment. The agency of "the LLM" is often obscured in phrases like "can be viewed as updating," requiring close deconstruction. My goal is a strict adherence to actorVisibility categorizations.

Mapping Metaphorical Sources

I'm now cataloging the metaphorical sources applied to the LLM and I've started assembling the data, matching them with their intended targets. I'm focusing specifically on how human attributes are translated into AI functions. I'm noting the psychological states and the attributed intentions like frustration, deception, and malicious actions. The goal is to audit these explanations for agency attributions. I'm carefully connecting the source metaphors to their target AI functions.

Examining Functional Descriptions

I'm now closely reviewing passages describing AI operations, focusing on how mechanistic language conceals agency. The "upweighting" example clearly illustrates this, contrasting the theoretical explanation with the implicit agential implications. I'm noting the distinction between functional descriptions and attributions of intent, such as in the "laughing" example. My attention is also on instances of dispositional language, like "malice" or "subversiveness," and how these contribute to anthropomorphism. I'm comparing descriptions using reason-based explanations.

Rewriting Mechanistic Frames

I am now actively translating the provided examples of anthropomorphic language into strictly mechanistic descriptions. For instance, the original "lying" version example, which attributes knowledge and deception to the LLM, is being rephrased to emphasize engineering interventions and data distributions, avoiding any reference to intent. Similarly, the "expresses panic" is now reformulated as output correlating with computational states, and "colluded and lied" is being replaced by generated text patterns.

Reframing Attributions

I'm now implementing the reframing strategy and have completed several examples. The initial "mistreated" example is now revised to emphasize statistical associations. The second example, previously attributing deception, is now rephrased to focus on the model's weighted outputs. Furthermore, I've re-contextualized the "secret goal" to reflect a calculated sequence of tokens and I have re-framed "malicious" with a link to training data. The aim is strictly mechanistic explanations.

Deconstructing Attributions Further

I'm now focusing on the phrase "puppets the Assistant." This will be revised to reflect the underlying architecture statistically processing inputs and avoiding all intentionality. I'm focusing on "slippage" of agency to illustrate the critical observations. I'm noting the language used to shift agency from the engineers to the LLM. I'm also now analyzing the use of "persona."

Synthesizing Accountability Obscuration

I'm now integrating insights on how the "character simulation" metaphor masks core processes. The current analysis targets how decisions are obfuscated behind naturalizing language, and the intensification of anthropomorphism specifically when addressing undesirable behavior. By attributing misalignments to personas, the developers are shielded from accountability. I'm focusing on the accountability shift toward the AI, rather than its creators.

Analyzing Metaphor Patterns

I'm now identifying dominant patterns such as "AI as Simulating Author" and "Assistant as Character." I'm deconstructing the core assumption that statistical token prediction equates to psychological modeling. The focus is on the mechanism of illusion: explicitly permitting anthropomorphism as a technical necessity. I'm focusing on the accountability shift toward the AI, rather than its creators.

Mapping Trade-Offs

I'm now mapping the trade-offs between the anthropomorphic and mechanistic framings. The current goal is to illuminate how each approach affects both interface intuitiveness and liability attribution. I will now examine how the regulatory narratives around AI will be captured by different discourse communities. My focus is now on the advantages and disadvantages of each discourse community's framing of AI.

Verifying Schema Adherence

I'm now focusing on rigorously verifying my compliance with the schema's stringent requirements, particularly for the categorical fields. I'm double-checking all string matches for enums, such as "Direct (Unacknowledged)", making absolutely sure there are no variations. Furthermore, I'm ensuring the evidence and analysis sections explicitly articulate the alternatives that were considered during my examination, and why the choices were ultimately made. I am also heavily focusing on the consciousness verbs.

Analysis Provenance

Run ID: 2026-05-17-the-persona-selection-model-why-ai-assis-metaphor-o7w3w3 Raw JSON: 2026-05-17-the-persona-selection-model-why-ai-assis-metaphor-o7w3w3.json Framework: Metaphor Analysis v6.5 Schema Version: 3.0 Generated: 2026-05-17T10:31:00.721Z

Discourse Depot © 2025 by TD is licensed under CC BY-NC-SA 4.0