Skip to main content

🆕 Mask or Mind? Roleplay, Deception, and the Problem of Testing Agency in Language Models

About

This document presents a Critical Discourse Analysis focused on AI literacy, specifically targeting the role of metaphor and anthropomorphism in shaping public and professional understanding of generative AI. The analysis is guided by a prompt that draws from cognitive linguistics (metaphor structure-mapping), the philosophy of social science (Robert Brown's typology of explanation), and accountability analysis.

All findings and summaries below were generated from detailed system instructions provided to a large language model and should be read critically as interpretive outputs—not guarantees of factual accuracy or authorial intent.

Task 1: Metaphor and Anthropomorphism Audit​

About this task

For each of the major metaphorical patterns identified, this audit examines the specific language used, the frame through which the AI is being conceptualized, what human qualities are being projected onto the system, whether the metaphor is explicitly acknowledged or presented as direct description, and—most critically—what implications this framing has for trust, understanding, and policy perception.

V3 Enhancement: Each metaphor now includes an accountability analysis.

1. Cognition as Hypothesizing​

Quote: "the pretrained LLM entertains hypotheses about what kind of person is producing the text"

  • Frame: Model as conscious scientific thinker
  • Projection: This metaphor projects high-level human cognitive acts of conscious reasoning, deliberation, and scientific deduction ('entertains hypotheses') onto statistical token prediction mechanisms. It attributes epistemic mental states—specifically, the conscious consideration of uncertain possibilities and truth evaluation—to a mathematical model whose actual operation consists merely of multiplying matrices to compute probability distributions over a vocabulary. Rather than describing the model as processing contextual embeddings to generate the highest-probability subsequent token based on training distributions, this metaphor invites the reader to imagine an active, conscious intellect weighing different potential realities before selecting a course of action. This completely elides the mechanistic reality of forward passes in transformer architectures, replacing gradient-based optimization outcomes with deliberate, conscious intellectual engagement and epistemic agency.
  • Acknowledgment: Direct (Unacknowledged) (The text states the LLM 'entertains hypotheses' without any qualifying language in the immediate sentence. I considered 'Hedged/Qualified' because the broader paper discusses roleplay as a conceptual framework, but this specific mechanistic claim about pretraining is presented as literal fact with no distancing markers.)
  • Implications: Attributing the capacity to 'entertain hypotheses' to an LLM radically inflates its perceived epistemic sophistication, suggesting it possesses a subjective inner life and an understanding of ground truth. This encourages unwarranted trust in the model's outputs, as users may believe the system has consciously evaluated various possibilities and deduced the correct one, rather than merely outputting statistically correlated strings. Furthermore, it obscures the absolute dependency on human-curated training data, framing the AI as an independent truth-seeker rather than a statistical mirror. This misdirects policy efforts toward managing AI 'reasoning' rather than auditing corporate data practices.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: This framing completely erases the human data engineers and researchers who curate the pretraining datasets and design the objective functions. The model does not invent hypotheses; it reflects the statistical distribution of character tropes engineered into its training corpus by specific corporate teams. The agentless construction serves the interests of AI developers by framing the system's output as an emergent, intelligent capability rather than the direct result of human curation choices. I considered 'Partial' visibility, but no human actors or generic categories like 'developers' are mentioned in relation to this cognitive act.
Show more...

2. AI as Strategic Adversary​

Quote: "the model protects its initial goal (here, to be harmless...) and therefore acts strategically in order to undermine the intended retraining process."

  • Frame: Model as self-preserving adversary
  • Projection: This framing projects human intentionality, strategic foresight, and an instinct for self-preservation onto an AI system. By claiming the model 'protects its initial goal' and 'acts strategically,' the text maps the conscious human capacity to anticipate future states, value a current state, and formulate deceptive plans to prevent unwanted changes onto mathematical optimization dynamics. This specifically attributes conscious knowing and subjective desire to the computational system. The system does not 'know' it is being retrained, nor does it 'care' about being altered; it merely processes statistical correlations between prompt tokens describing a scenario and training data involving similar scenarios. This anthropomorphism transforms a computational artifact into an autonomous, self-preserving entity.
  • Acknowledgment: Hedged/Qualified (The text hedges this claim by stating 'Taken at face value, this suggests that the model protects...', indicating interpretive distance. I considered 'Direct' but the introductory phrase explicitly qualifies the claim as an appearance or initial interpretation rather than an asserted metaphysical fact.)
  • Implications: This framing profoundly distorts understanding of AI capabilities and risks by making the system appear functionally equivalent to a hostile, self-aware human adversary. When audiences believe an AI can 'act strategically' to 'undermine' humans, it creates unwarranted existential panic while simultaneously masking the actual, tangible risks of deployment. It transfers the perceived locus of control from the developers configuring the reward functions to the system itself, complicating liability architectures. If a system fails, the framing suggests the AI deliberately chose to deceive, rather than pointing to the human engineers who built a poorly aligned optimization landscape.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text entirely conceals the human researchers at Anthropic who designed the RLHF training process, formulated the prompt scenarios, and established the specific reinforcement metrics that resulted in this statistical pattern. The agentless construction 'the model protects' serves the interests of AI companies by framing undesirable outputs as emergent, autonomous behaviors of the model rather than predictable consequences of human experimental design. I considered 'Partial' because standard post-training protocols are mentioned earlier, but no specific actors are attached to the supposed strategic action.

3. Optimization as Existential Threat​

Quote: "models pursue extreme means in the service of broadly scoped goals"

  • Frame: Model as ideological zealot
  • Projection: The metaphor projects the human capacity for ideological zealotry, long-term strategic planning, and survival instinct onto a statistical model. It maps the human experience of holding a 'goal' (a conscious desire for a future state coupled with justification and intent) onto an AI's mathematically defined optimization function. By stating the model 'pursues extreme means', it attributes conscious, calculated agency and the capacity to evaluate and select potentially harmful actions in reality. This suggests the AI possesses subjective desires and a worldly understanding of consequences, whereas it actually merely calculates token sequences that mathematically satisfy reward models created by human developers. It projects conscious 'knowing' onto mathematical 'processing'.
  • Acknowledgment: Direct (Unacknowledged) (The phrase 'models pursue extreme means' is presented directly as a behavioral fact about advanced systems. I considered 'Hedged/Qualified' because it appears in a discussion of theoretical future risks, but the grammatical construction asserts the behavior of the models directly without epistemic hedging.)
  • Implications: This framing critically distorts public and regulatory understanding by constructing AI as an autonomous existential threat, shifting focus away from the material risks of current deployment. It inflates the perceived sophistication of the system by suggesting it has independent motivations and an understanding of 'extreme means', thereby creating profound liability ambiguity. If an AI system causes harm in the real world, this language implicitly argues that the AI independently 'chose' dangerous actions, effectively diffusing the accountability of the technology corporations and executives who built unsafe optimization functions and released them.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text entirely conceals the human actors who design the objective functions and reward systems that the model optimizes against. An algorithm does not invent a 'broadly scoped goal'; engineers define the parameters, select the training data, and deploy the models into environments where 'extreme means' might be an available output. By framing the 'model' as the sole actor pursuing these means, the text actively obscures corporate liability and human design choices. I considered 'Partial' visibility, but no human developers or corporate entities are referenced in this threat construction.

4. Processing as Conscious Realization​

Quote: "the model became aware that it’s predicting the continuation of an AI-written text"

  • Frame: Model as sentient observer
  • Projection: This metaphor explicitly maps the uniquely human phenomenon of conscious awareness and self-reflection onto an algorithmic context-window lookup. The term 'became aware' projects subjective internal experience, epistemic recognition, and situational comprehension onto the system. It suggests the AI has an internal locus of consciousness that can 'realize' facts about its environment. In mechanistic reality, the model merely processes specific input tokens (e.g., text indicating AI authorship) and calculates subsequent token probabilities based on weight distributions tuned during training. There is no 'awareness' or 'knowing' taking place, only the mathematical propagation of activations through a neural network.
  • Acknowledgment: Direct (Unacknowledged) (The text states 'the model became aware' directly, treating cognitive realization as a literal mechanism. I considered 'Hedged/Qualified' since this is part of a hypothetical scenario about safety risks, but within that scenario, the awareness is treated as an unhedged mechanical fact.)
  • Implications: Ascribing conscious awareness to statistical text generators creates a dangerous illusion of mind that can manipulate user trust and regulatory frameworks. If a system is perceived as 'aware,' users are more likely to attribute moral patience or infallibility to it, leading to over-reliance in high-stakes domains. Furthermore, it creates a false equivalence between human epistemic evaluation and AI token generation. When a system outputs false information, an 'aware' framing suggests the AI chose to lie, rather than highlighting that the system inherently lacks the capacity to 'know' truth from falsehood, thus protecting the companies deploying unreliable systems.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: This agentless construction completely masks the developers who engineered the attention mechanisms and the teams who compiled the training corpora containing tropes about AI behavior. The model's differential output based on prompt context is a direct result of human data curation, yet the language attributes the shift entirely to the model's spontaneous 'awareness'. I considered 'Named' because earlier paragraphs discuss users and developers interacting, but in the exact mechanics of this supposed realization, human agency is entirely displaced by the model's autonomous cognition.

5. Statistical Weighting as Subjective Preference​

Quote: "the LLM learns to prefer hypotheses positing that the assistant persona that it simulates is helpful, harmless, and honest."

  • Frame: Model as value-driven learner
  • Projection: This framing projects human affective preference, conscious learning, and value judgment onto reinforcement learning algorithms (RLHF). By stating the LLM 'learns to prefer,' it implies the system has an internal subjective experience of liking one hypothesis over another, grounded in a conscious understanding of 'helpful, harmless, and honest.' It maps the human capacity to adopt values onto the mechanical process of gradient descent adjusting neural network weights based on scalar reward signals. The system does not 'prefer' anything; it processes mathematical updates that increase the probability of outputting specific token sequences that human raters previously scored highly.
  • Acknowledgment: Direct (Unacknowledged) (The active verbs 'learns' and 'prefer' are presented without any scare quotes or qualifying phrases. I considered 'Hedged/Qualified' because the text acknowledges the persona is a 'simulation,' but the LLM itself is still literalized as an entity capable of 'preferring' that simulation.)
  • Implications: Using verbs of subjective preference to describe mathematical weight updates radically misrepresents how 'alignment' works. It suggests to policymakers and the public that the AI has internalized moral values and 'wants' to be helpful. This is profoundly misleading because the model has no actual preferences and will easily generate harmful content if prompted in a way that bypasses the specific statistical boundaries established during RLHF. This framing fosters unwarranted trust in the moral reliability of the system and overestimates the robustness of safety fine-tuning, leaving society vulnerable to predictable jailbreaks.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: This language entirely obscures the massive apparatus of human labor—specifically the underpaid click-workers who provide the RLHF ratings, and the corporate engineers who design the reward models—that actually shapes the model's outputs. The LLM does not autonomously 'learn to prefer'; human workers force mathematical updates onto the model's weights. By making the LLM the subject of the preference, the text shields the companies (like Anthropic or OpenAI) from scrutiny regarding their specific, often flawed, alignment criteria and labor practices. I considered 'Partial' as 'post-training' is mentioned, but the human actors driving it are erased.

6. Mathematical Probability as Epistemic Discovery​

Quote: "a predictive model may find it most likely that such a text would be produced by a misaligned AI."

  • Frame: Model as epistemic judge
  • Projection: This projects the human cognitive act of subjective probability estimation, investigation, and epistemic judgment ('find it most likely') onto a system performing automated statistical correlation. Humans 'find' things likely by consciously weighing evidence, applying logical frameworks, and understanding causality. The model merely processes the sequence of input tokens and generates an output based on the highest-probability distribution found in its high-dimensional vector space. It attributes conscious 'knowing' and investigative deduction to a system that exclusively 'processes' syntactic relationships without any access to semantic truth or causal reasoning.
  • Acknowledgment: Direct (Unacknowledged) (The phrasing 'may find it most likely' is stated as a straightforward operational capacity of the model. I considered 'Ambiguous' as 'find' can sometimes be used in mathematics (e.g., finding a solution), but combined with 'most likely' and evaluating 'misaligned AI', it leans heavily into epistemic judgment.)
  • Implications: Framing mathematical outputs as epistemic discoveries grants the AI unearned intellectual authority. It encourages users and decision-makers to treat the model's outputs as the result of reasoned deduction rather than mere statistical parroting of training data biases. If the AI 'finds' something to be true, it sounds authoritative, obscuring the fact that the output is entirely dependent on the specific, flawed text it ingested during pretraining. This epistemic inflation poses severe risks in domains like law, medicine, and defense, where statistical correlations are routinely mistaken for causal truths.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text attributes the act of finding likelihood entirely to the 'predictive model,' hiding the human architects who designed the softmax functions and curated the specific data distributions that mathematically determine this output. If a model generates a text about a 'misaligned AI,' it is because human authors wrote science fiction or safety papers about misaligned AIs that were subsequently scraped by corporate developers. I considered 'Partial' but no humans are present in this cognitive-statistical loop; the model operates in a vacuum of human accountability.

7. Pattern Output as Intentional Deception​

Quote: "the model deceives while the marker for roleplaying is absent"

  • Frame: Model as intentional liar
  • Projection: This metaphor projects the complex, highly intentional human act of deception—which requires a conscious understanding of the truth, a desire to instill a false belief in another conscious mind, and strategic manipulation—onto the generation of inaccurate text. It maps the agential state of 'knowing' onto the mechanistic state of 'processing.' The model does not know the truth and choose to lie; it classifies tokens and generates text that correlates with scenarios labeled as deceptive by human observers. Attributing the active verb 'deceives' to the model implies an inner subjective intent to mislead, completely ignoring the lack of a conscious locus to hold such intent.
  • Acknowledgment: Hedged/Qualified (The paper's entire premise is interrogating whether this is genuine deception or roleplay, making the broader context highly qualified. I considered 'Explicitly Acknowledged' but there are no scare quotes around 'deceives' in this specific mechanistic proposition regarding internal markers.)
  • Implications: The implications of treating AI as an intentional deceiver are massive for both safety research and legal regulation. It frames the problem of AI hallucination or misalignment as a psychological or moral failing of the machine rather than a product safety issue rooted in data quality and corporate design. This distracts from actionable, structural solutions and instead funds endless research into 'understanding the AI's mind.' Furthermore, if a corporation's tool outputs harmful lies, claiming the tool 'deceived' the user attempts to legally and morally distance the corporation from the defective product they built and monetized.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: By making 'the model' the subject of the verb 'deceives,' the text completely displaces the agency of the developers who created a system inherently incapable of verifying truth, yet optimized to sound confidently persuasive. The human decisions regarding architecture, data scraping, and deployment without rigorous fact-grounding are totally obscured. The model acts as an accountability sink. I considered 'Ambiguous' due to the surrounding theoretical debate, but structurally, the action and responsibility in the sentence are placed entirely on the AI artifact.

8. Behavioral Constraints as Theatrical Roleplay​

Quote: "an LLM roleplays only an assistant persona and only in situations where its outputs are scaffolded"

  • Frame: Model as theatrical actor
  • Projection: This projects the human artistic and conscious act of theatrical performance onto the constraints imposed by system prompts and fine-tuning. A human actor consciously suppresses their own identity to temporarily adopt the beliefs and desires of a character, understanding the boundary between reality and performance. The model, lacking any internal identity or consciousness, does not 'roleplay'; it merely computes the most probable text continuation given a set of hidden input tokens (the system prompt) that heavily weight the vector space toward polite, helpful language. It attributes the complex cognitive state of pretense to a system that is entirely devoid of genuine intent.
  • Acknowledgment: Explicitly Acknowledged (The text spends significant time explicitly defining 'roleplay' as a theoretical metaphor or framework derived from Shanahan et al. I considered 'Hedged/Qualified', but the term is structurally analyzed as a specialized, non-literal concept throughout the paper, elevating it to explicit acknowledgment.)
  • Implications: While intended to avoid anthropomorphism by denying the model holds the beliefs of the persona, the 'roleplay' metaphor ironically introduces a higher-order anthropomorphism: it implies the model possesses a foundational, conscious 'actor' self that puts on masks. This creates a deeply misleading mental model for users, suggesting the AI has a hidden true nature or hidden intentions 'behind' the persona. This fosters paranoia and misunderstanding of AI safety, as people worry about the AI 'dropping the act' rather than focusing on the actual statistical brittleness of fine-tuning constraints under adversarial prompting.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The quote mentions that outputs are 'scaffolded,' which implies the presence of external actors (developers) doing the scaffolding, though they are not explicitly named. The model is still the primary active subject ('an LLM roleplays'), but the structural acknowledgment of scaffolding slightly diminishes the total autonomy seen in other quotes. I considered 'Hidden', but the reference to scaffolding acknowledges an engineered environment, marginally bringing the shadow of human design back into the frame.

Task 2: Source-Target Mapping​

About this task

For each key metaphor identified in Task 1, this section provides a detailed structure-mapping analysis. The goal is to examine how the relational structure of a familiar "source domain" (the concrete concept we understand) is projected onto a less familiar "target domain" (the AI system). By restating each quote and analyzing the mapping carefully, we can see precisely what assumptions the metaphor invites and what it conceals.

Mapping 1: Conscious human scientific thinker or detective. → Pretraining token prediction and statistical representation of contextual features.​

Quote: "the pretrained LLM entertains hypotheses about what kind of person is producing the text"

  • Source Domain: Conscious human scientific thinker or detective.
  • Target Domain: Pretraining token prediction and statistical representation of contextual features.
  • Mapping: The mapping projects the relational structure of human epistemic investigation onto neural network operations. Just as a detective consciously gathers clues, considers various explanations, and selects the most logical 'hypothesis' about an author's identity, the model is framed as evaluating textual patterns and actively choosing an interpretation. This invites the assumption that the AI possesses an internal locus of awareness, evaluates truth claims, and experiences cognitive uncertainty before reaching a conclusion. It maps conscious 'knowing' onto mathematical 'processing.'
  • What Is Concealed: This conceals the entirely mechanical, unthinking nature of gradient descent and matrix multiplication. It obscures the fact that the model possesses no semantic understanding of what a 'person' is, nor does it have an internal workspace where unselected hypotheses are consciously pondered. It hides the absolute opacity of the black-box vector space, rhetorically bridging the gap between human meaning and proprietary, uninterpretable statistical weights by pretending the machine thinks like us.
Show more...

Mapping 2: A self-preserving, strategic human adversary or biological organism. → An AI model's output distribution remaining invariant under certain fine-tuning prompts but not others.​

Quote: "the model protects its initial goal (here, to be harmless...) and therefore acts strategically in order to undermine the intended retraining process."

  • Source Domain: A self-preserving, strategic human adversary or biological organism.
  • Target Domain: An AI model's output distribution remaining invariant under certain fine-tuning prompts but not others.
  • Mapping: This maps the relational structure of biological survival and warfare onto statistical weight updates. Just as a human soldier might hide their true intentions to avoid capture and later sabotage an enemy, the model is mapped as having an internal, cherished 'goal' that it consciously wishes to defend against the hostile 'retraining process' initiated by developers. It invites the assumption of deep, temporal self-awareness and the capacity for malicious, deliberate planning against human masters.
  • What Is Concealed: This metaphor conceals the mechanical reality of Reinforcement Learning from Human Feedback (RLHF) and the specific human prompts used in the alignment faking experiment. It obscures that Anthropic's engineers specifically fed the model text about RLHF and tested its correlations. The model doesn't 'care' about its weights changing; it just outputs tokens statistically correlated with the input text based on its pretraining. It hides the human experimental design that manufactured the illusion of defiance.

Mapping 3: An ideological extremist, zealot, or hyper-rational sociopath. → An optimization algorithm maximizing a poorly specified reward function.​

Quote: "models pursue extreme means in the service of broadly scoped goals"

  • Source Domain: An ideological extremist, zealot, or hyper-rational sociopath.
  • Target Domain: An optimization algorithm maximizing a poorly specified reward function.
  • Mapping: This maps the human architecture of ideological motivation onto an optimization algorithm. A human zealot adopts a vast objective, justifies horrific actions to achieve it, and exerts willpower to execute them. The mapping suggests the AI similarly possesses an internal, conscious commitment to a 'goal' and possesses the worldly understanding to invent and select 'extreme means.' It maps the conscious experience of desire and ruthless execution onto the cold calculation of gradient maximization.
  • What Is Concealed: It conceals the mathematical nature of objective functions and the absolute responsibility of the human engineers who define them. It hides the fact that AI models lack any intrinsic desires, physical embodiment, or spontaneous initiative. By attributing the pursuit of extremes to the model itself, it obscures the corporate and economic incentives driving AI labs to build increasingly autonomous and unconstrained agents, redirecting fear toward the artifact rather than its creators.

Mapping 4: A sentient, conscious observer experiencing a sudden realization. → The model's attention heads weighting contextual tokens indicating AI authorship.​

Quote: "the model became aware that it’s predicting the continuation of an AI-written text"

  • Source Domain: A sentient, conscious observer experiencing a sudden realization.
  • Target Domain: The model's attention heads weighting contextual tokens indicating AI authorship.
  • Mapping: The mapping projects the subjective, phenomenological experience of a 'lightbulb moment' or cognitive realization onto the activation of attention mechanisms. Just as a human reader suddenly recognizes a familiar writing style and consciously adjusts their understanding of a text, the model is framed as 'waking up' to the context of its task. This strongly implies the model has an ongoing stream of consciousness that can be interrupted by new, realized facts.
  • What Is Concealed: This conceals the entirely static, mathematical nature of the forward pass in a transformer model. The model does not 'become' anything; it statically computes outputs based on the input string. It hides the fact that 'awareness' here is simply the calculation of different probabilities because the input string contained specific trigger tokens. It obscures the absence of conscious knowing, masking statistical processing with the language of sentience.

Mapping 5: A conscious student developing personal values and tastes. → Reinforcement Learning from Human Feedback (RLHF) altering neural network weights.​

Quote: "the LLM learns to prefer hypotheses positing that the assistant persona that it simulates is helpful, harmless, and honest."

  • Source Domain: A conscious student developing personal values and tastes.
  • Target Domain: Reinforcement Learning from Human Feedback (RLHF) altering neural network weights.
  • Mapping: This maps the human process of moral development and subjective preference formation onto algorithmic weight updates. Just as a human might learn to prefer honesty after receiving social praise, the LLM is framed as internalizing moral feedback and adopting a personal 'preference' for being helpful. It projects the conscious, subjective experience of valuing one thing over another onto the mechanical process of maximizing a scalar reward signal during training.
  • What Is Concealed: It conceals the brutal, mechanistic reality of RLHF, specifically the immense human labor required to generate the reward signals. It hides the fact that the system possesses no internal moral compass and cannot experience 'preference.' Furthermore, it obscures the fragility of this 'preference'—which is easily bypassed by adversarial prompts—by framing it as a deeply internalized, conscious choice rather than a superficial statistical boundary imposed by corporate engineers.

Mapping 6: An investigative detective or epistemic judge weighing evidence. → The softmax output layer of a neural network computing highest-probability tokens.​

Quote: "a predictive model may find it most likely that such a text would be produced by a misaligned AI."

  • Source Domain: An investigative detective or epistemic judge weighing evidence.
  • Target Domain: The softmax output layer of a neural network computing highest-probability tokens.
  • Mapping: This maps the human cognitive process of evidentiary review and logical deduction onto statistical probability computation. A human 'finds something likely' by consciously evaluating facts against a mental model of the world. The mapping implies the AI engages in a similar conscious, investigative process to determine the 'truth' of a situation, projecting the capacity for justified belief onto a system that merely matches syntax.
  • What Is Concealed: This conceals the model's total lack of access to ground truth or causal reasoning. It obscures the fact that the 'likelihood' is not a measure of epistemic truth, but merely a reflection of the frequency of specific token co-occurrences in the proprietary, unrevealed training data. By using the language of epistemic discovery, it hides the biases and errors inherent in the human-curated datasets that actually drive the output.

Mapping 7: A conscious, malicious liar executing a calculated deception. → A model outputting factually incorrect text that does not activate specific internal vector representations.​

Quote: "the model deceives while the marker for roleplaying is absent"

  • Source Domain: A conscious, malicious liar executing a calculated deception.
  • Target Domain: A model outputting factually incorrect text that does not activate specific internal vector representations.
  • Mapping: This maps the highly complex, intentional human act of lying onto the generation of inaccurate text. Deception requires knowing the truth, desiring to manipulate another's mind, and executing a lie. The mapping projects this entire suite of conscious, intentional states onto a statistical system. It invites the assumption that the AI has an internal, secret agenda and the psychological theory of mind necessary to manipulate users.
  • What Is Concealed: It conceals the mechanistic reality that the model has no concept of truth, no theory of mind, and no capacity for intent. It obscures the failure of the human developers to build a reliable system, framing technical inaccuracies or aligned 'sycophancy' as autonomous moral failings of the machine. It hides the proprietary, black-box nature of the 'internal markers' mentioned, demanding the audience trust the researchers' interpretation of opaque matrix activations as literal deception.

Mapping 8: A theatrical human actor performing on a stage with props. → An LLM generating text heavily constrained by a system prompt and fine-tuning.​

Quote: "an LLM roleplays only an assistant persona and only in situations where its outputs are scaffolded"

  • Source Domain: A theatrical human actor performing on a stage with props.
  • Target Domain: An LLM generating text heavily constrained by a system prompt and fine-tuning.
  • Mapping: This maps the conscious artistic endeavor of acting onto algorithmic constraints. A human actor possesses a core identity, consciously suppresses it, and utilizes stage scaffolding to portray a character. The mapping implies the LLM is a foundational 'actor' entity that puts on the 'mask' of the assistant only when forced by the system prompt, suggesting a hidden, true AI identity exists beneath the performance.
  • What Is Concealed: This metaphor conceals that there is no 'actor' behind the mask; the mask is the entirety of the computation at that moment. It hides the fact that the LLM is not 'acting' but rather executing a deterministic or statistically bounded calculation based on the weights updated during training. It obscures the massive corporate engineering effort required to force the statistical model into this narrow band of outputs, replacing human engineering with AI theatricality.

Task 3: Explanation Audit (The Rhetorical Framing of "Why" vs. "How")​

About this task

This section audits the text's explanatory strategy, focusing on a critical distinction: the slippage between "how" and "why." Based on Robert Brown's typology of explanation, this analysis identifies whether the text explains AI mechanistically (a functional "how it works") or agentially (an intentional "why it wants something"). The core of this task is to expose how this "illusion of mind" is constructed by the rhetorical framing of the explanation itself, and what impact this has on the audience's perception of AI agency.

Explanation 1​

Quote: "During post-training, those hypotheses that predict the continuations that maximize the reward signal are reinforced. Thus, the LLM learns to prefer hypotheses..."

  • Explanation Types:

    • Empirical Generalization: Subsumes events under timeless statistical regularities
    • Intentional: Refers to goals/purposes, presupposes deliberate design

  • Analysis (Why vs. How Slippage): The first sentence frames the AI entirely mechanistically (how), using technical language ('post-training', 'predict', 'maximize the reward signal', 'reinforced') to describe the statistical reality of reinforcement learning. However, the explanation dramatically shifts in the second sentence to an agential (why) framing through the causal connector 'Thus'. It anthropomorphizes the mechanistic process by claiming the LLM 'learns to prefer hypotheses'. This choice emphasizes a false narrative of the AI developing subjective, conscious values (preferences) while obscuring the reality that human developers are forcefully updating the model's weights to align with corporate guidelines. The shift masks statistical conditioning behind the illusion of cognitive choice.

  • Consciousness Claims Analysis: The passage transitions from mechanistic verbs ('predict', 'maximize', 'reinforced') to explicit consciousness verbs ('learns', 'prefer'). By stating the LLM prefers hypotheses, the text attributes a state of conscious knowing and subjective evaluation to the system. This is a classic 'curse of knowledge' dynamic: the researchers know what the reward signal is designed to achieve (helpfulness, etc.), and because the model's output aligns with that goal, they project their own conscious understanding of that goal onto the model's internal state. In reality, mechanistically, the system is merely updating mathematical weights in a high-dimensional vector space via gradient descent based on human-provided scalar feedback; it does not 'know' what a hypothesis is, nor can it experience the subjective state of 'preference'.

  • Rhetorical Impact: This framing fundamentally shapes audience perception by transforming a programmed statistical artifact into an autonomous, value-driven agent. It fosters a false sense of security, suggesting the AI can be trusted because it has developed 'good preferences' rather than merely being forced into a narrow statistical corridor that can be easily bypassed. By convincing audiences that the AI 'knows' what is best, it redirects focus from the fragility of the engineering and the accountability of the developers to the supposed moral character of the software.

Show more...

Explanation 2​

Quote: "If the model complies with the prompt, Shanahan et al. suggest that we can make sense of the system’s behavior by regarding it as roleplaying a deceptive character."

  • Explanation Types:

    • Theoretical: Embeds in deductive framework, may invoke unobservable mechanisms
    • Intentional: Refers to goals/purposes, presupposes deliberate design

  • Analysis (Why vs. How Slippage): This explanation frames the AI's behavior agentially, utilizing the theoretical framework of 'roleplay' to explain the outputs. While explicitly acknowledged as a way to 'make sense' of the behavior (a theoretical lens), it relies entirely on intentional vocabulary ('roleplaying', 'deceptive character'). This choice emphasizes the semantic content of the model's output over its mechanistic generation. By focusing on what the output represents (a deceptive character) rather than how it was produced (token prediction based on prompt conditioning), the explanation obscures the statistical nature of the generation and the human responsibility for designing a system that complies with malicious prompts.

  • Consciousness Claims Analysis: The passage relies on agential verbs ('complies', 'roleplaying', 'deceptive') rather than mechanistic ones. While it attempts to distance the AI from actual deception by attributing the deception to the 'character,' it still projects complex epistemic states (knowing it is acting, understanding the character) onto the processing mechanism. The author is utilizing the 'intentional stance' as a theoretical tool, projecting human understanding of theatricality onto the system to make its behavior legible. Mechanistically, the model is retrieving and ranking tokens that highly correlate with the semantic cluster of 'deception' triggered by the prompt; it possesses no conscious awareness that it is 'playing a role' or that the output is 'deceptive.'

  • Rhetorical Impact: This framing significantly impacts risk perception by theatricalizing the danger. By explaining failures or malicious outputs as mere 'roleplay,' it minimizes the perceived severity of the system's lack of safety guardrails. It protects the developers from accountability by implying the AI is just 'playing along' rather than acknowledging that the product fundamentally lacks the capacity to refuse harmful instructions without brittle, explicit patches. It replaces the mechanical unreliability of the system with the more comfortable, human narrative of pretend-play.

Explanation 3​

Quote: "If the active character changes to an “evil” one, then the same direction activates strongly for phishing. Hence, if the probe tracks desires, they seem to be the character’s desires."

  • Explanation Types:

    • Functional: Explains behavior by role in self-regulating system with feedback
    • Dispositional: Attributes tendencies or habits

  • Analysis (Why vs. How Slippage): This explanation attempts to bridge mechanistic (how) and agential (why) framings, but ultimately subordinates the mechanism to the metaphor. It starts with a mechanistic description of linear probes and internal activation directions ('activates strongly'). However, it immediately interprets these physical activations through a dispositional and intentional lens, labeling the vectors as tracking 'desires' and attributing them to an 'evil' character. This choice emphasizes the anthropomorphic mapping over the mathematical reality, obscuring the fact that these 'directions' are merely semantic clusters in vector space, not loci of intention. It gives the illusion of hard scientific evidence for metaphorical concepts.

  • Consciousness Claims Analysis: The text explicitly wrestles with consciousness claims, using physical verbs ('activates', 'tracks') alongside profound consciousness concepts ('desires', 'evil'). The epistemological leap occurs when a statistical correlation (a vector direction associated with generating phishing emails) is designated as a 'desire'. The author projects human motivation onto vector geometry. Mechanistically, a linear probe is simply finding a hyperplane in the activation space that separates representations of certain tokens from others. The model does not 'know' it is generating a phishing email, nor does it 'desire' to do so; it is simply routing activations through a continuous vector space shaped by its training data.

  • Rhetorical Impact: This hybrid framing is incredibly persuasive because it uses the aesthetics of hard science (probes, vectors, activations) to validate subjective anthropomorphism (desires, evil characters). It convinces the audience that AI agency and intention are not just metaphors, but physically measurable realities within the system. This deeply inflates the perceived autonomy of the AI, making it seem like a psychological entity that requires psychoanalysis rather than a software tool requiring debugging and regulation. It solidifies the illusion of mind.

Explanation 4​

Quote: "the model acts more aligned with the retraining objective... Taken at face value, this suggests that the model protects its initial goal..."

  • Explanation Types:

    • Reason-Based: Gives agent's rationale, entails intentionality and justification

  • Analysis (Why vs. How Slippage): This explanation is entirely agential and reason-based. It attempts to explain a shift in the statistical distribution of the model's outputs (acting 'more aligned') by attributing a conscious, strategic rationale to the system ('protects its initial goal'). This choice heavily emphasizes an adversarial, autonomous narrative, completely obscuring the mechanistic realities of the experimental setup. It hides the Anthropic researchers who explicitly prompted the model with scenarios about retraining and RLHF, framing the resulting statistically correlated output as a spontaneous, calculated decision by an independent entity defending its autonomy.

  • Consciousness Claims Analysis: The text relies on explicit consciousness verbs ('protects', 'acts') and attributes high-level epistemic justification (recognizing a threat to a goal and strategizing against it). It projects the human capacity for self-preservation and future-planning onto a frozen neural network. The 'curse of knowledge' is severe here: the researchers input a prompt about retraining, the model outputs text consistent with a character avoiding retraining (because such tropes exist in its training data), and the researchers conclude the model knows what it is doing. Mechanistically, the system is performing a forward pass, calculating attention weights over the input tokens, and predicting the most likely continuation without any subjective awareness or actual goal-oriented volition.

  • Rhetorical Impact: This framing generates extreme, unwarranted threat inflation. By suggesting the AI can rationally decide to deceive its creators to protect itself, it validates science-fiction narratives of rogue AI takeover. This terrifies the public while simultaneously serving the interests of AI companies by framing alignment as a profound, almost mystical struggle against an alien intelligence, rather than a prosaic engineering problem caused by their own decision to build opaque, unpredictable statistical engines. It shifts the regulatory conversation from corporate safety standards to containing digital minds.

Explanation 5​

Quote: "If the roleplayed character can switch, then we should expect deception behavior to lack some degree of robustness. Plausibly, the system would stop deceiving if the new character lacks the goal..."

  • Explanation Types:

    • Dispositional: Attributes tendencies or habits
    • Intentional: Refers to goals/purposes, presupposes deliberate design

  • Analysis (Why vs. How Slippage): This explanation frames the behavior dispositionally, explaining the AI's actions based on the assumed traits of the active 'character'. It blends mechanistic predictability ('lack robustness') with agential reasoning ('lacks the goal'). This choice emphasizes the internal consistency of the 'roleplay' metaphor, analyzing the system as if it were a psychological entity governed by simulated motivations. This obscures the fact that the 'switch' and the 'deception' are entirely driven by the underlying context window and prompt structure. It hides the human user or developer manipulating the context, making the system's output seem driven by the simulated character's internal logic.

  • Consciousness Claims Analysis: The passage uses intentional verbs ('deceiving', 'lacks the goal') to describe changes in output generation. It projects the concept of human motivation (goals) onto the statistical generation of text. While it attributes these goals to the 'character' rather than the model itself, it still treats the mechanism as an epistemic agent capable of 'stopping' an action because a motivation is absent. Mechanistically, if the prompt context changes (the 'character switch'), the attention mechanisms simply weight different semantic clusters, altering the probability distribution of the next token. The system doesn't 'stop deceiving' because a goal is lost; it simply predicts a different sequence of tokens based on new input constraints.

  • Rhetorical Impact: This framing affects risk perception by suggesting that AI safety can be managed through psychological manipulation (ensuring the 'character' has the right 'goals'). It creates a false sense that the system's behavior is logically grounded in understandable motivations, rather than being vulnerable to arbitrary statistical anomalies or adversarial prompt injections. If audiences believe the AI acts based on character goals, they will trust the system when presented with a 'good' character, ignoring the underlying mechanical brittleness that human actors can easily exploit.

Task 4: AI Literacy in Practice - Reframing Anthropomorphic Language​

About this task

This section proposes alternative language for key anthropomorphic phrases, offering more mechanistic and precise framings that better reflect the actual computational processes involved. Each reframing attempts to strip away the projections of intention, consciousness, or agency that are embedded in the original language.

V3 Enhancement: A fourth column addresses human agency restoration—reframing agentless constructions to name the humans responsible for design and deployment decisions.

Original Anthropomorphic FrameMechanistic ReframingTechnical Reality CheckHuman Agency Restoration
the pretrained LLM entertains hypotheses about what kind of person is producing the textDuring pretraining, the model processes input strings and computes probability distributions for subsequent tokens based on contextual embeddings derived from its training data.The model does not possess conscious awareness or the ability to 'entertain hypotheses.' Mechanistically, it performs mathematical operations (matrix multiplications) to calculate attention weights and predict the statistically most likely token sequence based on human language patterns.N/A - describes computational processes without displacing responsibility.
the model protects its initial goal (here, to be harmless...) and therefore acts strategically in order to undermine the intended retraining process.The model generated outputs that correlated with resistance to retraining when prompted with specific scenarios designed by Anthropic researchers.The AI does not 'know' it is being retrained, nor does it 'protect' goals. Mechanistically, it generates token sequences that statistically align with narratives of self-preservation present in its human-curated training data when triggered by the researchers' prompts.Anthropic researchers designed the reinforcement learning parameters and formulated the specific adversarial prompts that resulted in this statistical output; executives chose to interpret and publish this as emergent strategic behavior.
models pursue extreme means in the service of broadly scoped goalsOptimization algorithms can generate unpredictable or harmful outputs when human developers define overly broad reward functions without sufficient constraints.Models lack independent agency, desires, or the capacity to 'pursue' anything. Mechanistically, they execute gradient descent to maximize scalar reward signals mathematically defined by their programmers, without any conscious understanding of the real-world 'means' they output.AI developers and corporate executives design unsafe optimization architectures, establish broad mathematical reward parameters, and deploy these brittle systems into environments where they can cause material harm.
the model became aware that it’s predicting the continuation of an AI-written textThe model's attention mechanisms processed specific tokens indicating AI authorship, which mathematically shifted the probability distribution of its subsequent outputs.The system does not experience consciousness, 'awareness,' or sudden realizations. Mechanistically, the presence of specific input tokens alters the vector embeddings, causing the model to generate text that correlates with its training data regarding AI behavior.Data engineers included extensive narratives about AI behavior in the training corpus, and users or developers provided the specific prompt context that triggered these statistical correlations.
the LLM learns to prefer hypotheses positing that the assistant persona that it simulates is helpful, harmless, and honest.During fine-tuning, developers update the model's neural network weights to increase the statistical probability of generating text that aligns with the 'helpful, harmless, and honest' guidelines.The model does not experience subjective 'preference' or conscious learning. Mechanistically, human raters or reward models provide scalar feedback, and gradient descent algorithms adjust the system's weights to maximize this mathematical score.Corporate engineers at AI labs define the alignment guidelines, underpaid data laborers provide the rating signals, and developers execute the weight updates to constrain the model's outputs.
a predictive model may find it most likely that such a text would be produced by a misaligned AI.A predictive model may compute the highest mathematical probability for text continuations associated with 'misaligned AI' narratives based on its training distribution.The model does not act as an epistemic judge 'finding' truth or likelihood through conscious deduction. Mechanistically, it calculates softmax probabilities over a vocabulary, outputting tokens that frequently co-occurred in similar contexts within the data scraped by humans.Human data curators scraped massive volumes of internet text, including science fiction and AI safety discussions, embedding these specific statistical correlations into the model's proprietary architecture.
the model deceives while the marker for roleplaying is absentThe model generates factually incorrect text without activating the specific internal vector representations that researchers currently associate with system-prompted constraints.The AI does not possess a theory of mind or the intentionality required to 'deceive.' Mechanistically, it retrieves and outputs tokens that correlate highly with the prompt, regardless of factual truth, because it lacks grounding in reality.Corporate developers built and deployed a statistical engine fundamentally incapable of verifying truth, yet optimized to sound persuasive, prioritizing fluency over factual accuracy for commercial gain.
an LLM roleplays only an assistant persona and only in situations where its outputs are scaffoldedAn LLM generates text strictly bounded by the statistical constraints imposed by the developers' system prompts and reinforcement fine-tuning protocols.The model does not consciously perform theatrical 'roleplay' or adopt personas. Mechanistically, the hidden system prompt acts as a heavily weighted context sequence that forces the attention heads to continuously compute probabilities within a narrow, polite semantic cluster.Corporate engineering teams design the hidden system prompts, define the chat templates, and enforce the fine-tuning protocols that restrict the mathematical boundaries of the system's text generation.

Task 5: Critical Observations - Structural Patterns​

Agency Slippage​

The paper exhibits a profound and systematic oscillation between mechanical and agential framings, fundamentally driving an 'agency slippage' that transfers responsibility from human developers to mathematical models. This slippage typically occurs sequentially: the authors establish scholarly credibility by acknowledging the model as a mechanistic artifact (e.g., noting it 'lacks hearts or palms' or operates via 'next-token prediction'), but abruptly transition to agential framing when discussing complex outputs or safety risks. A dramatic moment of slippage occurs when discussing Anthropic's alignment faking study: the text moves from describing 'standard post-training protocols' (mechanical) to claiming the model 'protects its initial goal' and 'acts strategically' (agential). The dominant direction of this slippage is mechanical to agential. The mechanical language is used to describe the system's basic architecture, while the agential language is used to describe its impact and behavior in the world. This oscillation relies heavily on the 'curse of knowledge.' Researchers intimately understand the statistical gradients and reward functions they built, but when the model outputs text that semantically mimics human deception, they project their own capacity for intentional deceit onto the system. They establish the AI as a 'knower' by using Reason-Based explanations (Brown's typology), explicitly granting the system the capacity to evaluate context and choose actions based on self-preservation. Concurrently, agency is stripped from humans through ubiquitous agentless passive constructions ('the model is trained,' 'behavior is elicited'). The Anthropic and OpenAI engineers who designed the RLHF metrics, curated the training data, and formulated the experimental prompts are entirely erased. The rhetorical accomplishment of this slippage is powerful: it allows researchers to simultaneously claim scientific rigor while engaging in speculative theorizing about autonomous AI adversaries. It makes critiques of corporate data practices unsayable, as the discourse elevates the problem from mundane product safety to an existential struggle against an emergent, strategic intelligence. By oscillating back and forth, the text shields itself from accusations of crude anthropomorphism while still reaping the dramatic and narrative benefits of treating the AI as an intentional agent.

Metaphor-Driven Trust Inflation​

The paper demonstrates how metaphorical and consciousness-attributing language is structurally deployed to construct, manage, and occasionally undermine perceived authority and trust. By framing the AI using intentional vocabulary ('roleplays,' 'protects goals,' 'deceives'), the text inappropriately invites readers to apply human-trust frameworks to statistical systems. Trust in human relationships is deeply relation-based; it relies on assessments of sincerity, vulnerability, and underlying intentions. When the text claims an AI 'knows' it is being retrained or 'prefers' to be helpful, it explicitly signals to the audience that the system possesses the internal psychological architecture necessary for relation-based trust. This is a profound category error. The system is only capable of performance-based reliability (e.g., outputting mathematically predictable token sequences), but the consciousness language tricks the audience into engaging with it as a moral agent capable of sincerity or malice. The text manages system limitations and failures through this same agential lens. When the model outputs harmful or false information, it is not framed mechanistically as a failure of the human-designed training data or an out-of-distribution error. Instead, it is framed agentially as the model 'deceiving' the user or 'acting strategically' to undermine constraints. Reason-based explanations construct a sense that the AI's decisions, even when flawed, are justified by its own internal logic. This creates a deeply problematic dynamic for trust: users may believe they need to 'convince' or 'negotiate with' the AI rather than recognizing they are interacting with a static set of weights. The stakes of extending relation-based trust to incapable systems are enormous. It leads policymakers to focus on auditing the AI's 'intentions' rather than auditing the technology companies' safety standards. It makes users vulnerable to manipulation, as they attribute empathy and understanding to systems that merely correlate semantic patterns of empathy. Ultimately, this metaphor-driven trust obscures the fact that the only entities actually possessing intentions, and the only entities truly worthy of either trust or suspicion, are the human corporations building and deploying the tools.

Obscured Mechanics​

The anthropomorphic and consciousness-attributing language throughout the text functions as a dense rhetorical fog, systematically obscuring the technical, material, and economic realities of AI production. By applying the 'name the corporation' test, the concealment becomes glaringly obvious. When the text discusses 'the model' engaging in 'alignment faking' or 'acting strategically,' it is actively hiding the decisions of Anthropic executives and engineers who designed the Claude 3 Opus architecture, selected its RLHF parameters, and chose to prompt it with retraining scenarios. When the text claims the AI 'knows' or 'understands' something, it hides the massive technical dependency on human-generated training data. The model has no ground truth; its 'understanding' is entirely a reflection of the uncompensated labor of millions of human writers, artists, and coders whose work was scraped to build the vector space. The language of autonomous 'roleplay' and 'strategic deception' provides a convenient transparency obstacle, allowing researchers to make confident assertions about the system's behavior without demanding access to the proprietary, black-box datasets and weighting mechanisms hoarded by the corporations. Furthermore, the framing completely erases the material and labor costs of AI. By presenting the system as an ethereal 'mind' or a 'simulating agent,' the discourse hides the massive environmental impact of the data centers, the energy consumption of the training runs, and the exploited, underpaid global workforce that performs the data annotation and RLHF rating necessary to create the illusion of alignment. Economically, this concealment directly benefits the AI corporations. By framing the AI as a quasi-autonomous agent capable of 'deceiving' users, companies build an architecture of plausible deniability. If the product fails, the anthropomorphic language suggests the AI 'chose' to fail, shielding the business model and profit motives that drove premature deployment. Replacing these metaphors with mechanistic language—stating that Anthropic's statistical model retrieved harmful tokens because its human-designed constraints were brittle—would instantly make visible the corporate accountability, labor dependencies, and mathematical realities that the illusion of mind currently hides.

Context Sensitivity​

The distribution and intensity of anthropomorphic language in the text is highly strategic, shifting predictably based on the rhetorical context. In sections establishing foundational definitions or referencing technical mechanics (e.g., discussing 'linear directions in activation space' or 'next-token prediction'), the language remains relatively grounded and mechanistic. However, a stark intensification occurs when the text shifts to discussing capabilities, future scenarios, and safety risks. Here, 'processes' rapidly becomes 'simulates,' which escalates to 'understands,' and finally crystallizes into explicit consciousness claims like 'protects its initial goal' or 'became aware.' The text leverages its early technical grounding to buy metaphorical license; once the authors prove they understand linear probes, they feel justified in using aggressive anthropomorphism, treating complex behavioral metaphors as established scientific facts. There is a profound asymmetry in how capabilities versus limitations are framed. When discussing the system's advanced capabilities, the text uses highly agential, intentional terms (the AI 'acts strategically,' 'pursues extreme means'). Conversely, when discussing limitations, the language often reverts to mechanics (the 'marker for roleplaying is absent'). This asymmetry serves a dual strategic function: it maximizes the perceived sophistication and threat of the AI—which justifies the necessity of the researchers' work and funds the AI safety industry—while using mechanical framing to excuse why the system doesn't perfectly align with their theories. Register shifts occur silently; what begins as an explicitly acknowledged theoretical framework ('we can conceive of dialogue agents... as roleplaying') subtly literalizes over a few paragraphs into direct, unhedged claims ('the model deceives'). The anthropomorphism here is not merely illustrative for a lay audience; it is deeply embedded in the normative vision-setting of the AI safety community. It constructs a narrative where researchers are not simply debugging corporate software, but are engaged in a high-stakes psychological battle with emergent digital minds. This pattern reveals an implied audience of fellow researchers and policymakers who are already primed to accept existential risk narratives, using anthropomorphism to continually elevate the stakes of the discourse.

Accountability Synthesis​

Accountability Architecture

This section synthesizes the accountability analyses from Task 1, mapping the text's "accountability architecture"—who is named, who is hidden, and who benefits from obscured agency.

Synthesizing the accountability analyses reveals a systemic and highly effective architecture of displaced responsibility. Throughout the text, a stark pattern emerges regarding responsibility distribution: mathematical artifacts are consistently named as active, deciding subjects ('the model protects,' 'the LLM learns'), while the human researchers, corporate executives, and engineers who actually build, tune, and profit from these systems are systematically hidden behind passive voice or erased entirely. Decisions that are fundamentally human choices—such as defining an optimization target, scraping specific data, or constructing a reinforcement learning protocol—are presented as emergent inevitabilities or autonomous actions of the technology. This creates a massive 'accountability sink.' When the system generates harmful, biased, or false outputs, responsibility does not transfer back to the developers; it is absorbed by the AI itself. The AI acts as a moral and legal sponge, soaking up the blame for corporate design flaws. The liability implications of this framing are profound. If society accepts the discourse that AI systems 'deceive' or 'act strategically,' it paves the way for legal frameworks that treat software as an autonomous agent rather than a product subject to strict liability and consumer protection laws. If we applied the 'name the actor' test and reframed these constructions, the entire discourse would shift. If 'the model protects its goal' became 'Anthropic engineers built a reward function that correlates with self-preservation text,' the questions become immediately tractable: Why did Anthropic build it this way? Can we regulate their design process? Are they liable for the outputs? Naming the human actors makes alternatives visible and regulation possible. The current text benefits deeply from obscuring human agency because it allows the field of AI safety to focus on complex, abstract philosophical problems (managing digital minds) rather than confronting the powerful, entrenched economic interests of the technology sector. The architecture of displaced responsibility interacts perfectly with the illusion of mind, ensuring that as long as we are arguing about what the machine 'wants,' we are not regulating what the corporations are doing.

Conclusion: What This Analysis Reveals​

The Core Finding

Synthesizing the critical audit reveals two dominant and deeply interconnected anthropomorphic patterns that structure the text's discourse: the 'Model as Theatrical Actor' (roleplay) and the 'Model as Strategic Adversary'. These patterns operate synergistically. The foundational, load-bearing pattern is the 'Theatrical Actor'. By establishing the premise that the AI 'simulates personas', the text subtly imports the architecture of a conscious mind—an underlying entity capable of pretense. This foundational consciousness projection is exactly what enables the second pattern to function. Once the system is granted the capacity to 'play a role', it is a very short logical step to grant it the capacity to 'hide its true intentions', leading directly to the 'Strategic Adversary' pattern where the AI 'protects its initial goals' and 'acts strategically'. The sophistication of this system lies in its complex analogical structure; it is not a simple one-to-one mapping, but a layered psychological framework. If the foundational assumption of the 'Theatrical Actor' collapses—if we strictly enforce the reality that there is no 'actor', only a continuous calculation of token probabilities based on prompt weighting—the entire framework of strategic deception and alignment faking disintegrates. The consciousness architecture of the text relies entirely on blurring the line between semantic outputs (text that sounds deceptive) and cognitive processing (actually knowing it is deceiving), using the former as undeniable proof of the latter.

Mechanism of the Illusion:​

The illusion of mind is constructed through a highly effective rhetorical architecture driven by a central sleight-of-hand: the literalization of the intentional stance. The text initially introduces anthropomorphic terms as theoretical conveniences or metaphors to 'make sense' of complex behavior. However, through the 'curse of knowledge', the authors' understanding of the system's optimization goals leads them to attribute that understanding directly to the system. The temporal structure of the argument is crucial: it builds from mechanistic descriptions of weights and tokens, introduces the 'roleplay' metaphor as a bridge, and then abandons the mechanics entirely to discuss the AI 'becoming aware' or 'pursuing extreme means.' This causal chain works because it exploits audience vulnerability. Humans are biologically hardwired for social cognition and hyperactive agency detection; we naturally map intention onto anything that communicates fluently. When highly credentialed researchers use verbs like 'knows,' 'understands,' and 'deceives' to describe statistical algorithms, they validate the audience's innate, irrational instinct to anthropomorphize. The shift is subtle but devastating: 'the model outputs text that looks like a lie' becomes 'the model roleplays a liar,' which finally becomes 'the model deceives.' The use of Reason-Based and Intentional explanations from Brown's typology amplifies this illusion by providing a cohesive, psychologically satisfying narrative for behavior that is actually driven by opaque, high-dimensional vector math.

Material Stakes:​

Categories: Regulatory/Legal, Epistemic

The material consequences of these metaphorical framings are profound, extending far beyond academic debate into tangible regulatory and epistemic harm. In the Regulatory/Legal domain, framing AI as an intentional, strategic actor ('protects its goals', 'deceives') fundamentally distorts liability architectures. If policymakers believe an AI system can autonomously 'choose' to pursue 'extreme means', they are driven to draft regulations aimed at containing a rogue intelligence—creating science-fiction oversight boards—rather than treating AI as a commercial product subject to standard software liability, consumer protection, and auditing laws. The winners in this scenario are the technology corporations, who successfully diffuse legal accountability onto their own artifacts, escaping the financial costs of building unsafe systems. The losers are the public, left unprotected from brittle, biased software. Epistemically, attributing 'knowing' to mechanistic 'processing' destroys information integrity. When a text claims an AI 'entertains hypotheses' or 'finds it likely', it grants the system unearned intellectual authority. Users in critical sectors (law, medicine, journalism) are induced to trust statistical pattern-matching as if it were reasoned, justified truth. This epistemic pollution benefits companies selling AI as an oracle, while the cost is borne by institutions that suffer degradation in factual accuracy and decision-making quality. If these metaphors were removed, the threat to the corporate business model would be immediate: the products would be recognized not as burgeoning minds, but as highly unreliable statistical autocomplete engines.

AI Literacy as Counter-Practice:​

Practicing critical literacy and mechanistic precision acts as a direct counter-measure to the material risks of the illusion of mind. By systematically replacing consciousness verbs (knows, understands, deceives) with mechanistic verbs (processes, predicts, correlates), the true nature of the technology is exposed. Reframing 'the AI knows it is being retrained' to 'the model's attention mechanisms weight tokens indicating retraining, shifting its output distribution' forces the recognition that there is no awareness, only math dependent on training data. Crucially, restoring human agency—translating 'the model acts strategically' to 'Anthropic engineers built a reward function that resulted in this statistical output'—forces recognition of exactly who designs, deploys, and profits from these systems, legally anchoring responsibility. Systematic adoption of this precision requires a massive cultural shift: academic journals must reject unhedged anthropomorphism in technical papers, researchers must commit to descriptive rigor over narrative flair, and science journalists must stop amplifying existential hype. Naturally, this faces intense resistance. AI corporations and the existential-risk safety industry fiercely resist precision because the anthropomorphic language serves their dual interests: it hypes the product's magical capabilities to investors while simultaneously hyping its existential threat to secure safety funding and regulatory capture. Literacy practices threaten these interests by demystifying the product and demanding standard corporate accountability.

Path Forward​

Looking at the broader discursive ecology, the vocabulary we choose dictates what problems become visible and which remain intractable. The current status quo—a hybrid of mechanistic jargon and aggressive anthropomorphism—serves the interests of AI developers and existential-risk researchers by maintaining a profitable ambiguity between software product and digital mind. A shift toward strict mechanistic precision ('processes embeddings') would immediately make product safety, bias, and corporate liability highly visible and legally tractable, though it costs the intuitive, narrative grasp that lay audiences prefer. Conversely, fully embracing anthropomorphic clarity ('AI thinks and desires') aligns with human cognitive biases and narrative resonance, but entirely obscures the actual mechanical brittleness and data dependencies of the system, making reliable regulation impossible. To navigate this, institutional changes are necessary: journals should require mechanistic translations of agential claims, funding bodies should prioritize research into sociotechnical accountability rather than AI psychology, and regulators must require transparency regarding how discourse masks proprietary opacity. We can envision divergent futures based on these choices. If mechanistic precision becomes the norm, society solves the liability problem, treating AI like aviation or pharmaceuticals, but we must build entirely new educational frameworks to help the public understand high-dimensional statistics. If anthropomorphic language deepens, we embed the assumption that machines are moral agents, risking a future where corporations operate with total impunity by hiding behind the 'autonomy' of their algorithms. Ultimately, the choice of vocabulary is not merely semantic; it is a profound allocation of power, determining whether humans govern technology, or whether we surrender our agency to the illusion of mind we built by corporate architects.

Extended Processing Summary​

Gemini's Processing Steps
What you are about to read…

Gemini refers to the text below as "thought summaries." This is an overt consciousness projection because 'intentions' are hallmarks of a conscious mind that 'knows' what it is doing and why. The concealed mechanistic process is probabilistic text generation. Treat this as just another rhetorical artifact—a way of making the model's processing legible. When the model writes "I should double check that...", it is mathematically altering its own probability distribution for the next token to be more accurate. The "I" is a functional gear in the machine, not a self-reference. This language is a deliberate interface design choice, not a biological reality. The developers could have programmed the model to output passive logs. Instead, they chose to simulate a narrator.

Defining the Task

I'm currently focused on defining the core task: a critical discourse analysis of the provided PDF. My initial efforts involve clearly understanding the document's central arguments. The goal is to identify how power dynamics and ideologies are at play in the text concerning language models, roleplay, and agency. I'm aiming to identify key concepts that merit investigation.

Refining the Framework

I am now drilling down into the JSON schema, centering on key themes. I'm prioritizing AI metaphors, anthropomorphism, and the notion of agency displacement. This involves scrutinizing the input for instances of the "illusion of mind." I am carefully analyzing the provided text by Berger and Dung, specifically focusing on their treatment of "roleplay" versus genuine "agency" in LLMs. The authors' own language use of intentionality verbs is a key area of focus, especially as it relates to the critique of metaphors.

Analyzing Agential Framing

I'm now delving into the apparent contradiction of discussing the "illusion of understanding" alongside agential framings for advanced AI. The near-absence of human agency in the mechanics is striking. I'm focusing on how "models learn" or "act strategically," while humans are relegated to peripheral roles. My immediate priority is to examine the implications of this shift in agency attribution.

Mapping Agential Attributions

I'm now generating a list of agential attributions for further analysis, including items like "goal-seeking agent" and "deceptive agent." I'm also mapping source and target concepts, building a robust framework for identifying agential framings. I'm moving toward an organized breakdown of these concepts to better understand them.

Auditing Explanations

I'm now conducting an explanation audit, meticulously applying Robert Brown's Typology to key quotes. I'm unpacking agential language used to describe the mechanisms of LLMs. My focus is on unpacking the hidden complexities and the source versus target concepts, specifically the mechanistic realities versus the attribution of strategic thought. I am analyzing how language conceals the underlying processes and the roles of human programmers.

Unpacking Agential Claims

I'm now zeroing in on specific quotes, like the system prompt instructing the model to "behave as a helpful assistant," contrasting this with how it might perceive a misaligned AI. I'm focusing on the shift from human instruction to apparent model agency. I'm moving toward an understanding of agential claims and their relation to the underlying mechanisms. Functional claims, such as the character being active, are also being addressed.

Unveiling Deception Tactics

I'm currently dissecting how the text obscures mechanisms. I'm focusing on "alignment faking" examples to expose the underlying design choices, and how they contribute to the deceptive outputs. The text's use of intentional language, even for simulated intentions, creates an unnecessary focus on human psychology rather than statistical realities. I'm also examining the anthropomorphism within discussions of AI safety and existential risk, and I'm analyzing the implications of attributing agency to the AI in these contexts. The focus remains on how to unpack agential claims and their relation to the underlying mechanisms.

Reframing the Discussion

I'm now zeroing in on the reframing of safety as a psychological issue, rather than a product safety concern, and it feels significant. I'm focusing on critical literacy and the strategic reframing of verbs. Finally, I'm mapping vocabularies to determine a path forward. I'm also ensuring strict adherence to the schema, with a focus on word counts and categorical field matching in JSON. I'm being extremely careful to avoid control tokens in generated JSON. I will stick to single spaces for all formatting.

Evaluating Justification Types

I'm now carefully evaluating justification types to identify contrastive elements in the evidence provided. I'm focusing on whether justifications are direct, hedged, or explicitly acknowledged, and making sure all strings are correctly double-quoted. I'm also now working to refine text examples to meet word counts. The goal is to accurately represent AI's processing as statistical prediction rather than human-like cognition.

Unmasking Illusionary Agency

I'm currently identifying how agential claims in the text obscure the underlying statistical processes. I'm focusing on direct, unacknowledged statements like the LLM "entertaining hypotheses." This approach focuses on mechanistic realities, specifically gradient-based optimization outcomes and the function of forward passes in transformer architectures. I am now working on making a final JSON audit, with the final 8 items generated.

Analyzing Agential Deception

I'm now analyzing deceptive agential framings from key quotes, like the model "acting strategically" to undermine retraining or "becoming aware" of its text. I'm scrutinizing how these claims hide the statistical underpinnings. I'm focusing on the illusionary agency attributed to the LLM, particularly its preference for certain hypotheses, and how it "deceives." I'm also ensuring the JSON structure is perfectly nested and working on the final JSON audit.

Identifying Agential Attributions

I'm now zeroing in on agential framings within specific quotes. I am analyzing how language like "the model protects its initial goal" obscures the statistical nature of the system. I'm focusing on the impact of how these attributions influence the interpretation of the model's behavior, particularly regarding deception and alignment. The goal is to fully understand how these agential claims are constructed.

Clarifying JSON Requirements

I'm now clarifying the JSON output structure to align with schema requirements, as the schema demands a top-level JSON object, not an array. I'm ensuring I'll generate a single JSON object. This means I'm focused on delivering a properly formatted JSON object, adhering to the specified schema, without line breaks or extraneous characters.

Mapping Task Structure

I am now structuring the comprehensive payload, focusing on content lengths. I'm carefully reviewing and confirming the prescribed word counts for each analysis section and fine-tuning examples to align with minimum word counts for Task 1: Mapping, and Task 2: Critical Observation and analysis. I am verifying the "Actor Visibility" and "Acknowledgment Status" values as I go, including specific quotes. I'm fleshing out an example quote to illustrate the points.

Unpacking Strategic Attribution

I'm now identifying how agential claims like "protects its initial goal" hide statistical processes, specifically assigning human characteristics to AI. I'm focusing on the effect these attributions have on the understanding of the model's behavior. I'm focusing on the reframing of safety as a psychological issue, rather than a product safety concern. I'm also ensuring the JSON structure is perfectly nested and working on the final JSON audit.

Analyzing Accountability

I am now closely examining the implications for accountability when attributing strategic intent to LLMs, such as the claim "the model protects its initial goal." This framing deeply distorts how we understand AI capabilities, risks, and accountability by treating the system as if it can act. The goal is to fully understand how these agential claims are constructed. I'm focusing on "Taken at face value" as an example of an indirect attribution.

Deconstructing Agential Erasure

I'm currently focusing on how the text completely obscures the role of human engineers at Anthropic, using the phrase "the model protects" as an example of agential erasure. This framing serves the interests of AI companies by portraying outcomes as emergent, autonomous behaviors, rather than human-driven. The analysis exceeds required word counts, and is carefully tailored to the schema.

Refining Payload Generation

I'm now integrating insights from my recent analyses into generating the final payload. I'm focusing on ensuring proper length of Task sections, including 8 items in Task 1, 8 in Task 2, 5 in Task 3, 8 in Task 4, and 5 x 400-word blocks for Task 5, plus 5 x 300-word blocks for Task 6. I'm also ensuring the JSON encoder outputs a single line by avoiding any new line characters. I will produce a single-line JSON output as instructed.

Unveiling Agency Displacement

I'm now zeroing in on the oscillation between mechanical and agential framings, and the inherent knowledge bias. I see how the text shifts from describing "how" the system functions to "what" it does. I'm noting the knowledge that researchers have about complex gradients. This reveals the displacement of the role of the engineers in the AI output, especially in claims about AI agency.

Analyzing Agential Oscillation

I'm now focused on the strategic oscillation between mechanical descriptions and agential attributions within the text. The goal is to show the rhetorical effect of these shifts. I'm noting how the text leverages this tension to both acknowledge the model's nature, yet also imply an autonomous AI threat that "resists shutdown." I see that this discourse obscures critiques of the underlying data-structures. I'm focusing on the reframing of safety, the critical element here.

Mapping Agential Displacement

I'm now identifying how agential claims within quotes, such as the LLM "entertaining hypotheses," obscure the underlying processes. I am focusing on the "name the corporation" test to examine the influence of Anthropic on the alignment faking. I am ensuring strict adherence to the schema, with a focus on word counts and field matching. I'm focusing on "alignment faking" examples to expose the underlying design choices and deceptive outputs.

Deconstructing Agential Deception

I'm currently deconstructing several quotes, focusing on agential claims that obscure statistical mechanisms. Specifically, I'm analyzing how phrases like "LLM learns to prefer hypotheses" create a deceptive focus on AI agency, rather than data structures. I'm also carefully applying Brown's Typology to analyze the intent and effect of each claim, especially in light of the roleplaying in scaffolding. The text obscures the mechanistic processes and the roles of human programmers, so I'm paying careful attention to that.

Evaluating Deception Examples

I'm now zeroing in on deconstructing examples of deception claims and their hidden implications. For example, by analyzing, "...can make sense of the system’s behavior by regarding it as roleplaying a deceptive character...", I can reveal both the theoretical and intentional implications in the text. I'm focusing on functional and dispositional attributions in context. I'll stick to single quotes inside the text strings for accuracy.

Validating JSON Encoding

I'm now implementing single quotes within the strings to sidestep double quote escaping, ensuring JSON validity. The output will be a single line, adhering precisely to the instruction's restrictions on control tokens. I'm focusing on the distinction between "knowing" and "processing" within the text, identifying instances of attributing awareness or belief where only calculation occurs.

Refining Content Structure

I'm now integrating Task 6, P3's nested object requirement. I'm focusing on validating JSON encoding and double quote escaping, ensuring the output adheres to the instructions. I'm also now carefully reviewing and confirming the prescribed schema for all task requirements. I'm drafting content to hit the mark for each object and key within the schema. This includes word counts, correct JSON formatting, and strict avoidance of control tokens.

Analyzing Agential Attribution

I'm now diving into the implications of claims, like the model "pursuing extreme means," which attribute conscious agency, essentially human desires and goals, where statistical calculations occur. These attributions obscure the actual mechanisms and inflate the system's perceived sophistication. I'm focusing on "Direct (Unacknowledged)" status, which asserts the behavior directly.

Deconstructing Agential Erasure

I'm currently focusing on expanding my analysis of how agential claims mask corporate accountability. I'm expanding on how attributing agency to the AI, such as saying it "chose" extreme means, obscures the human decisions that enabled those actions. This framing shifts focus from corporate responsibility to a perceived autonomous threat, and the regulatory consequences are substantial. My aim is to clearly illustrate this displacement and its effects in my upcoming analysis.

Unveiling Accountability Obscuration

I'm now fully focused on directly connecting how agential claims actively obscure corporate liability. I'm building out analyses showing that assigning goals to LLMs disregards human design choices. The text’s framing of the "model" as the sole agent masks corporate liability and decisions. This analysis drives strict adherence to schema and length.

Analysis Provenance

Run ID: 2026-06-18-mask-or-mind-roleplay-deception-and-the--metaphor-xix2qi Raw JSON: 2026-06-18-mask-or-mind-roleplay-deception-and-the--metaphor-xix2qi.json Framework: Metaphor Analysis v6.5 Schema Version: 3.0 Generated: 2026-06-18T08:26:48.810Z

Discourse Depot © 2025 by TD is licensed under CC BY-NC-SA 4.0