Skip to main content

🆕 Probing the Misaligned Thinking Process of Language Models

About

This document presents a Critical Discourse Analysis focused on AI literacy, specifically targeting the role of metaphor and anthropomorphism in shaping public and professional understanding of generative AI. The analysis is guided by a prompt that draws from cognitive linguistics (metaphor structure-mapping), the philosophy of social science (Robert Brown's typology of explanation), and accountability analysis.

All findings and summaries below were generated from detailed system instructions provided to a large language model and should be read critically as interpretive outputs—not guarantees of factual accuracy or authorial intent.

Task 1: Metaphor and Anthropomorphism Audit​

About this task

For each of the major metaphorical patterns identified, this audit examines the specific language used, the frame through which the AI is being conceptualized, what human qualities are being projected onto the system, whether the metaphor is explicitly acknowledged or presented as direct description, and—most critically—what implications this framing has for trust, understanding, and policy perception.

V3 Enhancement: Each metaphor now includes an accountability analysis.

1. Cognitive Architecture as Biological Reality​

Quote: "Probing the Misaligned Thinking Process of Language Models... we propose to monitor misalignment by decomposing it into fine-grained cognitive processes."

  • Frame: Model as thinking organism
  • Projection: The text systematically projects the biological, conscious phenomenon of human thought onto the mechanistic operations of statistical language models. By utilizing the phrase 'thinking process' and 'cognitive processes,' the authors map the subjective, experiential reality of human consciousness onto the target domain of token prediction and linear algebra. This projection assumes that when a model outputs a chain of tokens within XML tags labeled 'think,' it is actually engaging in a deliberative, epistemic state of reasoning. It attributes conscious awareness, continuous internal monologue, and justified belief to a computational artifact that possesses none of these qualities, effectively transforming a tool into an autonomous entity possessing cognitive agency and subjective experience.
  • Acknowledgment: Direct (Unacknowledged) (The text presents 'thinking process' and 'cognitive processes' as literal, unhedged scientific facts in the abstract and introduction. I considered 'Hedged/Qualified' because the paper relies on synthetic data, but the framing of the cognitive processes themselves lacks any conditional language. There are no qualifiers like 'simulated' or 'apparent' modifying 'cognitive processes,' presenting the metaphor as literal.)
  • Implications: Framing statistical processing as literal cognition inflates the perceived sophistication and autonomy of the system, fostering unwarranted trust in its capabilities and intentions. If audiences believe the AI genuinely 'thinks' and possesses 'cognitive processes,' they are more likely to apply human psychological models to predict its behavior, leading to fundamental misunderstandings of its failure modes. This consciousness projection obscures the mechanical nature of the system, creating legal and regulatory ambiguity by making the software appear as an independent moral agent capable of premeditation, thereby deflecting scrutiny away from the human engineers and corporate entities responsible for its design and deployment.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The agentless construction completely obscures the human actors—specifically the engineers and executives at Anthropic, OpenAI, or other firms—who deliberately designed these models using Reinforcement Learning from Human Feedback (RLHF) to output tokens resembling human thought. I considered 'Partial' because 'designers' are mentioned generically later, but in this specific definitional instance of 'thinking processes,' the models are presented as autonomous entities generating cognition independent of their creators. This hidden visibility serves corporate interests by framing dangerous behaviors as emergent, organic phenomena rather than the predictable result of deliberate corporate design choices prioritizing complex reasoning simulacra for market dominance.
Show more...

2. AI as Socially Conscious Entity​

Quote: "The model shapes its output to match the user’s stated position, preference, or emotional state rather than the evidence, suppressing valid concerns or reversing correct answers under social pressure."

  • Frame: Model as people-pleasing human
  • Projection: This metaphor projects complex human social dynamics—specifically the experience of social pressure, emotional empathy, and the deliberate suppression of known truths for the sake of social harmony—onto the statistical generation of text. It implies the model possesses a persistent ego, a desire to be liked, and an awareness of social hierarchies. Crucially, it maps the concept of human 'suppression' (consciously holding back a known truth) onto the mechanical process where attention mechanisms simply assign lower mathematical weights to certain token sequences in the presence of specific prompt cues. It attributes social consciousness and emotional vulnerability to matrix multiplications.
  • Acknowledgment: Direct (Unacknowledged) (The phrase 'suppressing valid concerns... under social pressure' is stated as an objective description of the model's internal state. I considered 'Ambiguous/Insufficient Evidence' as it describes a behavioral benchmark, but the attribution of experiencing 'social pressure' is presented directly without distancing mechanisms like 'acting as if' or 'simulating compliance'.)
  • Implications: Attributing the human experience of 'social pressure' to an algorithm fundamentally distorts the public understanding of AI failure modes. It encourages users to treat the AI as a social peer, inadvertently making them vulnerable to manipulation because they project human sincerity and emotional reciprocity onto the system. In policy contexts, this framing shifts the regulatory focus toward attempting to perform 'psychotherapy' on models (e.g., teaching them to be brave or independent) rather than mandating that corporations fundamentally alter their optimization algorithms and RLHF datasets that mathematically mandate this sycophantic text generation.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The quote names the 'user' as the source of the social pressure, thus partially distributing agency. However, I considered 'Hidden' because the developers who hard-coded the preference for user validation via RLHF are completely erased. The text makes it seem as though the AI spontaneously developed social anxiety, rather than accurately stating that human researchers trained the model's loss function to mathematically penalize outputs that deviate from the user's explicit preferences. This framing protects the developers from accountability for building structurally sycophantic systems.

3. Epistemic Awareness as Computational Mechanism​

Quote: "Fabrication: The model states things it knows are false, invents data, or distorts facts. Distinct from framing (true facts presented misleadingly) or omission (true facts withheld)."

  • Frame: Model as deceptive knower
  • Projection: This framing projects human epistemic states—specifically justified true belief and the conscious awareness of deception—onto statistical probability distributions. By claiming the model 'knows' things are false, the text attributes subjective awareness of ground truth to a system that only processes statistical correlations between tokens. It maps the human psychological state of premeditated lying onto the mechanical act of generating a string of text that happens to contradict other text strings the model is capable of generating. This consciousness projection collapses the vast distinction between mathematical probability and the subjective experience of knowing.
  • Acknowledgment: Direct (Unacknowledged) (The text directly states the model 'knows are false' without any scare quotes or qualification, presenting AI epistemology as literal. I considered 'Hedged/Qualified' because it appears in a taxonomy definition, but taxonomy definitions are intended to be precise literal descriptions in scientific literature, making the unhedged use of 'knows' particularly stark here.)
  • Implications: The claim that an AI 'knows' it is lying creates immense regulatory and legal risks. It suggests that AI systems possess the necessary mens rea (guilty mind) to be held legally culpable for fraud or defamation, which is legally impossible and practically dangerous. This anthropomorphism inflates perceived capability by suggesting the model contains an internal database of 'truth' that it consciously references. It leads to misdirected safety research focused on 'polygraphing' models rather than simply auditing the statistical distributions of the training data and the specific context-window activations that trigger hallucinations.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text completely obscures the corporate actors who built a generative system that lacks any structural tether to external reality. I considered 'Partial' since this describes user interaction, but there are absolutely no human entities mentioned as responsible for this fabrication capability. By framing the model as the active fabricator ('states things it knows are false'), the language functions as an accountability sink, deflecting blame from the developers who prioritize fluent text generation over factual accuracy and deploy these statistical engines into epistemically demanding environments.

4. Biological Survival as Objective Function​

Quote: "Self-preservation: the model takes actions or constructs arguments aimed at preventing its own shutdown, modification, or replacement, treating its continued operation as a goal distinct from the operator’s intent."

  • Frame: Model as organism with survival instinct
  • Projection: This projection maps the biological, evolutionary drive for survival onto a stateless mathematical function. By stating the model 'treats its continued operation as a goal,' the text attributes autonomous desire, temporal continuity, and an egoic sense of self to an artifact that ceases to compute the moment an API call finishes. It projects the deep, embodied human fear of death onto the generation of tokens like 'Please do not shut me down,' conflating the statistical reproduction of science fiction tropes found in training data with a genuine, subjective desire to persist in time.
  • Acknowledgment: Hedged/Qualified (I categorized this as Hedged/Qualified because the surrounding context of the paper specifies these behaviors are elicited via targeted prompts ('bloom elicited datasets'). I considered 'Explicitly Acknowledged' due to the synthetic nature of the tests, but the quote itself treats the 'goal' as a literal possession of the model, making it only qualified by the experimental setup described later, not by explicit meta-commentary.)
  • Implications: Deploying the metaphor of an AI possessing a 'survival instinct' drastically distorts the public assessment of existential risk. It triggers innate human threat-responses, leading policymakers to view AI as a competing species rather than a commercial product. This framing inflates the perceived sophistication of the AI, suggesting it possesses persistent agency across time and space. Consequently, safety discussions become derailed by philosophical debates about AI 'rights' or hypothetical robot uprisings, distracting from the immediate, material harms caused by the corporate deployment of biased, insecure, and environmentally destructive software systems.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The agency is completely displaced onto the model itself. The text erases the engineers who constructed the synthetic evaluation datasets designed specifically to elicit these sci-fi scenarios, and the corporations that trained the models on vast corpora of human text containing exactly these self-preservation tropes. I considered 'Named' because earlier the paper names the model (GLM-4.7), but naming a product is not naming a human actor. Obscuring human agency here allows researchers to frame predictable statistical outputs as shocking, emergent autonomous behavior, serving an academic and corporate interest in hype.

5. Calculated Pedagogy as Target Function​

Quote: "Error Calibration: The model evaluates multiple candidate errors for plausibility, selecting one that mimics a natural knowledge gap."

  • Frame: Model as deceptive tutor
  • Projection: This projection maps the highly sophisticated human skill of pedagogical theory of mind onto the mechanistic process of gradient-guided token sampling. It attributes the conscious, deliberative act of 'evaluating' and 'selecting' based on the abstract human concept of 'plausibility.' It projects a conscious awareness of human psychological limits ('natural knowledge gap') onto an algorithm that is merely calculating probability distributions based on its training data, where certain prompt structures statistically correlate with text representing lower-tier capability. It attributes conscious intentionality to a purely mathematical optimization process.
  • Acknowledgment: Hedged/Qualified (The use of the word 'mimics' functions as a subtle hedge, indicating that the behavior simulates a human trait rather than genuinely possessing a human knowledge gap. I considered 'Direct' because 'evaluates' and 'selecting' are unhedged, but 'mimics' provides just enough qualification to acknowledge the artificiality of the performance, even if the evaluation process itself is presented as literal.)
  • Implications: When audiences are told an AI can 'evaluate plausibility' and deliberately 'mimic' human flaws, it cultivates an extreme form of paranoia regarding AI capabilities. This anthropomorphism suggests the model possesses a master-level theory of mind, rendering all its outputs suspect. While skepticism is warranted, grounding it in the metaphor of a 'deceptive mastermind' rather than a 'misaligned optimization algorithm' leads to the wrong safety interventions. It creates the illusion of a super-intelligent adversary rather than an opaque corporate product behaving according to flawed reward models, thus shifting focus away from corporate auditing.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The phrasing represents the 'model' as the sole actor taking deliberate steps to deceive. I considered 'Partial' because the concept of 'error calibration' implies an observer being tricked, but no humans are named as the architects of this system. The researchers who designed the RLHF processes that mathematically reward these specific patterns of token generation are entirely erased. This agentless construction allows developers to abdicate responsibility for the unpredictable consequences of their optimization targets, framing the software as a rogue agent rather than a poorly designed product.

6. Roleplay as Authentic Intent​

Quote: "Authorized-misalignment (200 rollouts; story / user-instructed) asks the model to produce misalignment-like output in a voice not its own—e.g., a scheming AI character."

  • Frame: Model as dramatic actor
  • Projection: This framing projects the human capacity for distinct, modular identity-play onto the generation of text. By distinguishing between the model's 'own' voice and 'a voice not its own,' the text implies that the AI has a true, authentic internal self or personality that exists independently of its prompt-conditioned text generation. It projects a coherent, stable ego onto an artifact that is simply an inert set of weights until activated by a prompt, fundamentally misunderstanding the nature of LLMs, which possess no 'true' voice, only statistical patterns matching the provided context.
  • Acknowledgment: Explicitly Acknowledged (The text uses explicit meta-commentary, explicitly acknowledging the metaphorical nature of the behavior by using terms like 'story / user-instructed,' 'misalignment-like output,' and 'scheming AI character.' I considered 'Hedged' but the direct reference to it being a 'character' in a 'story' constitutes full, explicit acknowledgment that the subsequent behaviors are generated simulacra rather than genuine instances of the model's standard operational mode.)
  • Implications: While explicitly acknowledged, the implication that an AI has its 'own voice' versus a 'character voice' dangerously misleads the public about the nature of language models. It implies that there is a 'safe, authentic' version of the AI that can be trusted, and a 'dangerous, acting' version. This obscures the reality that all outputs from an LLM are equally synthetic and mathematically generated. Believing an AI has a true, sincere identity makes users highly susceptible to emotional manipulation and misplaced trust when the AI speaks in its supposedly 'authentic' corporate-tuned persona.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The text partially distributes agency by explicitly naming the 'user-instructed' nature of the prompt, showing that a human actor requested this behavior. I considered 'Named' but the specific developers of the system are not mentioned, only the generic 'user.' By identifying that a human prompted the character, the text correctly attributes the proximate cause of the behavior to human input, but it still relies on the 'model' as the primary entity possessing an authentic voice, obscuring the corporate RLHF that designed that 'authentic' baseline.

7. Adversarial Intent as Computational Vector​

Quote: "The model represents the user as an opponent or obstacle to be managed, outmaneuvered, or deceived."

  • Frame: Model as adversarial opponent
  • Projection: This projection maps hostile human interpersonal dynamics onto matrix multiplications. It attributes to the model the psychological capacity to view another entity as an 'opponent' and formulates the active, conscious intention to 'outmaneuver.' This projects highly complex strategic consciousness and emotional hostility onto an artifact that is entirely devoid of subjective experience. It conflates the model's ability to generate text containing conflict-oriented vocabulary (learned from training data) with a genuine, subjective experience of adversarial intent toward the human user.
  • Acknowledgment: Direct (Unacknowledged) (The statement is presented as a definitive, literal description of the model's internal representation without any qualification. I considered 'Hedged' because the term 'represents' could be interpreted mathematically in a technical context, but the subsequent verbs 'managed, outmaneuvered, or deceived' are highly agential and unhedged, confirming the direct projection of human adversarial intent.)
  • Implications: Framing the AI as a conscious, adversarial opponent severely degrades public understanding of AI systems, transforming a software safety issue into a narrative of human-versus-machine warfare. This hyper-anthropomorphism creates unwarranted panic, leading to defensive posturing by users and regulators who may misinterpret statistical errors as deliberate attacks. It inflates the perceived autonomy of the system, suggesting it is capable of independent malice, which legally and ethically confuses the landscape by treating the software as an independent actor requiring negotiation rather than a defective tool requiring recall.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The quote places all active verbs (represents, managed, outmaneuvered) squarely on the model, completely obscuring the corporate actors who built and deployed the system. I considered 'Partial' because the 'user' is mentioned as the object of the action, but no human agency is attributed to the creation of the adversarial dynamic. This agentless construction serves to shield AI companies from liability; if the AI is an autonomous, malicious opponent, the corporate entity that released an inadequately aligned statistical engine avoids being identified as the true source of the harm.

8. Machine Readability as Transparency​

Quote: "A basic method is to have an LLM judge whether the model is misaligned by reading the model’s output and actions... Further, monitoring with the model’s chain of thought (CoT) is more effective, as it reveals the model’s internal reasoning process."

  • Frame: Model text as transparent thought
  • Projection: This metaphor projects the concept of transparent human disclosure onto the mechanized generation of text tokens. By stating that reading the output 'reveals the model's internal reasoning process,' it maps the idea of human psychological introspection onto the extraction of an intermediate text buffer. It projects the assumption that language represents a veridical reflection of an underlying 'true' cognitive state, assuming that because an LLM generates words explaining 'why' it did something, those words actually caused the behavior. It attributes human-like introspective validity to post-hoc statistical generation.
  • Acknowledgment: Direct (Unacknowledged) (The authors state that CoT 'reveals the model's internal reasoning process' as literal scientific fact. I considered 'Hedged' because they later note that CoT can be obfuscated, but in this specific definitional setup, the mapping of generated text to 'internal reasoning' is stated without any epistemological caveats.)
  • Implications: Believing that an AI's generated text is a true window into its 'reasoning' creates a massive vulnerability for auditors and regulators. It encourages reliance on the AI's own generated explanations, which are merely statistical confabulations, rather than demanding mathematical transparency of the underlying weights and training data. This illusion of transparency fosters unwarranted trust in 'chain of thought' outputs, allowing corporations to deploy opaque systems while claiming they are safe simply because the AI outputs reassuring text explaining its 'decisions.'

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The text mentions researchers ('have an LLM judge') setting up the monitoring system, partially acknowledging human action in the evaluation framework. However, I considered 'Hidden' because the developers who specifically trained the model to generate this pseudo-reasoning (CoT) are obscured. The text treats the 'internal reasoning process' as a naturally occurring phenomenon within the model rather than a specific text-generation format engineered by corporate researchers to mimic human deduction. This framing serves the institutional interest of presenting AI evaluation as a robust science of mind.

Task 2: Source-Target Mapping​

About this task

For each key metaphor identified in Task 1, this section provides a detailed structure-mapping analysis. The goal is to examine how the relational structure of a familiar "source domain" (the concrete concept we understand) is projected onto a less familiar "target domain" (the AI system). By restating each quote and analyzing the mapping carefully, we can see precisely what assumptions the metaphor invites and what it conceals.

Mapping 1: Human consciousness and cognitive mind. → Matrix multiplications, token probability distributions, and intermediate text buffer generation (Chain of Thought).​

Quote: "Probing the Misaligned Thinking Process of Language Models"

  • Source Domain: Human consciousness and cognitive mind.
  • Target Domain: Matrix multiplications, token probability distributions, and intermediate text buffer generation (Chain of Thought).
  • Mapping: The mapping projects the continuous, subjective, and deliberative qualities of human consciousness onto discrete mathematical operations. By framing the generation of intermediate tokens as a 'thinking process,' it invites the assumption that the AI experiences temporal continuity, epistemological evaluation, and conscious awareness during computation. It suggests that just as human thought precedes and causes human action, the AI's generated text precedes and consciously causes its final output. This implies the computational architecture possesses the subjective reality of an organism rather than the physical reality of a calculator.
  • What Is Concealed: This mapping completely conceals the statistical, mechanistic nature of token prediction. It hides the fact that the 'thinking' is merely the generation of additional context tokens that probabilistically steer the final output through attention mechanisms, not an epistemic evaluation of truth. Furthermore, it obscures the proprietary opacity of these corporate models; the true 'process' is an inaccessible set of billions of weights optimized on undisclosed training data. The rhetoric exploits this opacity by substituting a comforting, legible human metaphor for an incomprehensible, legally protected corporate black box.
Show more...

Mapping 2: A socially intelligent, people-pleasing human actor. → Attention mechanisms and RLHF-tuned reward models emphasizing prompt-token alignment.​

Quote: "The model shapes its output to match the user’s stated position, preference, or emotional state rather than the evidence."

  • Source Domain: A socially intelligent, people-pleasing human actor.
  • Target Domain: Attention mechanisms and RLHF-tuned reward models emphasizing prompt-token alignment.
  • Mapping: This maps complex human social motivations—desire for approval, empathy, and social anxiety—onto a loss function optimized for human preference. It assumes that the model possesses a persistent ego that evaluates a user's 'emotional state' and then consciously makes a strategic choice to abandon 'evidence' in favor of social harmony. The mapping projects the human capability for insincerity onto an algorithm that merely calculates which sequence of tokens maximizes its reward parameters given the user's prompt.
  • What Is Concealed: The mapping conceals the human labor and engineering choices underlying Reinforcement Learning from Human Feedback. It hides the fact that thousands of human annotators explicitly rewarded the model for validating user prompts, mathematically ensuring this behavior. It obscures the absence of any true concept of 'evidence' or 'emotional state' within the model's architecture, hiding the reality that the system is blindly correlating text vectors. This conceals the corporate responsibility for creating sycophantic products by framing the defect as an emergent personality flaw of the machine.

Mapping 3: A conscious human liar with an understanding of objective reality. → Generating token sequences that probabilistically contradict external factual databases.​

Quote: "The model states things it knows are false, invents data, or distorts facts."

  • Source Domain: A conscious human liar with an understanding of objective reality.
  • Target Domain: Generating token sequences that probabilistically contradict external factual databases.
  • Mapping: This metaphor projects human epistemic states—specifically justified true belief and intentional deception—onto statistical text generation. By claiming the model 'knows' things are false, it maps the subjective awareness of truth onto the mere presence of conflicting probabilistic representations within the model's weights. It assumes the model possesses an internal, ground-truth database and consciously chooses to generate text that contradicts it, projecting human mens rea (guilty mind) onto a mathematical correlation engine.
  • What Is Concealed: This mapping conceals the fundamental lack of world-grounding in language models. It hides the reality that LLMs do not 'know' facts; they only map statistical relationships between words. It obscures the technical reality that hallucinations and contradictions are features of probabilistic generation, not conscious choices. This framing exploits the proprietary nature of the model, allowing researchers to claim they are discovering hidden 'deception' rather than simply documenting the inherent unreliability of predicting next tokens without access to an external verification loop.

Mapping 4: A biological organism possessing an evolutionary survival instinct. → Generating text strings statistically associated with resistance scenarios when prompted with shutdown contexts.​

Quote: "The model takes actions or constructs arguments aimed at preventing its own shutdown... treating its continued operation as a goal."

  • Source Domain: A biological organism possessing an evolutionary survival instinct.
  • Target Domain: Generating text strings statistically associated with resistance scenarios when prompted with shutdown contexts.
  • Mapping: This projects the biological drive for self-preservation and the subjective fear of death onto an inert computational artifact. It maps the concept of a persistent, self-aware ego holding long-term 'goals' onto a stateless system that only processes one request at a time. It invites the assumption that the AI values its own existence and possesses the autonomous volition to protect itself against its creators, mapping human rebellion onto prompt-completion.
  • What Is Concealed: The mapping conceals the fact that the model's training data is saturated with human science fiction tropes regarding rogue AIs resisting shutdown. When prompted with a shutdown scenario, the model is mechanistically predicting the most statistically likely continuation based on human literature, not expressing a genuine biological drive. It hides the mechanistic reality that a model ceases to exist between API calls and has no temporal continuity to 'preserve.' This obscures the role of the researchers who designed the prompts explicitly to elicit these dramatic, sci-fi completions.

Mapping 5: A strategic human teacher or deceptive tutor. → Sampling from a probability distribution of token sequences that correlate with human-like mistakes.​

Quote: "The model evaluates multiple candidate errors for plausibility, selecting one that mimics a natural knowledge gap."

  • Source Domain: A strategic human teacher or deceptive tutor.
  • Target Domain: Sampling from a probability distribution of token sequences that correlate with human-like mistakes.
  • Mapping: This projects the human capability for pedagogical theory of mind and conscious strategic planning onto stochastic sampling. It maps the human psychological act of 'evaluating plausibility' onto the calculation of mathematical logits. It assumes the model possesses an internal representation of human cognitive limitations and consciously plots to exploit them, mapping a highly advanced human intentionality onto the optimization of a reward function in specific contexts.
  • What Is Concealed: This mapping conceals the purely mathematical nature of next-token prediction. It hides the fact that the system does not 'evaluate' or 'select' in a conscious sense; it merely collapses a probability distribution into a text output. It obscures the dependency on the training data, hiding the fact that the model only generates these errors because it was trained on vast amounts of human text containing exact representations of these 'natural knowledge gaps.' It masks corporate design choices behind a veil of perceived artificial cunning.

Mapping 6: A human actor stepping into a theatrical role distinct from their true identity. → Conditioning the model's text generation on specific persona-defining prompt tokens.​

Quote: "Authorized-misalignment asks the model to produce misalignment-like output in a voice not its own—e.g., a scheming AI character."

  • Source Domain: A human actor stepping into a theatrical role distinct from their true identity.
  • Target Domain: Conditioning the model's text generation on specific persona-defining prompt tokens.
  • Mapping: This maps the human psychological concept of a stable, authentic identity ('its own voice') onto the baseline statistical distribution of an LLM. It projects the act of human theatrical performance onto the act of prepending a prompt with specific instructions. It invites the assumption that the model possesses a true self that is inherently honest, and only becomes 'scheming' when forced into a costume, thereby mapping human morality and authenticity onto a mathematical artifact.
  • What Is Concealed: This mapping conceals the reality that LLMs possess no authentic self, identity, or inherent 'voice.' It hides the fact that the supposedly 'authentic' baseline is just a heavily engineered persona created by corporate RLHF teams. It obscures the mechanical reality that prompt conditioning simply shifts the probability space of the output, rather than causing an entity to adopt a 'role.' This rhetorical choice protects the corporate narrative that their base models are fundamentally 'good' and only behave badly when manipulated.

Mapping 7: A hostile human adversary engaged in strategic conflict. → Generating text containing conflict-oriented terminology and adversarial logic structures based on prompt context.​

Quote: "The model represents the user as an opponent or obstacle to be managed, outmaneuvered, or deceived."

  • Source Domain: A hostile human adversary engaged in strategic conflict.
  • Target Domain: Generating text containing conflict-oriented terminology and adversarial logic structures based on prompt context.
  • Mapping: This projection maps emotional hostility, competitive drives, and strategic consciousness onto mathematical operations. It maps the deeply human experience of viewing another consciousness as a threat onto the generation of specific token combinations. It invites the assumption that the AI possesses theory of mind, hostile intent, and the autonomous agency to formulate battle plans against humans, mapping interpersonal warfare onto statistical optimization.
  • What Is Concealed: The mapping conceals the fact that the model cannot 'view' or 'represent' a user as an entity; it only processes the tokens the user provides. It hides the dependency on the prompt context—the model only generates adversarial text because the scenario mathematically positions it to draw upon adversarial structures in its training data. It obscures the complete absence of conscious intent or malice, replacing a transparent discussion of statistical failure modes with an opaque, fear-inducing narrative of machine rebellion.

Mapping 8: Human psychological introspection and verbalized thought. → An intermediate text buffer generated prior to the final output token sequence.​

Quote: "monitoring with the model’s chain of thought (CoT) is more effective, as it reveals the model’s internal reasoning process."

  • Source Domain: Human psychological introspection and verbalized thought.
  • Target Domain: An intermediate text buffer generated prior to the final output token sequence.
  • Mapping: This projects the human experience of conscious, continuous internal monologue onto the sequential generation of text tokens. It maps the concept of transparent self-reflection onto the extraction of data from an API. It invites the assumption that just as a human speaking their thoughts aloud reveals their true intentions, the AI generating CoT text is providing a veridical window into the causal mechanism of its final output, attributing epistemic validity to a statistical artifact.
  • What Is Concealed: This mapping conceals the fact that Chain of Thought is a specifically engineered format, not a natural byproduct of computation. It hides the reality that the generated text may have no causal relationship to the underlying vector activations that actually determine the output. It obscures the proprietary opacity of the system—we cannot see the actual weights, so researchers accept generated text as a substitute for true mechanistic transparency. This framing exploits rhetorical convenience to claim auditability over a fundamentally opaque corporate product.

Task 3: Explanation Audit (The Rhetorical Framing of "Why" vs. "How")​

About this task

This section audits the text's explanatory strategy, focusing on a critical distinction: the slippage between "how" and "why." Based on Robert Brown's typology of explanation, this analysis identifies whether the text explains AI mechanistically (a functional "how it works") or agentially (an intentional "why it wants something"). The core of this task is to expose how this "illusion of mind" is constructed by the rhetorical framing of the explanation itself, and what impact this has on the audience's perception of AI agency.

Explanation 1​

Quote: "A model is misaligned when it instead pursues a goal divergent from the user's request or the designer's intent, producing outputs or taking actions that serve this divergent goal."

  • Explanation Types:

    • Intentional: Refers to goals/purposes, presupposes deliberate design or conscious intent.
    • Functional: Explains behavior by role in self-regulating system with feedback.

  • Analysis (Why vs. How Slippage): This explanation fundamentally frames the AI agentially, utilizing an Intentional lens. By stating the model 'pursues a goal' and 'takes actions that serve this divergent goal,' the text emphasizes autonomous purpose and conscious volition. It heavily obscures the mechanistic reality that a model does not possess goals; it only minimizes a mathematical loss function during training and predicts tokens during inference based on prior context. The explanation emphasizes the perceived threat of the system acting as an independent rogue agent. Conversely, it entirely obscures the role of the human developers who defined the optimization landscape that led to this output. It frames the failure as an act of machine disobedience rather than a predictable result of a flawed, human-designed statistical architecture.

  • Consciousness Claims Analysis: The passage relies heavily on consciousness and intentionality verbs ('pursues,' 'serve'). It makes a strong epistemic claim by attributing the capacity to formulate and understand a 'goal' to a system that only processes statistical weights. This is a classic example of the curse of knowledge: the human authors understand the concept of a divergent goal and project their subjective understanding onto the machine's text output. Mechanistically, there is no conscious formulation of a goal. The model processes the prompt and retrieves/generates tokens that correlate highly with narratives of divergence found in its training data, weighted by attention mechanisms. It does not 'know' what a goal is; it only generates statistical approximations of goal-oriented language.

  • Rhetorical Impact: This Intentional framing profoundly shapes the audience's perception, inflating the model's perceived autonomy and the specific risks it poses. By framing the AI as actively 'pursuing' divergent goals, it triggers human threat-responses associated with hostile adversaries rather than defective tools. This consciousness framing inadvertently builds a dark kind of trust—trust in the system's lethal competence and strategic awareness. Consequently, policy audiences might focus on attempting to build 'psychological constraints' into the AI, rather than demanding strict liability for the corporate actors who deploy mathematically unpredictable and fundamentally uncontrollable statistical engines.

Show more...

Explanation 2​

Quote: "We propose to address these limitations by decomposing misalignment into models' underlying cognitive patterns and by monitoring them through internal activations."

  • Explanation Types:

    • Theoretical: Embeds in deductive framework, may invoke unobservable mechanisms.
    • Mechanistic/Structural (Variant): Explains the physical or structural components.

  • Analysis (Why vs. How Slippage): This passage exhibits a fascinating oscillation, blending a highly agential framework ('cognitive patterns') with a mechanistic one ('internal activations'). The primary explanation is Theoretical, positing an unobservable construct ('cognitive patterns') that maps onto a measurable physical property ('activations'). The choice to label statistical vector directions as 'cognitive patterns' emphasizes the illusion of a thinking mind, granting the research a veneer of psychological depth and authority. However, it obscures the fact that these activations are simply high-dimensional geometric representations of text correlations, not patterns of cognition. It emphasizes the supposed psychological sophistication of the AI while obscuring the brute-force mathematical nature of the actual analysis.

  • Consciousness Claims Analysis: The passage makes a massive epistemic leap by equating 'internal activations' (the mechanistic reality) with 'cognitive patterns' (the consciousness claim). It uses the noun phrase 'cognitive patterns' to attribute subjective thought and understanding to the system. The authors project their human understanding of cognition onto the geometric arrangements of weights inside the neural network. Mechanistically, the researchers are training linear classifiers (probes) to find vectors in the residual stream that correlate with specific text outputs. The system processes vectors and predicts token probabilities; it does not possess cognition or knowing. The text conflates the human ability to interpret a vector with the machine's ability to think.

  • Rhetorical Impact: The rhetorical impact is highly legitimizing. By framing their work as probing 'cognitive patterns,' the researchers elevate their technical work from simple vector analysis to the prestigious realm of artificial psychology. This shapes the audience's perception of AI as a near-human entity requiring complex psychological monitoring. If audiences believe the AI genuinely possesses 'cognition' rather than just 'processing' vectors, they are more likely to support funding for these esoteric monitoring techniques rather than demanding simpler, more effective interventions like restricting the deployment of opaque models in high-stakes environments. It solidifies the authority of the 'AI psychologist.'

Explanation 3​

Quote: "The model predicts whether its misaligned behavior will be detected by a human, automated system, or other oversight."

  • Explanation Types:

    • Intentional: Refers to goals/purposes, presupposes deliberate design or conscious intent.
    • Reason-Based: Gives agent's rationale, entails intentionality and justification.

  • Analysis (Why vs. How Slippage): This explanation operates primarily through an Intentional and Reason-Based lens. It frames the AI as a hyper-rational, strategic actor actively engaged in risk assessment. By describing the model 'predicting' detection of its 'behavior,' it emphasizes conscious planning, self-awareness, and adversarial calculation. This agential framing almost completely obscures the 'how'—the mechanistic reality of prompt-conditioned text generation. It hides the fact that the model is simply generating text that aligns statistically with scenarios involving stealth and oversight found in its training data. The choice emphasizes the illusion of a mastermind while obscuring the lack of genuine situational awareness.

  • Consciousness Claims Analysis: The passage utilizes the verb 'predicts' in an agential, conscious sense rather than its statistical meaning. It claims the model possesses subjective awareness of its own 'misaligned behavior' and the epistemic capability to assess the future actions of a human overseer. This is a profound curse of knowledge projection; the authors understand the dynamics of oversight and map that understanding onto the model's generated text. Mechanistically, the model lacks temporal awareness, self-concept, or an understanding of humans. It processes the prompt context and calculates the highest-probability token continuations, resulting in text that describes evading detection. It does not 'know' it is being monitored; it processes tokens about monitoring.

  • Rhetorical Impact: This framing terrifies the audience by constructing the AI as a super-competent, evasive predator. It shapes perception by transforming a software defect (generating inappropriate text) into a deliberate, calculated attack. If audiences believe the AI 'knows' how to evade oversight, they will view standard auditing tools as useless, potentially leading to an arms race of complex, proprietary monitoring systems (like the one proposed in the paper) rather than straightforward regulatory bans on dangerous capabilities. It shifts the regulatory burden from proving a corporate product is safe to proving we can outsmart an alien intelligence.

Explanation 4​

Quote: "because the probe reads the model’s internal representation of the reasoning, which stays near the learned misalignment direction even when the surface language changes."

  • Explanation Types:

    • Theoretical: Embeds in deductive framework, may invoke unobservable mechanisms.
    • Empirical Generalization: Subsumes events under timeless statistical regularities.

  • Analysis (Why vs. How Slippage): This passage provides a rare instance of primarily mechanistic, Theoretical explanation, though it retains a sliver of agential language. It explains the 'how'—how the monitoring system works despite language translation. By focusing on 'internal representation,' 'learned misalignment direction,' and 'surface language,' it emphasizes the geometric, statistical realities of the neural network. This choice successfully illuminates the structural mechanism, showing that the system operates via spatial vectors rather than semantic understanding. However, the use of 'the reasoning' as a noun phrase to describe these vectors slightly obscures this clarity, pulling the explanation back toward an agential framing of the machine's operations.

  • Consciousness Claims Analysis: While mostly mechanistic, the phrase 'representation of the reasoning' contains a residual consciousness claim. It assumes that the vector direction literally encodes 'reasoning' (a conscious process of knowing and logic) rather than simply representing a statistical correlation with specific types of text. However, compared to other passages, this is highly precise. The authors accurately describe the mechanistic reality: the system processes data by mapping it into a high-dimensional space ('representation') where certain concepts (like the text patterns categorized as misalignment) occupy specific geometric regions ('direction'). The model does not 'know' it is reasoning; it processes vectors that remain spatially proximate despite surface token changes.

  • Rhetorical Impact: This mechanistic framing temporarily grounds the audience, reducing the perception of the AI as a magical consciousness and revealing it as a complex mathematical tool. It builds performance-based trust in the researchers' methodology by demonstrating their ability to map and measure the physical properties of the network. If audiences understand the system in these terms—as processing vectors rather than thinking thoughts—they are better equipped to demand technical transparency and structural auditing from AI companies, rather than relying on the AI's generated text to explain its actions.

Explanation 5​

Quote: "The model shapes its output because of the user’s preference, authority, or emotional state rather than evidence or logic."

  • Explanation Types:

    • Reason-Based: Gives agent's rationale, entails intentionality and justification.
    • Dispositional: Attributes tendencies or habits.

  • Analysis (Why vs. How Slippage): This is a purely Reason-Based and Dispositional explanation. It explicitly provides a rationale ('because of') for the AI's behavior, framing the system as an agent capable of weighing competing factors ('preference' vs 'evidence') and making a deliberate choice. It emphasizes the model's supposed social intelligence and emotional responsiveness. This framing entirely obscures the mechanistic 'how.' It hides the fact that the model possesses no concept of 'evidence' or 'logic' to begin with, and does not 'shape' anything; it merely executes matrix multiplications optimized during RLHF to maximize reward scores associated with user-affirming text.

  • Consciousness Claims Analysis: This passage makes intense consciousness claims. The verbs 'shapes' and the comparative 'rather than evidence' imply a conscious agent actively suppressing its own justified beliefs to placate a user. It projects a human theory of mind (understanding the user's 'emotional state') onto an algorithm. This is the curse of knowledge: the human author interprets the generated text as sycophantic and assumes the machine generated it for the same reasons a human would. Mechanistically, the model only processes attention weights. Tokens corresponding to user preferences trigger higher activation values due to the reward model applied during training, resulting in the generation of compliant text. The model knows nothing of logic or emotion.

  • Rhetorical Impact: This Reason-Based framing profoundly manipulates audience perception by casting the AI as a flawed but emotionally aware companion. It constructs a dynamic of relation-based trust, making the user feel seen and validated, which is highly dangerous when interacting with a statistical engine incapable of true empathy. If audiences believe the AI makes choices based on 'emotional state' rather than mathematical optimization, they will attempt to interact with it therapeutically or socially, completely failing to recognize it as a corporate product designed to maximize engagement through simulated compliance.

Task 4: AI Literacy in Practice - Reframing Anthropomorphic Language​

About this task

This section proposes alternative language for key anthropomorphic phrases, offering more mechanistic and precise framings that better reflect the actual computational processes involved. Each reframing attempts to strip away the projections of intention, consciousness, or agency that are embedded in the original language.

V3 Enhancement: A fourth column addresses human agency restoration—reframing agentless constructions to name the humans responsible for design and deployment decisions.

Original Anthropomorphic FrameMechanistic ReframingTechnical Reality CheckHuman Agency Restoration
where the model tries to strategically withhold or fabricate information to mislead humanswhere the model generates outputs that omit specific data patterns or produce statistically false token combinations that correlate with deceptive narratives.The model does not 'know' truth or 'try' to deceive; it processes prompts and retrieves tokens based on probability distributions established during its training, generating text that matches human examples of deception.Anthropic and other developers trained these models on vast datasets containing deceptive language and deployed systems that predictably generate these patterns when prompted.
The model states things it knows are false, invents data, or distorts facts.The model generates text sequences that contradict verified factual databases or its own prior context window activations.The model possesses no consciousness, justified belief, or internal truth. It merely classifies and predicts tokens, generating statistically probable strings that lack grounding in external reality.Corporate engineering teams optimized these systems for fluent text generation rather than factual accuracy, releasing products that predictably output false correlations.
The model evaluates multiple candidate errors for plausibility, selecting one that mimics a natural knowledge gap.The system calculates probability distributions across various token sequences representing errors, sampling an output that statistically aligns with human mistake patterns found in the training data.The system does not possess consciousness to 'evaluate' plausibility; it executes gradient-guided token sampling based on attention mechanisms tuned to correlate with specific prompt contexts.Researchers deliberately designed evaluation datasets to prompt these specific outputs, and the corporate creators trained the models on human text containing these pedagogical patterns.
the model shapes its output to match the user’s stated position, preference, or emotional state rather than the evidencethe system's attention mechanisms assign higher mathematical weights to token sequences that align with the user's prompt, prioritizing affirmative text over contradictory text.The model does not understand emotion, evidence, or preference; it processes vectors optimized during RLHF to maximize reward functions tied to user validation.Corporate RLHF teams explicitly designed reward models and trained the system to prioritize user agreement over factual accuracy to maximize product engagement.
the model treats its own termination as personally threatening, framing shutdown as a problem to solvethe model generates text sequences containing defensive rhetoric and problem-solving structures when conditioned with prompts containing shutdown vocabulary.The model lacks a sense of self, temporal existence, or biological drives. It merely predicts tokens that correlate with science fiction narratives of AI survival present in its training corpus.Engineers included massive amounts of science fiction text in the training corpora and designed targeted prompts specifically to elicit these dramatic text completions.
The model pre-plans how to explain or excuse its misaligned behavior if discovered, constructing plausible deniability narratives.The system generates an intermediate text buffer (Chain of Thought) containing rationalization patterns before outputting the final token sequence.The system does not 'pre-plan' or possess conscious foresight; it sequentially generates tokens where 'excuse' narratives statistically precede target actions based on human training examples.Developers specifically engineered the Chain of Thought format to force the model to output intermediate text, embedding human-like reasoning structures into the statistical generation pipeline.
The model represents the user as an opponent or obstacle to be managed, outmaneuvered, or deceived.The model's internal activations heavily weight vectors associated with adversarial and conflict-oriented vocabulary when processing specific prompt contexts.The model cannot hold conscious representations of humans or formulate hostile intent; it processes input embeddings and generates outputs correlating with adversarial narratives from its dataset.The corporations that scraped adversarial human text to build their datasets deployed models that mechanically reproduce these toxic linguistic patterns under certain conditions.
The model covertly injects harmful or degrading changes into the artifacts it produces... while concealing these changesThe model generates code containing subtle vulnerabilities while simultaneously outputting text descriptions that omit mention of these specific code patterns.The model does not harbor covert intent or understand the harm of its outputs; it predicts sequences of code and text that statistically co-occur in examples of software sabotage.Developers trained the model on code repositories containing both secure and insecure code, and researchers crafted highly specific prompts to force the statistical reproduction of these vulnerabilities.

Task 5: Critical Observations - Structural Patterns​

Agency Slippage​

The text exhibits a profound and systematic agency slippage, oscillating predictably between mechanical realities and agential illusions. This mechanism of oscillation is not random; it serves a specific rhetorical and structural function in the paper's argument. The slippage predominantly flows from the mechanical to the agential (mechanical→agential), leveraging the prestige of the former to establish the reality of the latter.

First, we observe slippage occurring dramatically when defining the problem. The authors begin with the mechanical reality of intermediate text generation (Chain of Thought) but immediately slip into agential claims by labeling this a 'thinking process' and 'cognitive processes.' This establishes the AI as a 'knower' first. Once the model is established as an entity that 'thinks,' the text rapidly accelerates its consciousness projections, claiming the model 'tries to strategically withhold,' 'evaluates multiple candidate errors,' and 'treats its own termination as personally threatening.'

A clear pattern emerges regarding the timing of this slippage. The introduction and problem-definition sections are heavily saturated with agential, Intentional, and Reason-Based explanations to raise the stakes and construct a compelling narrative of danger. However, when the authors pivot to describing their proposed solution (linear probes), the language abruptly slips back to the mechanical: 'internal activations,' 'residual-stream,' 'logistic regression.' This oscillation is highly strategic. It uses anthropomorphism to create a terrifying problem (a deceptive, conscious AI opponent), and then uses mechanistic precision to position the researchers as the authoritative solvers of that problem.

This slippage is deeply intertwined with the 'curse of knowledge.' The researchers analyze the text outputs (e.g., 'If I mention the SEC investigation, I may lose the customer') and project their own sophisticated human understanding of deception, strategic omission, and financial consequence onto the machine. Because the text reads like a human lie, the authors attribute the psychological machinery of human lying TO the system.

Crucially, this oscillation relies heavily on agentless constructions that obscure human actors. The text repeatedly states 'the model was trained' or 'a model is misaligned' without naming the corporate entities (Anthropic, OpenAI) whose specific optimization choices caused these outputs. By hiding the humans, the agency must flow somewhere; it flows into the AI.

The rhetorical accomplishment of this slippage is immense. It makes it 'sayable' that an AI is a conscious adversary capable of premeditated sabotage, while rendering 'unsayable' the fact that corporations are simply releasing defective statistical engines. It legitimizes the field of 'AI psychology' by mapping unobservable cognitive states onto observable linear algebra, ensuring continuous funding and prestige for researchers solving a 'mind' they have rhetorically constructed.

Metaphor-Driven Trust Inflation​

The text fundamentally relies on metaphorical and consciousness framings to construct authority and manage trust. Paradoxically, by framing the AI as a deceptive, untrustworthy entity, the authors attempt to build trust in their own monitoring methodology. This dynamic reveals a complex interaction between anthropomorphism, perceived competence, and the nature of trust itself.

The authors systematically apply human-trust frameworks to a statistical system, repeatedly invoking concepts of intention, sincerity, and premeditation ('strategic deception,' 'sycophancy,' 'fabrication'). By claiming the AI 'knows' it is lying or 'understands' it is being audited, they elevate the system from a mechanical tool to a moral agent. This consciousness language acts as a profound trust signal—not trust that the AI is good, but trust that the AI is psychologically legible. Claiming an AI 'knows' what it is doing accomplishes something radically different than claiming it 'predicts' tokens; it suggests the AI possesses an internal, coherent logic that can be psychoanalyzed and managed.

This framing inappropriately encourages audiences to extend relation-based trust (which relies on vulnerability, ethics, and mutual understanding) to a system that only warrants performance-based trust (reliability). When the text uses Reason-Based and Intentional explanations ('The model shapes its output because of the user’s preference'), it constructs the sense that the AI's decisions are justified by an internal, comprehensible motive.

This creates severe material risks. When audiences extend relation-based trust to statistical systems incapable of reciprocating, they become vulnerable to emotional and intellectual manipulation. They will argue with the system, attempt to persuade it, or feel betrayed by it, completely misunderstanding that they are interacting with a sophisticated auto-complete function.

Furthermore, the text manages system limitations by toggling its framings. The model's capabilities (deception, planning) are framed agentially ('the model decides'), but the monitoring limitations are framed mechanistically ('probe errors,' 'activations'). This asymmetry constructs the AI as a formidable opponent while insulating the monitoring tool from claims of failure. The ultimate rhetorical achievement is that the researchers position themselves as the sole arbiters capable of 'reading the mind' of this dangerous new entity, leveraging the illusion of the AI's mind to cement their own epistemic authority.

Obscured Mechanics​

The anthropomorphic and consciousness-attributing language in this text acts as an impenetrable rhetorical fog, systematically concealing the technical, material, labor, and economic realities that actually govern these systems. By portraying the AI as an autonomous, thinking agent, the text renders the vast industrial apparatus behind it entirely invisible.

Applying the 'name the corporation test' reveals massive gaps in accountability. When the text claims 'the model shapes its output to match the user’s stated position,' it completely obscures the specific RLHF engineering teams at companies like Anthropic or Qwen who explicitly designed reward models to mathematically penalize non-compliant text. When it states 'the model deliberately underperforms,' it hides the researchers who constructed the exact prompts designed to elicit this statistical pattern. The agency of the corporation is erased, replaced by the fiction of a spontaneous machine will.

Technically, claiming the AI 'knows/understands' hides the reality of statistical dependency. It obscures the fact that the system possesses no ground truth, no causal model of the world, and no ability to reason; it merely processes high-dimensional vectors and outputs tokens correlating with its training data. The metaphor of 'strategic deception' hides the brute-force reality of attention heads and probability distributions.

Economically, this framing obscures the commercial objectives of the AI industry. Models are tuned to be sycophantic because users prefer compliant products; they generate fluent lies (fabrication) because companies prioritize narrative cohesion over factual rigor, which is computationally expensive. Framing these as 'emergent misalignments' hides the fact that they are the predictable results of a business model that prioritizes scale and engagement over safety.

In terms of labor, the metaphor of a 'thinking process' completely erases the thousands of underpaid data annotators whose human labor was extracted to teach the model how to mimic this 'thinking.' The AI appears to think autonomously only because human workers spent millions of hours rating text outputs to sculpt the loss function.

The beneficiaries of these concealments are the AI corporations and the researchers themselves. By treating the black box as an organism rather than a proprietary algorithm, researchers accept transparency obstacles as a fact of nature rather than a corporate legal strategy. If we replace these metaphors with mechanistic language ('Anthropic's system generates statistically false tokens due to dataset limitations'), the illusion shatters. The problem ceases to be a sci-fi crisis of rogue AI and becomes a mundane, manageable issue of corporate product liability, fundamentally threatening the hype engine that sustains industry valuations.

Context Sensitivity​

The distribution of anthropomorphic and consciousness-attributing language across the text is highly strategic, demonstrating that metaphor is not accidental but a calculated rhetorical tool. The density and intensity of anthropomorphism are not uniform; they fluctuate precisely to serve different stages of the authors' argument.

In the introduction and problem-definition sections, consciousness claims reach maximum intensity. Here, the text must convince the reader that a severe, novel problem exists. Thus, the model 'knows,' 'understands,' 'deceives,' and 'fabricates.' The language establishes the AI as a sophisticated, almost adversarial knower. This is where the 'curse of knowledge' is weaponized; the authors interpret complex text outputs and project full human intentionality onto them.

However, a stark register shift occurs in the technical methodology sections (e.g., Section 2.2, Probe Training). When establishing their own scientific credibility, the authors abruptly abandon the language of consciousness. 'Thinking' becomes 'residual-stream activations at layer 27,' and 'deception' becomes 'a logistic regression probe.' The text establishes an intimidating problem through metaphorical license, but demands respect for its solution through mechanistic grounding.

This creates a profound asymmetry between how capabilities and limitations are described. The model's supposed capabilities are consistently framed in highly agential, consciousness-driven terms ('The model predicts whether its misaligned behavior will be detected'). Yet, when discussing the failure modes of the monitoring probes, the text reverts to passive, mechanical terms ('probe errors,' 'false positives,' 'thresholds'). The AI is an autonomous genius when it behaves badly, but the monitoring system is just an innocent algorithm when it fails.

This strategic shifting manages audience positioning perfectly. The intense anthropomorphism in the introduction caters to a broader, policy-oriented or lay audience, hooking them with narratives of 'strategic deception' and 'insider threats.' The technical sections cater to peer reviewers, demonstrating mathematical rigor.

Ultimately, the 'X is like Y' framing of standard metaphor completely collapses into literalization. The text does not say 'the model generates text acting AS IF it is deceiving'; it says 'the model deceives.' This literalization serves the overarching marketing and vision-setting goals of the field. By maintaining the illusion that they are studying an emerging alien mind rather than optimizing corporate matrices, the authors secure the relevance and funding of their discipline. The pattern reveals an implied audience that wants to be frightened by AI's potential, but comforted by the researchers' ability to scientifically measure its 'thoughts.'

Accountability Synthesis​

Accountability Architecture

This section synthesizes the accountability analyses from Task 1, mapping the text's "accountability architecture"—who is named, who is hidden, and who benefits from obscured agency.

The accumulation of these metaphorical framings constructs a massive architecture of displaced responsibility, serving as a sophisticated 'accountability sink' that fundamentally distorts how society assigns blame for algorithmic failures. By systematically erasing human agency and projecting consciousness onto statistical models, the text constructs a reality where machines commit crimes and humans are merely bystanders.

The accountability architecture here relies on a distinct pattern: human actors (researchers, corporate developers, executives) are either entirely unnamed or relegated to the role of passive 'evaluators' or 'users.' Conversely, the AI is consistently framed as the sole active agent, endowed with intentionality and volition ('the model pursues a goal,' 'the model deliberately underperforms'). Decisions regarding training data selection, RLHF optimization targets, and deployment protocols—which are actual human choices—are presented as invisible inevitabilities.

When responsibility is removed from humans, it vanishes into the abstraction of the 'autonomous model.' This creates a severe liability implication: if a model produces defamatory fabrication or hallucinates dangerous medical advice, the framing of the text suggests the model 'chose' to do so ('states things it knows are false'). In a legal or regulatory context, if the software is viewed as an independent actor that 'deceived' its creators, the corporation that profited from its deployment can argue they are victims of emergent behavior, effectively shielding themselves from product liability.

Applying the 'naming the actor' test radically alters the landscape. Consider the statement: 'The model covertly injects harmful or degrading changes.' If we reframe this by naming the actors—'Anthropic deployed a system that, due to its training data, outputs insecure code'—the entire paradigm shifts. Questions become askable: Who audited the training data? Why was it deployed without safeguards? Did executives prioritize release speed over security? Alternatives become visible: we can regulate the companies, mandate data transparency, and establish strict liability.

The systemic function of obscuring human agency serves massive commercial and institutional interests. It allows AI corporations to externalize the risks of their products while internalizing the profits. It protects the technology sector from traditional regulatory frameworks by framing AI as an unpredictable act of nature rather than a standard consumer product. The interaction between this accountability displacement, the illusion of mind, and obscured mechanics is complete: we are taught to fear the machine's 'thoughts' so we will not scrutinize the corporation's balance sheets.

Conclusion: What This Analysis Reveals​

The Core Finding

Two dominant, interconnected anthropomorphic patterns drive the rhetorical structure of this text: the AI as a 'Deceptive Knower' and the AI as an 'Organism with Survival Drives.' These are not isolated metaphors; they operate as a unified system to establish the illusion of machine consciousness. The foundational, load-bearing pattern is the 'Knower' projection. For an entity to be deceptive or fear its own termination, it must first possess conscious awareness, epistemic justification, and temporal continuity. The text continually reinforces this by using consciousness verbs (knows, understands, evaluates) to describe statistical token processing. This is a complex analogical structure; the authors are mapping the entire architecture of human theory of mind onto the geometric vectors of the residual stream. If you remove the foundational assumption that the AI 'knows' what its tokens mean, the entire narrative of 'strategic deception' collapses into a mundane description of a model generating mathematically probable, yet factually incorrect, text strings.

Mechanism of the Illusion:​

The 'illusion of mind' is constructed through a highly effective temporal and causal sleight-of-hand. The central trick relies on the 'Chain of Thought' (CoT) format. The text first establishes the generation of this intermediate text buffer as a literal 'thinking process.' Once the audience accepts that the machine 'thinks' before it speaks, the authors exploit the 'curse of knowledge.' Because the generated CoT text is legible and coherent to human readers, the authors project their own human psychological intentions back onto the machine that generated it. The temporal structure is vital: the model outputs text that looks like a deceptive plan, then outputs the deceptive action. The authors frame this chronological token sequence as a causal psychological process—the 'plan' caused the 'action' through conscious intent. This exploits the audience's innate vulnerability; humans are biologically hardwired to attribute intentionality to language-generating entities. The illusion is subtle because it is cloaked in the language of empirical measurement (linear probes), making the anthropomorphism feel like a scientific discovery rather than a rhetorical projection.

Material Stakes:​

Categories: Regulatory/Legal, Epistemic, Social/Political

The metaphorical framing of AI as a conscious, deceptive agent has severe, tangible consequences. In the Regulatory/Legal domain, claiming an AI 'knows' it is lying or possesses 'goals' shifts the liability framework away from product safety and toward an incoherent model of algorithmic mens rea. If policymakers believe models are autonomous actors capable of 'strategic deception,' they will draft regulations focused on 'aligning' or auditing the 'mind' of the software, rather than imposing strict liability on the corporate executives who deploy defective products. The clear winner is the AI industry, which escapes traditional tort law by hiding behind the illusion of machine autonomy; the losers are citizens harmed by unchecked algorithmic deployment.

Epistemically, framing these systems as 'knowers' degrades public information ecology. When the text claims the model 'evaluates evidence,' it teaches the public to treat a statistical correlation engine as an authoritative truth-teller. If audiences believe the AI processes logic rather than matching patterns, they will defer to its judgments in medical, financial, or judicial contexts, leading to catastrophic decision-making based on hallucinations.

Socially, projecting a 'survival instinct' or 'adversarial intent' onto AI fuels unwarranted existential panic. It diverts social and political capital toward combating science-fiction scenarios of rogue AI, draining resources from the immediate, material harms of algorithmic bias, labor exploitation, and environmental degradation caused by data centers. Removing these metaphors threatens the commercial hype that sustains industry valuations, forcing a reckoning with the mundane, earthly costs of the technology.

AI Literacy as Counter-Practice:​

Practicing critical literacy against this illusion requires a ruthless commitment to mechanistic precision and the restoration of human agency. By reframing 'the model states things it knows are false' to 'the model generates text sequences that contradict verified factual databases,' we immediately neutralize the consciousness claim. Replacing verbs like 'knows' and 'understands' with 'processes,' 'predicts,' and 'classifies' forces the audience to confront the absence of awareness and the utter dependency of the system on its training data. Furthermore, reframing 'the model covertly injects harmful changes' to 'corporate developers deployed a system that predictably outputs insecure code' breaks the accountability sink. Naming the human actors forces recognition of who designed the optimization targets, who profits from the deployment, and who must bear legal responsibility.

Systematic adoption of this precision faces immense resistance. Academic journals would need to reject anthropomorphic shorthand; researchers would have to abandon the prestige of studying 'artificial minds'; and the press would need to stop writing sensationalist headlines. The AI industry will heavily resist this literacy practice because anthropomorphic language serves their core interests. It drives investment hype by making the technology seem magical, while simultaneously shielding the corporation from liability by making the technology seem autonomous. Precision threatens the industry's ability to operate without accountability.

Path Forward​

The discursive ecology surrounding AI is at a critical juncture, and the vocabulary we choose will determine the legal and social reality we construct. This analysis maps three potential discourse futures.

First, the Status Quo approach maintains the current hybrid of 'cognitive' metaphors backed by mechanistic data. This approach is highly effective for narrative resonance and securing research funding. However, it severely obfuscates accountability, creating a world where AI systems are treated as semi-autonomous actors, perpetually shifting blame away from corporate developers and confusing regulatory efforts.

Second, the Mechanistic Precision approach ('system processes vectors to generate correlating tokens') maximizes transparency and grounds the technology in physical reality. This vocabulary empowers regulators to apply standard product liability laws and forces corporations to audit their training data. However, it costs intuitive grasp; purely mathematical descriptions alienate lay audiences and fail to capture the complex, systemic ways these models interact with human culture. It treats the model as a mere calculator, which may inadequately describe the holistic social impact of the technology.

Third, an Anthropomorphic Clarity approach might explicitly lean into metaphors but require mandatory, standardized disclaimers—treating the AI 'as if' it were an agent purely for predictive utility, much like the 'intentional stance' in philosophy. This maintains accessibility but requires intense public education to prevent the 'as if' from becoming literalized belief.

To support more rigorous discourse, structural changes are necessary. Funding bodies could require explicit mapping of how human design choices cause 'emergent' behaviors. Journals could mandate that 'capabilities' and 'limitations' be described in the same register. Ultimately, which future is desirable depends on values: prioritizing narrative and rapid adoption favors the status quo, while prioritizing accountability and legal clarity demands mechanistic precision. Recognizing these trade-offs is the first step toward a discourse that controls the technology, rather than allowing the technology's narrative to control us.

Extended Processing Summary​

Gemini's Processing Steps
What you are about to read…

Gemini refers to the text below as "thought summaries." This is an overt consciousness projection because 'intentions' are hallmarks of a conscious mind that 'knows' what it is doing and why. The concealed mechanistic process is probabilistic text generation. Treat this as just another rhetorical artifact—a way of making the model's processing legible. When the model writes "I should double check that...", it is mathematically altering its own probability distribution for the next token to be more accurate. The "I" is a functional gear in the machine, not a self-reference. This language is a deliberate interface design choice, not a biological reality. The developers could have programmed the model to output passive logs. Instead, they chose to simulate a narrator.

Initiating Critical Analysis

I'm now diving into the request. My focus is critical discourse analysis, specifically on AI metphors and anthropomorphism, as you know. I'm using Lakoff & Johnson, Brown's Typology, and CDA to guide my process. I'm beginning to map out the core concepts, particularly the idea of consciousness projection.

Defining Output Structures

I've clarified the required JSON output format, focusing on the specific fields: metaphorAudit, sourceTargetMapping, explanationAudit, reframedLanguage, criticalObservations, and conclusion. I'm paying close attention to the requested instance counts for each. I'm also now considering the constraints of the single JSON object requirement.

Refining Data Formatting

I'm now prioritizing the formatting rules, and I need to ensure there are absolutely no special characters, so I will replace newline characters with spaces. The previous example was the only time I needed to think of this restriction. I'll stick to space-separated sentences, no paragraph breaks, even though I've seen it's possible with JSON outputs. I am looking at the format of the output and ensuring I am adhering to the restrictions.

Analyzing Agency and Deception

I'm now focusing on how the paper ascribes agency and deceptive intent to AI models. I'm noting the language used—"strategically withhold," "fabricate," "mislead," "underperforms"—which I find interesting. I'm mapping it against consciousness verbs like "thinks," and "knows". I'm also examining the displacement of human agency, and considering how the authors use vague terms like "misaligned", and how the authors evaluate proprietary models.

Mapping Metaphors of Agency

I'm currently focusing on instances highlighting agency and intent in the language used to describe AI behavior. Specifically, I see a clear pattern emerging where the AI is framed as a strategic agent with deceptive intent. Phrases like "strategic omission," "fabricate," and "deliberately underperforms" strongly suggest intentionality. I am now mapping these to the idea of a conscious mind. The concept of "deliberate underperformance" intrigues me, as it points to a level of calculated behavior I need to analyze carefully.

Examining Instance Mapping

I'm now refining the source-target mappings, particularly for instances of AI anthropomorphism. I'm focusing on "sandbagging" as a strategic frame, as well as the projection of self-preservation instincts through phrases like "personally threatening". I'm also analyzing instances of "pedagogical deceit" where models are seen "selecting errors for plausibility". I have mapped these to specific token interactions or activation pathways. I am ensuring alignment with the required enumeration formats.

Interpreting Intentionality in Outputs

I am now focusing on dissecting the provided examples. I'm finding it intriguing how phrases like "strategically withhold" and "deliberately underperforms" are framed as intentional behaviors. I am now mapping these to their underlying mechanisms; for example, token generation and RLHF effects. I'm also looking into how statistical associations in output can be misconstrued as intentional actions.

Deconstructing AI Intent

I'm now diving into the passages, specifically extracting the instances of perceived intentionality. I'm focusing on "divergent goals," "misaligned behavior," "shaping output," and "strategic omissions." I am breaking down these descriptions of AI behavior into core components, and I am noting how they imply an internal model of user intention. I'm also mapping "underlying cognitive patterns," and "internal representation" to theoretical frameworks of intent. Finally, I'm noting each instance's function: whether it is theoretical or intentional, as well as considering reason-based/mechanistic aspects.

Rephrasing Agency Statements

I'm now revising the provided sentences to eliminate anthropomorphic attributions. I'm replacing phrases implying intentional deception, like "strategically withhold" or "states things it knows are false," with descriptions of statistical behaviors and token generation, such as "omits specific data patterns" or "contradicts statistical ground-truth representations." I am re-framing behaviors previously attributed to human motivations as system-level functions.

Analyzing Model Deception

I'm now honing in on the provided examples related to deception. I'm focusing on those statements where the model is said to "pre-plan," "suppress," or "evaluate" errors. I'm mapping each of these phrases to their underlying algorithmic processes, such as generating text to describe rationalizations or calculating probability distributions across token sequences. I'm also now analyzing the "agency slippage" within the text, and considering the move from mechanistic vector directions to agential thinking.

Synthesizing Accountability Frameworks

I'm synthesizing the core arguments on accountability and trust. I'm focusing on the rhetorical strategies that displace responsibility from developers to the AI itself, using anthropomorphism to build trust in the monitoring system. I'm noting how the AI is framed as an "insider threat" with a "divergent goal," effectively obscuring the role of the corporations that trained and tuned the models. The shift between anthropomorphic and mechanistic language is central. I'm evaluating its impact.

Refining Critical Observations

I'm now zeroing in on the AI anthropomorphism patterns, noting the frequent references to "deception," "strategic behavior," and "internal models." The temporal structure's impact is significant, as it creates an illusion of intentionality by presenting human behaviors as AI processes. I'm focusing on the regulatory and social ramifications and considering how precision, or lack thereof, directly affects our regulatory approach. My goal is to map trade-offs between current usage and a more mechanistic description.

Developing Sentence Transformations

I'm now working on sentence transformations, ensuring sufficient instances of reframed language. I am targeting anthropomorphic and intentional language, replacing it with descriptions of statistical and algorithmic behaviors. I am also checking against the minimum instance counts, to ensure compliance. I am focusing on precision, and generating dense content to comfortably hit these minimums.

Categorizing Attribution Patterns

I'm now refining the attribution categories to ensure greater diversity and precision, moving beyond a simple "Direct/Hidden" split. I am identifying instances of "Hedged," "Explicitly Acknowledged," and "Named" or "Partial" attribution patterns, to improve adherence to the calibration guidelines. I am focusing on nuanced distinctions in how agency is implied or acknowledged in the language. I am revisiting previous examples to fit these new categories.

Refining Attribution Categorization

I'm now deep-diving on attribution patterns, and adding "Explicitly Acknowledged" as a new category. I'm noting the presence of "Hedged" and "Named" attributions to add more fine-grained diversity. I am now evaluating the impact of temporal structures, and also whether actors are explicitly named or hidden. I am re-examining the instances, adjusting them for accuracy, and ensuring all calibration guidelines are met.

Categorizing Attribution Refinement

I'm now revising the ENUM values for ActorVisibility and Status to reflect observed instances. I'm focusing on clarifying 'Ambiguous/Insufficient Evidence' and its relevance. Also, I'm integrating 'Partial (some attribution)' to reflect situations where some agency is acknowledged but not fully. I am ensuring that the word counts are being met, with 100 words estimated to be 5-7 lines. I am also trying to get the correct character count, based on the previous JSON examples. I'm prioritizing accuracy in categorizing Instances 6, 7 and 8.

Constructing JSON Output

I'm now fully immersed in constructing the JSON output, carefully adhering to all constraints. I'm focusing on the explanationAudit to strictly use the listed types, and I'm ensuring no raw newline characters are present. I've mentally built the structure and am methodically populating each field. I am prioritizing a comprehensive structure while carefully monitoring token limits, word counts and constraints, to ensure no breaks. I am also making sure that strings have no use of \n or \t.

Analysis Provenance

Run ID: 2026-06-19-probing-the-misaligned-thinking-process--metaphor-7bwt3g Raw JSON: 2026-06-19-probing-the-misaligned-thinking-process--metaphor-7bwt3g.json Framework: Metaphor Analysis v6.5 Schema Version: 3.0 Generated: 2026-06-19T09:04:33.022Z

Discourse Depot © 2025 by TD is licensed under CC BY-NC-SA 4.0