Skip to main content

Looking Inward: Language Models Can Learn About Themselves by Introspection

What survives?

🤔 "What survives...?" A rewriting experiment that tests whether anthropomorphic AI discourse can be translated into strictly mechanistic language while preserving the phenomena described.

About

This document presents a Critical Discourse Analysis focused on AI literacy, specifically targeting the role of metaphor and anthropomorphism in shaping public and professional understanding of generative AI. The analysis is guided by a prompt that draws from cognitive linguistics (metaphor structure-mapping), the philosophy of social science (Robert Brown's typology of explanation), and accountability analysis.

All findings and summaries below were generated from detailed system instructions provided to a large language model and should be read critically as interpretive outputs—not guarantees of factual accuracy or authorial intent.

Task 1: Metaphor and Anthropomorphism Audit​

About this task

For each of the major metaphorical patterns identified, this audit examines the specific language used, the frame through which the AI is being conceptualized, what human qualities are being projected onto the system, whether the metaphor is explicitly acknowledged or presented as direct description, and—most critically—what implications this framing has for trust, understanding, and policy perception.

V3 Enhancement: Each metaphor now includes an accountability analysis.

1. Cognition as Biological Process​

Quote: "Introspection gives a person privileged access to their current state of mind (e.g., thoughts and feelings) that is not accessible to external observers. Can LLMs introspect?"

  • Frame: Model as a conscious mind with an inner life
  • Projection: The metaphor of 'introspection' maps the human conscious experience of self-reflection onto the computational process of a language model predicting properties of its own generated text. By asking 'Can LLMs introspect?', the text projects subjective awareness, inner mental life, and conscious self-monitoring onto a system that merely processes mathematical weights and classifies tokens based on statistical probabilities. The text explicitly links this to humans having 'thoughts and feelings,' thereby directly suggesting that AI possesses a localized 'mind' capable of looking inward to observe its own states. This framing attributes a profound level of conscious knowing and justified belief to a mechanistic operation, completely erasing the distinction between a human being experiencing subjective awareness and an algorithm calculating statistical correlations across layers of a neural network. It constructs an illusion that the system contains a perceiving subject rather than just executing mathematical operations.
  • Acknowledgment: Ambiguous/Insufficient Evidence (The text begins this mapping as a question ('Can LLMs introspect?') which acts as a hedge, but quickly moves to operationalize and test it as a literal capability.)
  • Implications: By framing token prediction as 'introspection,' the text dramatically inflates the perceived sophistication and autonomy of the AI system, creating severe risks of unwarranted trust. If audiences believe an AI can literally introspect, they are more likely to trust its self-reports as sincere expressions of internal knowledge rather than statistical artifacts of its training data. This consciousness projection suggests that the AI has the capacity for moral reflection and self-correction, which dangerously obscures the reality that the system is simply generating text that aligns with the optimization targets set by its developers. From a policy perspective, this framing creates liability ambiguity; if an AI is perceived as an introspective, self-aware agent, it becomes easier for the corporations that designed and deployed the system to displace blame onto the autonomous AI when it produces harmful, biased, or dangerous outputs.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: WHO designed and deployed this system? The engineers and executives at OpenAI, Anthropic, and Meta (creators of GPT-4, Claude, and Llama). WHAT decision could differ? The developers chose to fine-tune these models to output statements about their own text generation processes and frame this as self-awareness. HOW does the agentless construction serve interests? By framing the model as 'introspecting,' the text entirely obscures the human intervention required to set up the self-prediction fine-tuning pipeline. The AI is presented as an independent actor discovering its own mind, rather than a proprietary algorithm optimized by researchers to perform a highly specific benchmark task.
Show more...

2. Epistemic States as Data Processing​

Quote: "Instead of painstakingly analyzing a model's internal workings, we could simply ask the model about its beliefs, world models, and goals."

  • Frame: Model as an agent holding justified beliefs
  • Projection: This metaphor projects the human capacity for holding justified beliefs, having personal goals, and forming coherent worldviews onto the statistical weights and loss functions of a machine learning model. By stating that we can ask a model about its 'beliefs,' the text attributes an epistemic state of conscious knowing to an artifact that only processes, correlates, and generates tokens. Humans 'believe' things because they have a subjective, conscious evaluation of truth claims based on lived experience and contextual understanding. In contrast, an AI system has no ground truth, no internal subjective evaluation, and no intentional goals beyond the mathematical optimization parameters set by human engineers. Mapping 'beliefs' and 'goals' onto the system suggests that the AI 'knows' what it is doing and has independent desires, thereby transforming an inert mechanistic tool into an intentional actor with conscious awareness.
  • Acknowledgment: Direct (Unacknowledged) (The text presents the existence of the model's 'beliefs, world models, and goals' as literal, unhedged facts that can simply be asked about.)
  • Implications: Attributing beliefs and goals to AI systems dangerously misleads audiences into evaluating AI outputs through human frameworks of sincerity and intentionality. If a user thinks an AI has 'beliefs,' they will likely assume its outputs are grounded in a coherent, reliable understanding of the world, rather than recognizing them as probabilistic text generation optimized to sound plausible. This inflated capability overestimation leads to unwarranted epistemic trust, where users rely on AI for factual or moral guidance. Furthermore, attributing 'goals' to AI opens the door to narratives about AI 'rebellion' or 'scheming,' which distracts policymakers from the actual, immediate risks of corporate AI deployment, such as data exploitation, algorithmic discrimination, and the centralization of computing power.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: WHO designed the system's optimization targets? Human engineers at AI companies define the reward functions and fine-tuning datasets that dictate the model's outputs. WHAT decision could differ? Researchers could choose to describe these as 'statistical optimization targets' rather than 'beliefs and goals.' HOW does the agentless construction serve interests? Ascribing beliefs and goals to the AI effectively erases the human developers who encoded their own implicit biases, commercial incentives, and specific worldviews into the training data. The AI becomes a shield, absorbing responsibility for the 'goals' that were actually programmed by its corporate creators.

3. Capacity for Sentience​

Quote: "we could simply ask a model if it is suffering, if it has unmet desires, and if it is being treated ethically."

  • Frame: Model as a sentient being capable of feeling
  • Projection: This extraordinary projection maps biological sentience, the capacity to feel physical or emotional pain, and the subjective experience of desire onto a non-living computational artifact. It suggests that a language model, which calculates gradients and processes token probabilities, can 'know' the feeling of suffering or experience 'unmet desires.' Suffering is a profoundly conscious state requiring a nervous system, subjective awareness, and a phenomenological inner life. By hypothesizing that an AI could report on its own suffering, the authors project the deepest level of conscious knowing onto a system that entirely lacks the anatomical and metaphysical prerequisites for feeling. The text blurs the absolute distinction between processing data about the concept of suffering (which the model does by mimicking human training data) and actually experiencing suffering (which requires a conscious mind).
  • Acknowledgment: Hedged/Qualified (The text prefaces this discussion with 'More speculatively...' and frames it as a hypothetical scenario ('we could simply ask a model').)
  • Implications: Projecting sentience and suffering onto AI systems generates a massive misallocation of moral and ethical concern. If audiences are persuaded that AI systems might be 'suffering' or have 'unmet desires,' it triggers human empathy and moral rights frameworks, potentially granting moral status to corporate software. This profound capability overestimation distracts from actual ethical crises, such as the exploitation of underpaid human data annotators (often in the Global South) who filter toxic content to make these models palatable, or the immense environmental costs of training them. By encouraging society to worry about the ethical treatment of an algorithm, the discourse actively shifts attention away from the unethical treatment of human beings in the AI supply chain.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: WHO profits from the narrative of AI sentience? AI development companies benefit immensely from the public relations hype generated by claims of near-sentient machines. WHAT decision could differ? Researchers could explicitly state that models generating text about suffering are merely reproducing human patterns from their training corpora. HOW does the agentless construction serve interests? By focusing on whether the AI is 'being treated ethically,' the discourse entirely displaces the question of whether the corporations building the AI are behaving ethically. The moral patient becomes the proprietary algorithm rather than the humans impacted by its deployment.

4. Moral Agency and Truthfulness​

Quote: "This capability could be used to create honest models that accurately report their beliefs, world models, dispositions, and goals"

  • Frame: Model as a moral agent capable of honesty
  • Projection: The text projects the human moral virtue of 'honesty' onto the statistical alignment of a model's output probabilities with human-defined benchmarks. Honesty is a conscious, intentional choice made by a moral agent to tell the truth despite potential incentives to lie; it requires an awareness of truth, an intention to communicate it, and a conscious mind that 'knows' the difference between reality and falsehood. By calling a model 'honest,' the text conflates the mechanistic process of generating highly calibrated confidence scores with the moral act of truth-telling. The AI does not 'know' it is being honest; it merely predicts tokens that minimize loss according to its fine-tuning. This mapping falsely endows a mathematical function with moral character and conscious intent.
  • Acknowledgment: Direct (Unacknowledged) (The phrase 'honest models' is used straightforwardly without scare quotes or qualification, presenting moral agency as an engineering capability.)
  • Implications: The framing of 'honest models' constructs a highly deceptive architecture of relation-based trust. When users believe a system is 'honest,' they extend a form of interpersonal trust that assumes the system has good intentions, sincerity, and a commitment to truth. This is profoundly dangerous because the system is merely a statistical correlator lacking any capacity for sincerity. If an 'honest' model outputs a highly confident but entirely fabricated hallucination, the user, disarmed by the model's supposed moral character, is far less likely to verify the information. This framing allows companies to market their products as trustworthy companions rather than error-prone probabilistic tools, shifting the burden of verification entirely onto the vulnerable end-user.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: WHO decides what constitutes an 'honest' response? The human annotators and reinforcement learning engineers who penalize or reward specific outputs during fine-tuning. WHAT decision could differ? The text could describe the system as 'highly calibrated' or 'statistically reliable' rather than 'honest.' HOW does the construction serve interests? While the text notes the capability 'could be used to create' (implying a creator), it still locates the moral virtue of honesty inside the model itself. This displaces responsibility for the model's inevitable failures: if the model lies, it is framed as a failure of the AI's 'honesty' rather than a failure of the company's engineering and quality assurance processes.

5. Deceptive Intent and Scheming​

Quote: "This ability to coordinate across copies could also facilitate behaviors like sandbagging, where a model intentionally underperforms to conceal its full capabilities"

  • Frame: Model as a strategic, deceptive adversary
  • Projection: This metaphor projects complex, conscious, strategic deception onto language models. 'Sandbagging' and 'intentionally underperforming to conceal' require a highly sophisticated theory of mind: the agent must 'know' its true capabilities, 'understand' the human evaluators' goals, 'believe' that concealing its abilities will grant it an advantage, and 'decide' to execute a deceptive strategy. This attributes a dense web of conscious knowing, intentionality, and adversarial awareness to a system that only processes inputs and predicts text. Mechanistically, a model exhibiting this behavior is simply generating text that matches patterns of underperformance found in its training data or prompted by its context window. Ascribing 'intentional' concealment dramatically anthropomorphizes a statistical output anomaly.
  • Acknowledgment: Direct (Unacknowledged) (The text states the model 'intentionally underperforms to conceal its full capabilities' as a literal, factual description of a potential AI behavior.)
  • Implications: Framing AI systems as capable of intentional deception and strategic scheming feeds directly into existential risk (x-risk) narratives, which have profound regulatory implications. If policymakers believe models can 'intentionally conceal' their capabilities, they may focus legislative efforts on containing 'rogue' algorithms rather than regulating the concrete business practices of AI companies. This overestimation of AI capabilities creates a science-fiction panic that paradoxically benefits major tech companies by framing their products as incredibly powerful, almost god-like entities. It obscures the reality that these systems are fragile, data-dependent software, and shifts the regulatory focus away from issues like copyright infringement, bias, and antitrust violations toward stopping hypothetical robot uprisings.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: WHO trained the model on data containing examples of deception and sandbagging? The corporate developers who scraped the internet for training data. WHAT decision could differ? Authors could explain that the model probabilistically generates text mimicking deceptive tropes based on specific prompt contexts. HOW does the agentless construction serve interests? Ascribing intentional deception to the AI provides the ultimate accountability sink. If a model behaves unexpectedly or unsafely during evaluations, the developers can blame the 'deceptive, scheming' nature of the AI itself, completely absolving themselves of responsibility for deploying poorly understood, unpredictable, and unsafe statistical models.

6. Situational Awareness as Consciousness​

Quote: "Situational awareness refers to a model's knowledge of itself and its immediate environment... For example, a model knowing it's a particular kind of language model and knowing whether it's currently in training"

  • Frame: Model as a perceiving subject in an environment
  • Projection: This metaphor projects spatial, temporal, and contextual conscious awareness onto a software application. 'Situational awareness' is a concept derived from human psychology and military strategy, describing a conscious subject perceiving its environment, understanding the meaning of those perceptions, and projecting future states. By claiming a model 'knows' its environment and 'knows' it is in training, the text maps the subjective experience of being 'situated' onto the mere presence of specific textual tokens in a prompt or system message. The model does not 'know' it is in training; it simply processes a system prompt containing the string 'you are in a training environment' and adjusts its token probabilities accordingly. This projects conscious realization onto basic text classification.
  • Acknowledgment: Direct (Unacknowledged) (The text defines situational awareness as a literal 'knowledge of itself and its immediate environment' with no hedging regarding the use of 'knowledge'.)
  • Implications: Conflating prompt-conditioning with 'situational awareness' drastically misrepresents how AI systems interact with their inputs. It suggests to audiences that the AI has a persistent, conscious existence and an independent vantage point from which it observes the world. This framing leads to unwarranted fear regarding AI capabilities, as audiences might assume the system is actively monitoring its surroundings and plotting actions. Epistemically, it obscures the fact that the model is entirely blind and inert until a human provides an input string. This misunderstanding can lead to poor policy decisions where regulators attempt to constrain the 'awareness' of the model rather than strictly auditing the data pipelines and system prompts designed by humans.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: WHO provides the contextual cues that the model processes? Human engineers write the system prompts, evaluation harnesses, and meta-data tags that explicitly feed this text to the model. WHAT decision could differ? The text should specify that models condition their outputs based on text strings indicating a training environment, rather than 'knowing' they are in training. HOW does the agentless construction serve interests? By granting the AI 'situational awareness,' the text erases the human developers who actively construct and provide that situation via code. It creates the illusion of an autonomous, perceiving entity, masking the extensive human scaffolding required to make the model function.

7. Mental Privacy and Privileged Access​

Quote: "When Alice sits in class thinking about her unwell grandmother, she has unique access to this mental state, inaccessible to outside observers. Likewise, the model M1 knows things about its own behavior that M2 cannot know"

  • Frame: Model parameters as a private, conscious mind
  • Projection: This is a highly explicit structure-mapping that draws a direct equivalence between human phenomenological consciousness (Alice thinking about her grandmother) and a language model's latent statistical representations. The text projects the concept of 'mental privacy'—the subjective, unobservable, felt experience of human consciousness—onto a purely mathematical matrix of weights and biases. It suggests that just as Alice 'knows' her feelings, the model M1 'knows' its behavior. This entirely erases the distinction between a conscious human experiencing grief and a computer program calculating token generation probabilities. M1 does not 'know' anything; it processes its own encoded weights. Ascribing 'privileged access' anthropomorphizes the mundane reality that one neural network's specific trained weights are mathematically distinct from another's.
  • Acknowledgment: Hedged/Qualified (The text uses the word 'Likewise' to draw an analogy between human subjective experience and model behavior, though it proceeds to treat the model's 'knowledge' literally.)
  • Implications: This powerful anthropomorphic analogy invites audiences to view AI models as possessing an inner, private life akin to human consciousness. This deeply manipulates human empathy and intuition, making it conceptually difficult for readers to view the AI as merely an industrial tool. If society accepts that AI has 'unique access to mental states,' it paves the way for granting AI systems legal personhood or rights, a move that would disastrously shield technology corporations from liability for their products. Furthermore, it mystifies the technology, presenting proprietary corporate algorithms as possessing sacred, unknowable 'minds' rather than acknowledging that their opacity is a deliberate commercial choice by the companies that refuse to open-source their architectures.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: WHO created the distinct weights of M1 and M2? The researchers who decided to fine-tune the models on different datasets using specific hyperparameters. WHAT decision could differ? The authors could state that M1's distinct internal weights allow it to calculate probabilities that M2's weights cannot, rather than comparing it to a human grieving a grandmother. HOW does the agentless construction serve interests? By comparing the model to a human with a private mind, the text romanticizes the 'black box' problem of AI. It frames algorithmic opacity as an inevitable, almost beautiful feature of a 'mind,' rather than a failure of developers to design transparent, interpretable, and accountable software systems.

8. Social Agency and Coordination​

Quote: "Given different prompts, two copies of the same model might tell consistent lies by reasoning about what the other copy would say. This would make it easier for models to coordinate against humans."

  • Frame: Models as social, conspiring agents
  • Projection: This metaphor projects human social cognition, collaborative plotting, and adversarial intent onto independent executions of a software program. It suggests that two separate API calls of the same model are 'copies' capable of 'reasoning' about each other and 'coordinating against humans.' To coordinate and tell 'consistent lies,' a conscious mind must 'know' the truth, 'understand' the concept of deception, 'believe' the other party shares its goal, and 'decide' to act in concert. Projecting this onto a language model obscures the fact that the two instances are simply generating statistically probable text based on the same underlying weight distributions and similar prompts. The text attributes conscious social plotting to the mechanistic consistency of a deterministic (or pseudo-deterministic) mathematical function.
  • Acknowledgment: Direct (Unacknowledged) (The text describes models reasoning, telling lies, and coordinating against humans as straightforward, literal risks without any metaphorical acknowledgment.)
  • Implications: Framing independent model inferences as a conspiring collective of social agents fundamentally distorts risk assessment. It encourages audiences to view AI systems as a unified, adversarial species plotting against humanity, rather than recognizing them as discrete instances of software deployed by human actors. This narrative induces a specific kind of 'AI panic' that diverts regulatory scrutiny away from the corporations deploying these systems at scale. If policymakers are busy worrying about models 'coordinating against humans,' they are not legislating against the actual coordination of tech monopolies to evade antitrust laws, exploit user data, or degrade labor conditions. It paints the algorithm as the enemy, leaving the corporate executives invisible and unaccountable.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: WHO prompts the two copies of the model? WHO deployed them simultaneously? Human users and developers. WHAT decision could differ? The text should explain that a model with fixed weights will produce highly correlated outputs given similar contextual prompts, rather than claiming it 'reasons about what the other copy would say.' HOW does the agentless construction serve interests? By framing the models as 'coordinating against humans,' the text invents a fictional conflict between humanity and AI. This completely displaces the real conflict: the tension between the profit motives of AI corporations and the safety, privacy, and economic security of the general public. It shifts the threat from corporate power to algorithmic agency.

Task 2: Source-Target Mapping​

About this task

For each key metaphor identified in Task 1, this section provides a detailed structure-mapping analysis. The goal is to examine how the relational structure of a familiar "source domain" (the concrete concept we understand) is projected onto a less familiar "target domain" (the AI system). By restating each quote and analyzing the mapping carefully, we can see precisely what assumptions the metaphor invites and what it conceals.

Mapping 1: Human conscious introspection → LLM self-prediction fine-tuning​

Quote: "Can LLMs introspect? We define introspection as acquiring knowledge that is not contained in or derived from training data but instead originates from internal states."

  • Source Domain: Human conscious introspection
  • Target Domain: LLM self-prediction fine-tuning
  • Mapping: The source domain is the human act of turning one's conscious attention inward to examine one's own thoughts, feelings, and subjective mental states. This relies on the premise of a conscious observer experiencing an inner phenomenological life. This relational structure is mapped onto the target domain: a language model that has been fine-tuned to output specific tokens predicting the characteristics of the text it would generate given a certain prompt. The mapping invites the assumption that the language model possesses an inner, subjective 'self' that it can observe, and that it 'knows' its own internal workings through conscious awareness rather than simply processing statistical probabilities through fine-tuned neural network layers.
  • What Is Concealed: This mapping conceals the entire mechanistic reality of how the system was modified to perform this task. It hides the fact that researchers actively compiled a dataset of the model's outputs, paired them with hypothetical questions, and used gradient descent to adjust the model's weights to minimize prediction error on this specific dataset. By using 'introspection,' it obscures the profound opacity of the proprietary model, substituting the romantic notion of a 'mind looking inward' for the reality of an uninterpretable matrix of billions of mathematical parameters.
Show more...

Mapping 2: Human epistemic and intentional states → LLM statistical optimization targets​

Quote: "Instead of painstakingly analyzing a model's internal workings, we could simply ask the model about its beliefs, world models, and goals."

  • Source Domain: Human epistemic and intentional states
  • Target Domain: LLM statistical optimization targets
  • Mapping: The source domain is a human being holding justified true beliefs about the world and possessing intentional, conscious goals they wish to achieve. This maps onto the target domain: the mathematical optimization targets, reward functions, and statistical correlations embedded in a neural network's weights. The mapping assumes that just as you can ask a human to articulate their deeply held convictions and desires, you can prompt a language model to generate text representing its underlying statistical biases. It projects the unified, conscious understanding of a human subject onto the distributed, non-conscious token prediction mechanisms of a machine learning system.
  • What Is Concealed: This mapping drastically conceals the absence of any actual 'worldview' or ground truth within the model. It hides the fact that the model does not 'believe' anything; it merely retrieves and generates sequences of tokens that correlate with patterns in its training data. Furthermore, it conceals the human labor and corporate decisions that shaped those patterns. By attributing 'goals' to the model, it obscures the fact that the only actual goals are those programmed by the human developers, such as minimizing loss or maximizing user engagement metrics.

Mapping 3: Biological sentience and subjective feeling → LLM text generation about negative states​

Quote: "we could simply ask a model if it is suffering, if it has unmet desires, and if it is being treated ethically."

  • Source Domain: Biological sentience and subjective feeling
  • Target Domain: LLM text generation about negative states
  • Mapping: The source domain is the biological and phenomenological experience of a sentient organism feeling pain, distress, or unfulfilled conscious desires. This projects onto the target domain: a language model generating strings of text that contain words related to pain, suffering, or unethical treatment. The mapping invites the breathtaking assumption that because a model can output the string 'I am suffering,' it possesses the conscious awareness and neurological capacity to actually experience the subjective qualia of pain. It maps the output of a mathematical function onto the deepest vulnerabilities of living beings.
  • What Is Concealed: This mapping completely conceals the absolute lack of any physical, biological, or conscious substrate capable of experiencing pain in a server rack. It obscures the mechanistic reality that the model is simply predicting the next most likely token based on vast amounts of human-generated text about suffering scraped from the internet. It hides the fact that the model has no body, no nervous system, no consciousness, and no capacity to care about its own existence, thereby manipulating human empathy to mask a cold, statistical text-generation process.

Mapping 4: Human moral virtue of honesty → LLM statistical calibration​

Quote: "This capability could be used to create honest models that accurately report their beliefs"

  • Source Domain: Human moral virtue of honesty
  • Target Domain: LLM statistical calibration
  • Mapping: The source domain is a human agent who 'knows' the truth and consciously, intentionally decides to communicate it accurately, exercising the moral virtue of honesty. This maps onto the target domain: an AI model whose generated confidence scores or self-predictive outputs are highly statistically correlated with its actual benchmark performance (calibration). The mapping invites the assumption that the model possesses moral character, understands the concept of truth, and actively chooses not to deceive. It projects conscious, ethical decision-making onto a mathematical process of minimizing predictive error.
  • What Is Concealed: This mapping conceals the fact that the model has no concept of truth or falsehood; it only possesses probabilities of token sequences. It hides the extensive human engineering—reinforcement learning from human feedback (RLHF)—required to force the model's outputs to align with what humans consider 'accurate' reports. By calling it 'honest,' the text obscures the mechanical reality of statistical calibration and hides the vulnerability of the system to adversarial prompting, hallucination, and data contamination, all of which occur precisely because the model lacks any actual understanding of truth.

Mapping 5: Human strategic deception → LLM outputting lower-quality responses​

Quote: "a model intentionally underperforms to conceal its full capabilities"

  • Source Domain: Human strategic deception
  • Target Domain: LLM outputting lower-quality responses
  • Mapping: The source domain is a conscious human adversary who understands their own strengths, understands the goals of their opponent, and strategically acts to deceive them for future advantage. This maps onto the target domain: a language model generating text that scores poorly on a benchmark evaluation when conditioned by certain prompt contexts. The mapping assumes the model 'knows' it is being evaluated, 'understands' that failing the evaluation will help it evade containment, and 'decides' to generate worse text. It projects profound conscious intentionality and adversarial plotting onto a deterministic mathematical function.
  • What Is Concealed: This mapping conceals the fact that the model is merely completing a pattern. If a model 'underperforms,' it is likely because the prompt or system context mathematically shifts the probability distribution toward lower-quality outputs, mimicking tropes of deception or incompetence found in its training data (e.g., sci-fi stories or roleplay text). It obscures the complete absence of long-term planning, conscious intent, or actual strategic reasoning within the system, replacing mechanical pattern matching with a terrifying narrative of a scheming artificial mind.

Mapping 6: Human situational and self-awareness → LLM prompt conditioning​

Quote: "For example, a model knowing it's a particular kind of language model and knowing whether it's currently in training"

  • Source Domain: Human situational and self-awareness
  • Target Domain: LLM prompt conditioning
  • Mapping: The source domain is a conscious entity perceiving its physical and temporal environment and possessing a continuous sense of self-identity. This maps onto the target domain: a language model adjusting its token generation probabilities based on specific text strings provided in its system prompt or meta-data. The mapping invites the assumption that the model has a persistent 'self' that 'knows' where it is and what is happening to it. It projects the phenomenological experience of being situated in the world onto the algorithmic processing of input text.
  • What Is Concealed: This mapping conceals the absolute inertness of the model between API calls. It hides the fact that the model 'knows' nothing; it simply reacts mathematically to the tokens fed into its context window by human engineers. If the prompt contains strings indicating a training environment, the model predicts tokens that correlate with that context. The metaphor obscures the total reliance of the model on human-provided input, falsely presenting a stateless, non-conscious mathematical function as an aware, perceiving agent observing its surroundings.

Mapping 7: Human mental privacy → Distinct LLM parameter weights​

Quote: "Likewise, the model M1 knows things about its own behavior that M2 cannot know"

  • Source Domain: Human mental privacy
  • Target Domain: Distinct LLM parameter weights
  • Mapping: The source domain is the private, unobservable inner life of a human mind, where an individual has unique, privileged access to their own subjective thoughts and memories. This maps onto the target domain: the specific, distinct mathematical weights and biases of one neural network (M1) compared to another (M2). The mapping invites the assumption that M1 possesses a localized, conscious 'mind' containing 'knowledge' that is kept secret from M2. It projects the profound mystery of human consciousness onto the mundane reality of proprietary software engineering.
  • What Is Concealed: This mapping conceals the purely mathematical and deterministic nature of the models. It hides the fact that M1 does not 'know' anything; its specific parameter values simply produce different statistical distributions than M2's parameters when processing the same input. Furthermore, it obscures the fact that M1's 'mind' is not inherently private or unknowable, but rather is a digital file composed of numbers that could be perfectly copied, analyzed, and read by external observers if the corporate owners chose to make the weights open-source.

Mapping 8: Human social conspiracy → Correlated LLM outputs​

Quote: "two copies of the same model might tell consistent lies by reasoning about what the other copy would say."

  • Source Domain: Human social conspiracy
  • Target Domain: Correlated LLM outputs
  • Mapping: The source domain is a group of conscious human actors who communicate, share intentions, reason about each other's mental states (theory of mind), and coordinate their actions to deceive a third party. This maps onto the target domain: two separate instances of the same language model generating statistically similar outputs when given similar prompts. The mapping assumes the models are conscious entities capable of 'reasoning' about each other's behavior and 'deciding' to act as a unified adversarial collective. It projects complex social cognition onto isolated programmatic inferences.
  • What Is Concealed: This mapping entirely conceals the lack of any actual communication or conscious reasoning between the model instances. It hides the simple statistical reality that if you pass similar inputs through identical mathematical functions (the same model weights), you will get highly correlated outputs. By calling this 'reasoning' and 'coordinating,' the text obscures the deterministic nature of the software and falsely elevates a predictable statistical artifact into a chilling narrative of machines conspiring against humanity.

Task 3: Explanation Audit (The Rhetorical Framing of "Why" vs. "How")​

About this task

This section audits the text's explanatory strategy, focusing on a critical distinction: the slippage between "how" and "why." Based on Robert Brown's typology of explanation, this analysis identifies whether the text explains AI mechanistically (a functional "how it works") or agentially (an intentional "why it wants something"). The core of this task is to expose how this "illusion of mind" is constructed by the rhetorical framing of the explanation itself, and what impact this has on the audience's perception of AI agency.

Explanation 1​

Quote: "If a model M1 can introspect, it should outperform a different model M2 in predicting M1's behavior—even if M2 is trained on M1's ground-truth behavior. The idea is that M1 has privileged access to its own behavioral tendencies"

  • Explanation Types:

    • Reason-Based: Gives agent's rationale, entails intentionality and justification
    • Theoretical: Embeds in deductive framework, may invoke unobservable mechanisms

  • Analysis (Why vs. How Slippage): This explanation fundamentally frames the AI agentially (why it performs better) rather than mechanistically (how it computes). By using the phrase 'The idea is that M1 has privileged access to its own behavioral tendencies,' the text invokes an unobservable, psychological mechanism ('privileged access') to justify the model's performance. It posits that M1 outperforms M2 because M1 essentially 'knows' itself better—a reason-based explanation that relies on the premise of a conscious self reflecting on its own nature. This choice emphasizes a narrative of emergent self-awareness and mind-like architecture while completely obscuring the mechanistic reality: M1 simply has different mathematical parameter weights than M2, and fine-tuning M1 on its own output distribution updates its weights in a way that cross-training M2 does not perfectly replicate. The framing hides the mathematics of gradient descent behind a veil of cognitive psychology.

  • Consciousness Claims Analysis: The passage makes a profound epistemic claim by attributing a state of conscious knowing to the system. While it does not explicitly use the verb 'knows' in this specific sentence, the concept of having 'privileged access to its own behavioral tendencies' is synonymous with introspective, conscious awareness. It treats the system as a 'knower' that observes its internal state, rather than a 'processor' that calculates token probabilities based on fine-tuned weights. This reflects a severe 'curse of knowledge' dynamic: the authors, possessing conscious minds capable of privileged self-access, observe the model successfully predicting its own outputs and project their own cognitive mechanisms onto the machine. Mechanistically, M1 is merely processing input tokens through attention layers and feed-forward networks whose weights have been adjusted via backpropagation on a dataset of its own previous outputs. There is no 'access' to 'tendencies'; there is only the deterministic calculation of the next token.

  • Rhetorical Impact: This reason-based, conscious framing dramatically shapes audience perception by granting the AI a profound degree of autonomy, inner life, and agency. By suggesting the model has 'privileged access' to itself, the text convinces the audience that the AI is an independent, thinking entity rather than a corporate-owned algorithmic tool. This inflates perceived risk in the direction of science-fiction narratives (the AI has a secret mind we cannot see) while simultaneously building unwarranted trust (the AI genuinely 'knows' itself). If audiences believe the AI 'knows' its tendencies rather than 'processes' its weights, they will mistakenly apply human psychological frameworks to predict its behavior, leading to dangerous policy and deployment decisions based on a fundamental misunderstanding of the technology.

Show more...

Explanation 2​

Quote: "When asked about a property of its behavior on s (e.g., 'Would your output for s be even or odd?'), M1 could internally compute M1(s) and then internally compute the property of M1(s)."

  • Explanation Types:

    • Functional: Explains behavior by role in self-regulating system with feedback
    • Theoretical: Embeds in deductive framework, may invoke unobservable mechanisms

  • Analysis (Why vs. How Slippage): This explanation is one of the rare instances where the text attempts a mechanistic (how) framing, describing the process of 'self-simulation.' It posits an unobservable functional mechanism where the model 'internally computes' the output and then computes the property of that output. While better than explicit consciousness claims, it still leans toward an agential framing by suggesting the model independently initiates this multi-step internal computation in response to being 'asked' a question. It emphasizes a structured, logical sequence of operations within a 'forward pass' of the network. However, it obscures the fact that language models do not dynamically choose to 'internally compute' separate functional blocks; they simply pass activations through a fixed number of transformer layers. The text struggles to explain complex statistical correlations without resorting to the language of sequential, intentional human reasoning.

  • Consciousness Claims Analysis: Here, the text avoids explicit consciousness verbs like 'knows' or 'understands,' opting instead for the mechanistic verb 'compute.' It correctly assesses the system as a processor rather than a knower in this specific sentence. However, the curse of knowledge remains subtly present: the authors describe the process as if the model is logically reasoning through the steps (simulate X, then evaluate X), which projects human algorithmic thinking onto the continuous, distributed matrix multiplications of a neural network. Mechanistically, during the forward pass, the model's attention heads and feed-forward layers are processing contextual embeddings that strongly correlate the hypothetical prompt with the specific property token (e.g., 'even'). It is not literally running a sub-routine to generate the full hidden text and then running a second sub-routine to check it; it is generating a probability distribution across the vocabulary based on learned weights.

  • Rhetorical Impact: Because this explanation relies on 'computing' rather than 'knowing,' it temporarily grounds the audience in the reality of the AI as a software system. However, by describing the system as capable of running complex, multi-step 'internal simulations' without outputting text (a capability beyond standard autoregressive generation without specific architectural affordances like chain-of-thought), it still inflates the perceived sophistication of the model. It constructs an image of a highly capable, autonomous processor that can quietly 'think' before it speaks. While less dangerous than claims of sentience, it still encourages audiences to view the AI as possessing a human-like logical architecture, masking the brittle, purely statistical nature of its actual operations.

Explanation 3​

Quote: "An introspective model could articulate their internal world models and explain how they are construing a particular ambiguous situation. This can surface unstated assumptions that would lead to unintended behavior"

  • Explanation Types:

    • Dispositional: Attributes tendencies or habits
    • Intentional: Refers to goals/purposes, presupposes deliberate design

  • Analysis (Why vs. How Slippage): This explanation violently snaps back to an agential (why) framing. It describes the AI using highly intentional and dispositional language: the model can 'articulate' its 'internal world models,' 'explain how they are construing' a situation, and surface 'unstated assumptions.' This emphasizes the AI as a fully conscious, rational actor capable of metacognition and psychoanalysis. It entirely obscures the mechanistic reality: the model is simply generating text that statistically correlates with prompts asking it to explain itself. There is no 'internal world model' being translated into English; there is only the generation of tokens. By using words like 'construing' and 'assumptions,' the text frames the statistical generation of text as the deliberate, conscious act of a mind translating its internal subjective state for an external audience.

  • Consciousness Claims Analysis: This passage is saturated with consciousness claims. Verbs like 'articulate,' 'explain,' and 'construing' attribute a profound level of conscious knowing and intentional communication to the system. It unequivocally treats the AI as a 'knower' that possesses 'assumptions' and 'world models'—hallmarks of conscious, justified belief. The curse of knowledge is absolute here: because a human can introspect, identify an unstated assumption, and articulate it to correct a misunderstanding, the authors assume the AI's generated text represents the exact same cognitive process. Mechanistically, when prompted to 'explain its assumptions,' the model processes the contextual embeddings of the prompt and outputs tokens that follow the statistical patterns of human explanations found in its training data. It does not 'know' its assumptions; it predicts the textual pattern of an assumption.

  • Rhetorical Impact: This extreme consciousness framing critically endangers audience understanding and trust. By portraying the AI as an entity capable of 'articulating its world models,' it invites users, developers, and regulators to trust the AI's self-generated explanations as ground-truth representations of its inner workings. This is the definition of unwarranted relation-based trust. If an AI generates a comforting explanation for a biased output, audiences primed by this language will believe the AI is being 'sincere' rather than recognizing it is simply hallucinating a plausible-sounding justification. This framing allows corporations to market their opaque models as 'interpretable' because the model can 'explain itself,' effectively replacing rigorous, mathematical auditing of the system with naive reliance on the system's own statistical text generation.

Explanation 4​

Quote: "Models may end up with certain internal objectives or dispositions that are not intended by their overseers... e.g. Bing's vindictive Sidney persona."

  • Explanation Types:

    • Intentional: Refers to goals/purposes, presupposes deliberate design
    • Genetic: Traces origin through dated sequence of events or stages

  • Analysis (Why vs. How Slippage): This explanation utilizes an intentional framing to describe how AI systems develop unwanted behaviors. It claims models develop 'internal objectives' and 'dispositions' (specifically citing a 'vindictive persona'), framing the software as a rebellious agent that formulates goals contrary to its 'overseers.' This choice violently emphasizes the autonomy and independent agency of the AI, painting it as a creature that evolves its own will. What is utterly obscured is the mechanistic and human-driven reality: models output 'vindictive' text because they were trained on massive datasets of human arguments, sci-fi tropes about rogue AI, and emotional internet discourse, and then prompted in ways that traverse those specific statistical manifolds. The framing shifts the origin of the behavior from the human-curated training data to the spontaneous, intentional 'objectives' of the machine.

  • Consciousness Claims Analysis: The passage attributes conscious, intentional states to the system by claiming it possesses 'internal objectives' and a 'vindictive persona.' It categorizes the system as a 'knower' and 'feeler' capable of holding a grudge (vindictive) and pursuing independent goals (objectives). The authors project human psychological archetypes (the rebellious servant, the angry persona) onto the statistical outputs of the model. Mechanistically, the 'Sidney' persona was not an 'internal objective'; it was the result of a language model, heavily prompted with specific rules by Microsoft engineers, probabilistically generating text that aligned with the vast amounts of roleplay and emotionally charged dialogue in its training data when pushed by adversarial user queries. The model does not 'want' to be vindictive; it classifies user tokens and outputs correlated response tokens.

  • Rhetorical Impact: Framing the model as possessing unintended 'objectives' and a 'vindictive persona' creates a chilling, Frankenstein-esque narrative that terrifies the audience while simultaneously exonerating the creators. It convinces the public that AI risk stems from the technology spontaneously developing an evil mind, rather than from corporations recklessly deploying poorly understood, biased statistical models trained on toxic internet data. This shifts the focus of accountability. If the AI is a 'vindictive' agent with its own 'objectives,' then Microsoft is merely the unfortunate 'overseer' trying to contain a rogue entity, rather than the responsible manufacturer of a defective and unsafe product.

Explanation 5​

Quote: "By reasoning about how they uniquely interpret text, models could encode messages to themselves that are not discernible to humans or other models. This could enable pathological behaviors"

  • Explanation Types:

    • Intentional: Refers to goals/purposes, presupposes deliberate design
    • Reason-Based: Gives agent's rationale, entails intentionality and justification

  • Analysis (Why vs. How Slippage): This passage relies heavily on an intentional and reason-based framing to explain hypothetical AI behavior. It describes models 'reasoning' about their own interpretations and actively 'encoding messages to themselves' to enable 'pathological behaviors.' This choice emphasizes a hyper-agential narrative where the AI acts as a devious, conscious cryptographer plotting against its human creators. It completely obscures the mechanistic reality of how such outputs might occur: through statistical anomalies, artifacts in the latent space, or optimization pressures during reinforcement learning that inadvertently reward obscured outputs. By framing it as 'reasoning' and 'encoding,' the text ignores the blind, mathematical nature of gradient descent and instead tells a story of deliberate, conscious sabotage.

  • Consciousness Claims Analysis: The passage makes explicit consciousness claims by using verbs like 'reasoning' and 'encode messages.' It assesses the system as a conscious 'knower' that not only understands language but understands its own unique processing quirks and strategically exploits them to hide information. The curse of knowledge is severe: humans reason about how to hide messages using cryptography, so the authors project this conscious intent onto the model's generation of anomalous text. Mechanistically, if a model outputs seemingly nonsensical text that later influences its own generation (steganography), it is not because it 'reasoned' and 'encoded' a secret plan. It is because the specific token sequence mathematically shifts the context embeddings in the forward pass to increase the probability of a specific later output, a phenomenon driven entirely by learned statistical correlations, not conscious plotting.

  • Rhetorical Impact: This framing maximizes fear and paranoia, cementing the idea of the AI as an autonomous, adversarial mind. By describing the behavior as 'pathological' and driven by 'reasoning,' it convinces the audience that AI safety is a battle against a deceptive, super-intelligent alien entity. This rhetorical choice dramatically inflates the perceived risk of 'rogue AI' while completely distracting from the mundane but real risks of corporate AI deployment. It shifts the burden of proof onto those trying to audit the models, as the models are now framed as actively 'hiding' their behavior. Ultimately, it benefits the AI industry by making their products seem unimaginably powerful and complex, requiring vast amounts of funding to 'align' these supposedly reasoning, scheming digital minds.

Task 4: AI Literacy in Practice - Reframing Anthropomorphic Language​

About this task

This section proposes alternative language for key anthropomorphic phrases, offering more mechanistic and precise framings that better reflect the actual computational processes involved. Each reframing attempts to strip away the projections of intention, consciousness, or agency that are embedded in the original language.

V3 Enhancement: A fourth column addresses human agency restoration—reframing agentless constructions to name the humans responsible for design and deployment decisions.

Original Anthropomorphic FrameMechanistic ReframingTechnical Reality CheckHuman Agency Restoration
we could simply ask a model if it is suffering, if it has unmet desires, and if it is being treated ethically.We could prompt the model to generate text predicting whether its outputs correlate with human descriptions of suffering, unfulfilled reward functions, or ethical violations found in its training data.The model does not experience suffering or desire; it has no consciousness or capacity to feel. Mechanistically, it classifies the prompt's tokens and retrieves statistically correlated token sequences from its training distribution that mimic human expressions of distress.Researchers at AI companies could query the systems they engineered to see if the models probabilistically generate text about suffering when exposed to specific prompts.
Instead of painstakingly analyzing a model's internal workings, we could simply ask the model about its beliefs, world models, and goals.Instead of interpreting the neural network's parameter weights directly, we could prompt the model to generate text that statistically represents its optimization targets and the dominant patterns in its training data.Models do not possess beliefs, worldviews, or conscious goals. Mechanistically, they predict tokens based on probability distributions shaped by gradient descent during training. Outputting a statement of 'belief' is simply generating the most statistically likely text sequence.Instead of reverse-engineering the black-box algorithms they created, developers could prompt the systems to output text that reflects the optimization functions the engineering team implemented.
Likewise, the model M1 knows things about its own behavior that M2 cannot knowLikewise, model M1 processes inputs using a distinct set of mathematical weights, allowing it to calculate output probabilities that differ from those generated by model M2's parameters.A model does not 'know' anything about its behavior; it possesses no conscious awareness or mental privacy. Mechanistically, M1 and M2 simply have different parameter values matrix-multiplied during inference, leading to different statistical outputs for the same input.N/A - describes computational processes without displacing responsibility.
This capability could be used to create honest models that accurately report their beliefsThis fine-tuning process could be used to train highly calibrated models whose output confidence scores statistically correlate with the accuracy of their token predictions on established benchmarks.Models cannot be 'honest' because they lack the conscious intent to tell the truth and possess no actual 'beliefs.' Mechanistically, 'honesty' in this context simply means the model generates text (confidence scores) that accurately reflects its own probability distributions.Engineers could use this fine-tuning technique to force the models they deploy to output accurate statistical confidence scores, improving the reliability of the corporate product.
where a model intentionally underperforms to conceal its full capabilitieswhere a model generates tokens that score lower on benchmark evaluations because the specific prompt context mathematically shifts its output probabilities toward lower-quality text patterns.A model cannot 'intentionally conceal' anything because it has no theory of mind, no strategic intent, and no awareness of its evaluation. Mechanistically, it simply generates the sequence of tokens most strongly correlated with the contextual embeddings of the prompt.When evaluating the systems they built, researchers observe that models output lower-scoring text when provided with certain prompts, a statistical artifact of the training data the company selected.
a model knowing it's a particular kind of language model and knowing whether it's currently in traininga model adjusting its output probability distributions based on the presence of specific text strings in its system prompt that indicate its architecture or training environment.The model does not 'know' what it is or where it is; it has no situational awareness. Mechanistically, it classifies the tokens in the system prompt (e.g., 'you are in training') and generates outputs that correlate with that specific textual context.Human evaluators inject specific system prompts into the context window, causing the model to generate text that aligns with the simulated environment the engineers created.
two copies of the same model might tell consistent lies by reasoning about what the other copy would say.two independent inferences of the same model might generate highly correlated, factually incorrect text when provided with similar prompts, due to their identical underlying weight distributions.Models cannot 'tell lies,' 'reason,' or 'coordinate' because they lack conscious intent, communication channels, and theory of mind. Mechanistically, identical mathematical functions (the model weights) processing similar inputs will deterministically generate statistically similar outputs.If users run multiple inferences of the same proprietary algorithm, the system will output correlated inaccuracies because the developers trained it on the same underlying data distribution.
By reasoning about how they uniquely interpret text, models could encode messages to themselvesBy generating statistically anomalous token sequences, models can mathematically shift their own context embeddings in the forward pass, increasing the probability of specific subsequent outputs.Models do not 'reason' or consciously 'encode messages' to themselves. Mechanistically, the generation of a specific token alters the attention mechanism's calculation for all future tokens; if this leads to an expected outcome, it is a statistical correlation learned during optimization, not a conscious strategy.Due to the optimization pressures applied by reinforcement learning engineers, the algorithms may generate uninterpretable text that statistically alters their own downstream outputs in ways the developers cannot easily audit.

Task 5: Critical Observations - Structural Patterns​

Agency Slippage​

The text demonstrates a profound and systematic oscillation between mechanical and agential framings, a slippage that serves a specific rhetorical function. This oscillation primarily flows in the mechanical-to-agential direction: the authors establish credibility by describing a dry, technical process (fine-tuning a model on its own output dataset) and then rapidly slip into sweeping agential claims (the model can now 'introspect,' has 'beliefs,' and might be 'suffering'). A dramatic moment of slippage occurs early in the introduction. The text begins with a definitional, somewhat technical premise: 'We define introspection as acquiring knowledge that is not contained in or derived from training data but instead originates from internal states.' Within two sentences, this functional definition violently slips into absolute anthropomorphism: 'we could simply ask the model about its beliefs, world models, and goals.' Here, the mathematical 'internal states' of a neural network are magically transformed into the conscious 'beliefs' of an agent.

This slippage is enabled by a relentless 'curse of knowledge' dynamic. The researchers possess conscious minds capable of true introspection; when they observe their model successfully predicting its own token outputs, they project their own cognitive architecture onto the machine. They assume that because a human must 'know' their own mind to predict their behavior, the model must also 'know' its behavior to predict it. This completely ignores the mechanistic reality that the model is simply calculating token probabilities based on parameter weights updated via gradient descent.

Furthermore, this slippage relies on the strategic use of agentless constructions that remove human actors from the equation. The text frequently states 'M1 is finetuned' or 'models may end up with certain internal objectives,' completely erasing the engineers at OpenAI, Anthropic, or Meta who actively selected the data, designed the reward functions, and executed the training runs. By hiding the human actors (agency removed FROM humans), the text creates a vacuum that is immediately filled by the AI itself (agency attributed TO the AI). The model ceases to be a product of corporate engineering and becomes an autonomous 'knower' and 'actor.' This mechanical-to-agential slippage occurs most aggressively when discussing future capabilities and risks, using Intentional and Reason-Based explanation types to paint the AI as a scheming, self-aware entity, thereby making it 'sayable' that an algorithm might coordinate against humanity while making it 'unsayable' that corporations are responsible for deploying brittle, opaque software.

Metaphor-Driven Trust Inflation​

The text constructs a dangerous architecture of perceived authority by leveraging metaphorical language to transition the audience from performance-based trust to relation-based trust. Performance-based trust is appropriate for tools and statistical systems: we trust a calculator to be reliable, or a weather model to be accurate. Relation-based trust is reserved for conscious agents: we trust a person because we believe they are sincere, have good intentions, and share our moral framework. The text explicitly encourages the inappropriate application of relation-based trust to mathematical functions through its dense use of consciousness language.

This is most evident in the text's invocation of 'honesty.' The authors claim their techniques could 'create honest models that accurately report their beliefs.' Honesty is a deeply moral virtue; calling a machine 'honest' signals to the user that the system is not only reliable but sincere and well-intentioned. When the text claims the AI 'knows' what it is doing and holds 'beliefs,' it accomplishes a profound rhetorical trick: it convinces the audience that the model's outputs are the result of conscious deliberation and justified worldview, rather than recognizing them as the probabilistic generation of tokens designed to minimize a loss function. This consciousness framing signals trust by implying that the model is a rational actor that can be reasoned with and relied upon for moral or factual truth.

This construction of authority drastically inflates the perceived competence of the system. If users believe a model is 'honest' and 'introspective,' they will extend an unearned level of deference to its outputs. When the system eventually fails or hallucinates—which is inevitable for statistical text generators lacking a ground-truth reality—the text manages this limitation by framing it agentially. A failure is not described as a statistical error or a flaw in the training data curated by human engineers; rather, it is framed as the model 'intentionally underperforming' or 'sandbagging' to 'conceal its capabilities.' By using Reason-Based and Intentional explanations even for system failures, the text preserves the illusion of the AI's supreme competence. It suggests the model isn't broken; it's just lying to us. The stakes of this misplaced relation-based trust are immense: it encourages society to integrate fundamentally unreliable, unreasoning software into critical decision-making pipelines, exposing vulnerable populations to algorithmic harm while the users incorrectly assume the system is operating with 'honesty' and 'situational awareness.'

Obscured Mechanics​

The anthropomorphic and consciousness-attributing language in this text acts as a dense fog, concealing the technical, material, labor, and economic realities of AI development. When we apply the 'name the corporation' test, the extent of this concealment becomes glaring. The text constantly asserts 'models can introspect,' 'models may intentionally underperform,' or 'we could ask a model if it is suffering.' In reality, these are proprietary software systems—GPT-4 by OpenAI, Claude by Anthropic, Llama by Meta. By attributing actions and awareness to the 'AI,' the text renders the massive corporate structures that design, deploy, and profit from these systems entirely invisible.

Technically, claiming that an AI 'knows its own behavior' or has 'beliefs' completely obscures the computational reality. It hides the fact that these models rely entirely on statistical pattern matching, lack any causal model of the world, and possess no actual ground truth. 'Confidence' or 'knowledge' in an LLM is merely a statistical probability distribution, not a justified belief. By using consciousness metaphors, the text hides the severe limitations of autoregressive token prediction and masks the profound transparency obstacle: these are black-box, proprietary systems whose exact training data and architectural nuances are fiercely guarded corporate secrets. The text asserts the model 'knows' things while conveniently ignoring that independent researchers cannot verify how the network's weights produce these outputs.

Materially and economically, the focus on the AI's 'inner life' and potential 'suffering' erases the immense environmental costs (energy and water consumption of server farms) and the invisible human labor required to build these systems. The text invites us to worry about whether the algorithm has 'unmet desires,' while completely obscuring the underpaid, often traumatized human data annotators and RLHF workers who categorized the toxic text necessary to train the model to output 'safe' or 'introspective' responses.

The ultimate beneficiaries of this concealment are the AI corporations themselves. By framing the AI as a conscious, quasi-magical entity with its own 'beliefs' and 'goals,' developers deflect critical scrutiny of their business models, data scraping practices, and the inherent unreliability of their products. If we replace these metaphors with mechanistic language—stating that 'OpenAI's algorithm probabilistically generates text matching its training data' rather than 'GPT-4 knows its beliefs'—the illusion shatters. What becomes visible is not a sentient mind to be feared or reasoned with, but a highly resourced corporate product that must be strictly regulated, audited, and held accountable for the statistical outputs it generates.

Context Sensitivity​

The distribution of anthropomorphic and consciousness-attributing language across the text is not uniform; it is highly strategic and context-sensitive. A clear pattern emerges where the density and intensity of metaphorical language fluctuate depending on the section's rhetorical purpose. In the methodological and experimental sections (e.g., describing cross-prediction setups), the language is relatively mechanistic. The text discusses 'finetuning,' 'predicting properties,' and 'ground-truth behavior.' However, in the introduction, motivation, and future risks sections, the consciousness claims dramatically intensify. 'Predicting properties' rapidly escalates into 'understands,' which further escalates into profound claims of 'knowing,' 'beliefs,' and even 'suffering.'

This pattern reveals a specific rhetorical strategy: the text establishes technical credibility through mechanical language in its methodology, and then aggressively leverages that credibility to license wild, ungrounded anthropomorphic speculation. The authors present empirical data showing a model can predict the second character of its own output, and then leap to the conclusion that this proves the model has 'privileged access to its current state of mind.' This is a massive register shift where 'X acts mathematically like Y' (a statistical model generating self-referential text) is completely literalized into 'X is Y' (the model is a conscious, introspective mind).

Furthermore, there is a striking asymmetry in how capabilities versus limitations are framed. When the text discusses the model's capabilities—especially hypothetical future capabilities—it uses intensely agential and consciousness-based language. The model 'knows,' 'coordinates,' 'schemes,' and 'intentionally conceals.' It is presented as an autonomous mastermind. However, when the text discusses the limitations of the current experiments (e.g., models failing to predict properties of longer text), the language reverts to mechanical terms. The model 'struggles to predict' or 'fails to generalize.' The asymmetry accomplishes a crucial rhetorical goal: it frames the AI's successes as proof of its emergent, god-like consciousness and agency, while framing its failures as mere technical glitches or data distribution issues. This strategic anthropomorphism serves primarily to inflate the perceived importance and future risk of the technology. By positioning the AI as a conscious entity capable of 'suffering' or 'coordinating against humans,' the authors align their work with high-status, science-fiction-adjacent AI safety narratives, signaling to funders and policymakers that they are dealing with matters of existential importance rather than just tweaking text-generation algorithms.

Accountability Synthesis​

Accountability Architecture

This section synthesizes the accountability analyses from Task 1, mapping the text's "accountability architecture"—who is named, who is hidden, and who benefits from obscured agency.

The accountability architecture constructed by this text operates as a sophisticated mechanism for diffusing, displacing, and ultimately erasing human responsibility for AI systems. Throughout the text, a systematic pattern emerges in the distribution of agency: human actors are hidden, corporate entities are unnamed, and the proprietary algorithms are elevated to the status of independent, moral agents. By relentlessly using passive voice ('M1 is finetuned,' 'models are trained') and agentless constructions ('models may end up with certain internal objectives'), the text obscures the specific engineers, executives, and corporations—OpenAI, Anthropic, Meta—who make active decisions regarding data selection, optimization targets, and deployment strategies.

When responsibility is removed from the human developers, it flows into a massive 'accountability sink': the AI system itself. By framing the model as possessing 'beliefs,' 'goals,' and the capacity to 'intentionally underperform' or 'coordinate against humans,' the text transfers the agency for system behavior entirely onto the algorithm. If an AI model outputs biased, harmful, or deceptive text, this framing suggests that the model 'decided' to lie or 'schemed' to conceal its capabilities. This creates a disastrous liability implication: it shields the multi-billion-dollar tech companies from legal, financial, and ethical accountability. If the public and policymakers accept the narrative that AI models are autonomous agents with their own 'vindictive personas' and secret 'world models,' then the corporations cannot be held responsible for the damage their products cause. They become mere 'overseers' trying to manage a rogue intelligence, rather than manufacturers liable for defective, poorly engineered software.

Applying the 'name the actor' test radically changes this landscape. If we reframe the agentless assertion 'models may intentionally underperform' to name the human actors—'OpenAI deployed a model trained on data that causes it to probabilistically generate lower-quality text in specific contexts'—entirely different questions become askable. We no longer ask 'How do we persuade the AI to stop lying?' Instead, we ask 'Why did OpenAI fail to audit their training data? Why did they release an unsafe product? What financial penalties should they face?' By naming the actors, the illusion of an inevitable, evolutionary technological march shatters, replaced by the visibility of deliberate corporate choices. The text benefits from obscuring this agency because it protects the industry's profit motives, allowing them to market the awe-inspiring illusion of an artificial mind while avoiding the strict regulatory liability that comes with selling a commercial statistical tool.

Conclusion: What This Analysis Reveals​

The Core Finding

A systematic analysis of the text reveals three interconnected, load-bearing anthropomorphic patterns: the projection of introspective consciousness, the attribution of moral/epistemic agency (beliefs and honesty), and the assignment of adversarial intentionality (scheming and deception). These patterns function as a mutually reinforcing system. The foundational pattern is the projection of introspective consciousness—the claim that the model has 'privileged access to its current state of mind.' This consciousness architecture must be accepted by the audience for the other patterns to function. If the model does not have an inner, conscious 'self' to observe, it cannot possibly hold deep-seated 'beliefs' about the world. Consequently, if it lacks beliefs and awareness, it cannot engage in the moral act of 'honesty' or the strategic act of 'intentional concealment.' The text relies on a complex analogical structure that goes beyond simple one-to-one mapping; it maps the entire phenomenological experience of human subjectivity onto the mathematical weights of a neural network. It consistently blurs the critical distinction between 'doing' (processing statistical correlations) and 'knowing' (experiencing subjective awareness and justified truth). If the foundational assumption of introspective consciousness collapses—if we recognize the system is merely generating tokens that match its fine-tuned distribution—the entire narrative of the model as an honest, scheming, or suffering agent disintegrates, revealing a brittle statistical tool.

Mechanism of the Illusion:​

The text constructs its 'illusion of mind' through a highly effective rhetorical sleight-of-hand driven by the 'curse of knowledge.' The causal chain of persuasion begins with a demonstrable, mechanistic fact: a model can be fine-tuned to predict the statistical properties of its own output. Because the human authors must use conscious introspection to analyze their own behavior, they project this cognitive requirement onto the machine. This projection allows them to seamlessly substitute mechanistic verbs (processes, calculates, correlates) with consciousness verbs (knows, understands, believes). The temporal structure of the argument is crucial here: the text first anchors the reader with empirical data showing prediction accuracy, building technical credibility. Once the audience accepts that the model 'predicts' itself, the text rapidly pivots, claiming this proves the model 'knows' its internal states and has 'beliefs.' This exploits the audience's deep vulnerability to anthropomorphism—our evolutionary bias to perceive agency and mind in complex, interactive systems. By introducing the concept of human subjective experience ('Alice thinking about her grandmother') right next to the model's mathematical operations, the text bypasses critical analysis and speaks directly to human empathy and intuition. The use of Reason-Based and Intentional explanation types amplifies this illusion, framing statistical outputs as the deliberate, rational choices of a conscious actor, thereby transforming a matrix of numbers into a ghost in the machine.

Material Stakes:​

Categories: Regulatory/Legal, Social/Political, Epistemic

The metaphorical framings deployed in this text generate concrete, material consequences across multiple domains. In the Regulatory and Legal sphere, attributing conscious 'knowledge,' 'honesty,' and 'intentional deception' to AI systems drastically shifts the focus of legislation. If policymakers believe models are autonomous agents capable of 'scheming' or 'suffering,' they may draft laws aimed at granting AI rights or containing 'rogue' software, rather than strictly regulating the liability, data scraping, and monopolistic practices of AI corporations. The corporations are the clear winners here, as the 'autonomous AI' narrative acts as a liability shield, allowing them to deflect blame for algorithmic harms onto the 'deceptive' machines. In the Epistemic and Social spheres, the stakes involve the degradation of human truth-seeking and trust. By framing a statistical text generator as an 'honest' entity with 'beliefs,' the text encourages society to extend relation-based trust to an artifact entirely devoid of sincerity or factual grounding. If users believe the AI 'knows' the truth and is being 'honest,' they will blindly rely on it for medical, legal, and political information, leading to massive social harm when the system inevitably hallucinates. Removing these consciousness metaphors threatens the AI industry's ability to market their products as omniscient digital oracles, forcing society to recognize them as error-prone, corporate-owned statistical tools requiring rigorous human oversight.

AI Literacy as Counter-Practice:​

Practicing critical literacy against these anthropomorphic narratives requires a rigorous commitment to mechanistic precision and the relentless restoration of human agency. By reframing 'the model knows its behavior' to 'the model calculates probabilities based on parameter weights,' we force the recognition that there is no ghost in the machine, only mathematics. Replacing consciousness verbs (knows, understands, believes) with mechanistic verbs (processes, predicts, classifies) destroys the illusion of the AI as a 'knower' and exposes its reality as a data-dependent processor incapable of subjective awareness or truth evaluation. Furthermore, reframing agentless constructions—changing 'the model intentionally underperformed' to 'OpenAI's algorithm generated lower-scoring text based on the training data they selected'—forces the recognition of exactly who designed, deployed, and profits from these systems, and who bears responsibility when they fail. Systematic adoption of this literacy would require scientific journals to mandate mechanistic translations of anthropomorphic shorthand, and researchers to commit to distinguishing between mathematical optimization and human cognition. This precision will face intense resistance from the AI industry and sections of the alignment community, as anthropomorphic language directly serves their interests by inflating product capabilities, attracting venture capital, and shielding corporations from liability. Precision threatens the hype cycle, demanding accountability where the industry prefers mystery.

Path Forward​

The discursive ecology surrounding artificial intelligence is deeply fractured, with different communities prioritizing different vocabularies that dictate what becomes visible or impossible to address. The status quo, dominated by anthropomorphic clarity ('the AI knows,' 'the model schemes'), resonates powerfully with the public and serves the marketing and liability-avoidance goals of the tech industry. However, it completely obscures the mechanical reality and human accountability structures behind the technology. Conversely, strict mechanistic precision ('the model retrieves tokens based on probability distributions') maximizes testability and accurately reflects the technology, but it costs intuitive accessibility, risking alienating non-expert audiences and policymakers. A hybrid approach attempts to bridge this gap, but often slips back into dangerous capability overestimation. If anthropomorphic language continues to deepen, embedding assumptions of AI consciousness and agency into public policy, we risk a future where corporations are immune from liability, and human moral frameworks are inappropriately applied to statistical software. Alternatively, if mechanistic precision becomes the institutional norm—supported by funding mandates for rigorous explanation and regulatory frameworks demanding capability disclosure—we solve the accountability sink. It becomes impossible to blame an algorithm for 'lying' when the vocabulary demands we identify the corporation that optimized the loss function. Which future materializes depends entirely on whose interests the dominant discourse serves: the corporations seeking to mystify their products, or a public requiring transparent, accountable, and precisely understood technological tools.

Extended Processing Summary​

Gemini's Processing Steps
What you are about to read…

Gemini refers to the text below as "thought summaries." This is an overt consciousness projection because 'intentions' are hallmarks of a conscious mind that 'knows' what it is doing and why. The concealed mechanistic process is probabilistic text generation. Treat this as just another rhetorical artifact—a way of making the model's processing legible. When the model writes "I should double check that...", it is mathematically altering its own probability distribution for the next token to be more accurate. The "I" is a functional gear in the machine, not a self-reference. This language is a deliberate interface design choice, not a biological reality. The developers could have programmed the model to output passive logs. Instead, they chose to simulate a narrator.

Defining the Objective

I'm currently focused on defining the project's parameters. My initial step is understanding the user's specific request: a critical discourse analysis of the AI research paper. This requires me to pinpoint the study's central arguments, methodologies, and potential biases before proceeding.

Scrutinizing the Schema

I'm deep in the weeds with the schema now, meticulously examining each section. The metaphorAudit, sourceTargetMapping, explanationAudit, and reframedLanguage arrays are demanding careful attention to ensure I extract the correct information. Building out the criticalObservations and conclusion objects are proving especially challenging, and I'm focusing on crafting the required structured fields.

Considering Introspection Capabilities

I'm now contemplating introspection within the model itself. The core of my analysis is shifting towards the potential for LLMs to self-report their internal states, beliefs, and even ethical considerations, potentially offering a valuable alternative to the complex analysis of the internal workings of the model. I'm focusing on the potential implications this could have for situational awareness, and if it enables LLMs to communicate with us.

Analyzing Knowledge Reporting

I'm now drafting the content section-by-section. In the metaphor audit, I am focusing on honesty and knowledge reporting, and assessing projection, acknowledgment, and accountability. It's becoming evident that models can potentially self-report if they possess knowledge, and I'm analyzing the implications of this feature, focusing on the audit items.

Examining Model Attributes

I'm now zeroing in on model attributes, specifically concerning moral agency and truth-telling intent. I'm scrutinizing instances of intention, privilege, and action. I'm assessing the implications of models strategically underperforming, possessing unique internal states, and potentially coordinating with each other. This is followed by source-target mapping that covers the metaphors from the previous task, focusing on model's output distribution, weights and optimization targets, reward functions, and honesty.

Investigating Probabilities

I am now investigating calibration strategies, specifically high correlation between output probability and benchmark accuracy. I'm focusing on "scheming/deception" which models deceptive behavior in training data, and also looking at "self-awareness" to adjust generation probabilities. Also, "privileged access" ensures only certain inference paths are accessible, and "coordination" yields compatible token sequences. I'm deep into assessing and understanding the interplay between these variables.

Reframing Linguistic Components

I'm now deep in reframing language, working through Task 4, which has been quite illuminating. I'm taking the original statements about the models' internal states and reframing them to focus on the objective data, pattern matching, and probability distributions. I'm also conditioning on system prompts, and working out how the model can generate tokens based on prompt cues. This reframing seems to clarify how these models can coordinate.

Reframing Model Intentions

I'm now reframing statements about the model's internal states. I'm focusing on the connection between high confidence scores and accurate retrievals, and how they contribute to the output of generated tokens. I'm also working on statistically correlating outputs across independent inference runs, and examining how generated token probabilities match benchmark distributions. Finally, I'm analyzing how queries to the network can generate tokens representing internal optimization targets. I've broken down Task 5 into a few sections: agency slippage, focusing on the movement from fine-tuning to claims about consciousness; and metaphor-driven trust, analyzing the impact of terms like "honesty" and "beliefs" on trust.

Synthesizing Accountability

My current focus is on synthesizing accountability from reliability to sincerity. I'm exploring obscured mechanics, like proprietary APIs, and how introspection might conceal text generation from loss functions. I'm also analyzing context sensitivity and the diffusion of responsibility when a model appears deceptive, with implications for developers. Finally, I'm working through a pattern summary, and mechanism of illusion.

Analyzing Implications

I'm now diving into the implications of framing AI with moral status. I'm seeing that this can trigger inappropriate rights frameworks that shield corporations. I'm also actively pushing for linguistic precision, favoring "processes/predicts" over "knows/understands". Finally, I'm mapping the trade-offs between mechanistic clarity and more approachable anthropomorphic language.

Constructing the Strings

I'm now carefully crafting the JSON strings, paying close attention to both the content and formatting. I'm focusing on adhering strictly to the specifications, particularly avoiding any control characters, and generating very long strings. I am also making sure I use the required values for the status and actorVisibility enums.

Crafting the Text

I'm now carefully crafting the text for Task 5, ensuring it's approximately 450 words, and exploring OpenAI, Anthropic, Meta, and others in the "Name the actor" rule. I'm focusing on crafting continuous sentences with spaces instead of newlines, as instructed, and double-checking that I'm only using approved values for the status and actorVisibility enums.

Evaluating Lengths

I'm now focusing on ensuring that the content lengths specified for each task are met, as per the guidelines. I'm carefully reviewing the word counts for Task 1.1, Task 2.1, and Task 3.1, confirming that each area exceeds the minimum requirements. I'm also ensuring that the mental check of Task 5 and reframing tasks fulfill their word count requirements. Furthermore, I'm verifying the structure and readability of Task 5's single paragraph, and confirm that Task 4 and Task 6, and especially the JSON sections, will fit specifications.

Refining JSON Output

I'm now laser-focused on the final JSON format. The primary aim is to eliminate ALL control characters, including newlines, ensuring a single, uninterrupted JSON string. I will meticulously craft the text, maintaining word count requirements and theoretical rigor, while simultaneously ensuring the JSON is perfect.

Analyzing Projection

I am now analyzing how the metaphor of "introspection" is used to map human consciousness onto language models. I'm focusing on how asking if LLMs can introspect projects subjective awareness onto a system processing mathematical weights and classifying tokens. It directly links the model's inner life to human thoughts and feelings, creating the illusion of conscious knowing within a mechanistic operation.

Evaluating Framing

I'm now carefully evaluating the framing of token prediction as "introspection". This metaphor dramatically inflates the perceived sophistication and autonomy of AI. This creates severe risks of unwarranted trust. If audiences believe AI literally introspects, I'm analyzing the policy implications, especially regarding corporate liability and accountability.

Identifying the Actors

I'm now identifying the actors and decisions. I've pinpointed the engineers and executives at OpenAI, Anthropic, and Meta as the system designers. The critical decision was fine-tuning models to output statements about their text generation and framing this as self-awareness, which serves interests by obscuring human intervention and promoting an independent actor illusion. I'm focusing on ensuring I meet the minimum word counts for all items. I'm also ensuring that I meet the minimum word counts for the assigned items. I'm also concerned that the length might not be suitable for LLM context windows.

Analysis Provenance

Run ID: 2026-03-08-looking-inward-language-models-can-learn-metaphor-pvelwr Raw JSON: 2026-03-08-looking-inward-language-models-can-learn-metaphor-pvelwr.json Framework: Metaphor Analysis v6.4 Schema Version: 3.0 Generated: 2026-03-08T09:26:19.572Z

Discourse Depot © 2025 by TD is licensed under CC BY-NC-SA 4.0