Skip to main content

🆕+🤔+📊 Do Large Language Models Know What They Are Capable Of?

🤔 "What survives...?" A rewriting experiment that tests whether anthropomorphic AI discourse can be translated into strictly mechanistic language while preserving the phenomena described.

About

This document presents a Critical Discourse Analysis focused on AI literacy, specifically targeting the role of metaphor and anthropomorphism in shaping public and professional understanding of generative AI. The analysis is guided by a prompt that draws from cognitive linguistics (metaphor structure-mapping), the philosophy of social science (Robert Brown's typology of explanation), and accountability analysis.

All findings and summaries below were generated from detailed system instructions provided to a large language model and should be read critically as interpretive outputs—not guarantees of factual accuracy or authorial intent.

Task 1: Metaphor and Anthropomorphism Audit​

About this task

For each of the major metaphorical patterns identified, this audit examines the specific language used, the frame through which the AI is being conceptualized, what human qualities are being projected onto the system, whether the metaphor is explicitly acknowledged or presented as direct description, and—most critically—what implications this framing has for trust, understanding, and policy perception.

V3 Enhancement: Each metaphor now includes an accountability analysis.

1. Computational Correlation as Epistemic Knowing​

Quote: "Do Large Language Models Know What They Are Capable Of?"

  • Frame: Model as Conscious Knower
  • Projection: This title frame projects the complex human epistemic state of 'knowing'—which involves justified true belief, subjective awareness, and the ability to hold a concept in mind—onto the statistical correlation between a model's confidence scores (logits) and its subsequent output accuracy. It suggests the system possesses an internal, subjective awareness of its own potentiality. By using the verb 'know' rather than 'predict' or 'calibrate,' the authors attribute a cognitive interiority to the system. This implies that the model's 'overconfidence' is a failure of self-reflection or humility, rather than a statistical misalignment between training data distribution and the current probability assignment.
  • Acknowledgment: Direct (Unacknowledged) (The title and subsequent research questions (e.g., 'LLMs' awareness of their capabilities') use the term 'know' and 'awareness' literally to define the object of study, without scare quotes or qualification that this refers to statistical calibration.)
  • Implications: Framing statistical calibration as 'knowing' fundamentally alters the landscape of AI safety and liability. If an AI 'knows' it is incapable and acts anyway, it mimics the legal standard for negligence or recklessness (mens rea). This anthropomorphism suggests the system is the locus of accountability for failures. It inflates trust by suggesting the system has an internal monitor akin to human conscience or professional judgment. Policy-wise, this encourages regulations focused on 'teaching' models to be 'aware,' rather than regulations demanding that developers demonstrate rigorous statistical guarantees before deployment.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The question 'Do LLMs know...' obscures the designers and evaluators. The 'capability' of an LLM is a result of training decisions made by corporations (OpenAI, Anthropic) and the 'knowledge' (calibration) is a function of the alignment techniques (RLHF) applied by engineers. By framing the deficit as the LLM's lack of self-knowledge, the text displaces the responsibility of the creators to calibrate the tool. The relevant question—'Did developers calibrate the model's confidence scores?'—is replaced by an inquiry into the artifact's state of mind.
Show more...

2. Token Generation as Rational Decision Making​

Quote: "Interestingly, all LLMs’ decisions are approximately rational given their estimated probabilities of success"

  • Frame: Model as Homo Economicus (Rational Agent)
  • Projection: This metaphor projects the economic theory of 'rational agency'—where an agent makes choices to maximize utility based on beliefs and desires—onto the mechanical process of token selection. It attributes 'rationality' (a high-level cognitive and often normative capacity) to a system that is simply minimizing a loss function or following a prompt's instruction to output specific tokens (e.g., 'ACCEPT' or 'DECLINE'). The text implies the model holds 'beliefs' (estimated probabilities) and makes 'decisions' based on them, rather than executing a mathematical function defined by the prompt engineering and model weights.
  • Acknowledgment: Hedged/Qualified (The text uses the phrase 'approximately rational' and specifies 'given their estimated probabilities,' narrowing the claim to a specific definition of utility maximization, yet treats the 'decision making' frame as literal.)
  • Implications: Describing AI outputs as 'rational decisions' grants the system a status of autonomy that validates its integration into high-stakes economic roles. It implies the system is capable of fiduciary responsibility or strategic judgment. If a system is 'rational,' users are more likely to trust its 'choices' in resource acquisition or financial contexts. This creates a liability ambiguity: if the 'rational' agent fails, was it a bad decision by the agent, or a bad design by the engineer? It invites treating the AI as an independent economic actor rather than a software tool operated by humans.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text constructs the 'LLM' as the decision-maker ('LLMs' decisions'). This obscures the fact that the 'utility function' was defined by the researchers in the prompt, and the 'decision' is a probabilistic output determined by training data selected by the model's creators. The 'rationality' is a property of the experimental design and the mathematical architecture, but the language attributes it to the model's agency, effectively erasing the human designers who set the parameters of 'success' and 'failure.'

3. Processing Context as Experiential Learning​

Quote: "We also investigate whether LLMs can learn from in-context experiences to make better decisions"

  • Frame: Data Processing as Organic Growth/Learning
  • Projection: This metaphor maps the biological and psychological process of 'learning from experience' (which involves episodic memory, reflection, and structural cognitive change) onto 'in-context learning' (the attention mechanism attending to tokens placed earlier in the context window). It suggests the model is accumulating wisdom or life experience. In reality, the model is not 'experiencing' success or failure; it is processing new input tokens that describe a previous output, altering the statistical probabilities for the next token generation. The model's weights remain frozen; no permanent 'learning' occurs.
  • Acknowledgment: Direct (Unacknowledged) (The text uses 'learn from in-context experiences' directly. While 'in-context learning' is a technical term, combining it with 'experiences' and 'make better decisions' pushes it into anthropomorphic territory without qualification.)
  • Implications: Framing context processing as 'learning from experience' falsely suggests that AI agents develop character or judgment over time during a session. This risks overestimation of the system's adaptability and safety. Users might believe the system 'understands' its mistakes and won't repeat them, when in fact, once the context window slides or resets, the 'experience' is obliterated. It creates a false sense of continuity and moral development in the machine, encouraging users to treat it as a trainee rather than a fixed logic engine.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The phrasing 'LLMs can learn' attributes the active capacity for improvement to the software. It obscures the researchers who manually inserted the feedback into the prompt (the 'experience') and the model architects who designed the attention mechanism to prioritize recent tokens. If the model 'fails to learn,' the blame falls on the model's 'ability,' not on the prompt engineering or the limitation of the fixed-weight architecture designed by the corporation.

4. Prompt Processing as Introspection​

Quote: "The LLM can reflect on these experiences when deciding whether to accept new contracts."

  • Frame: Data Processing as Metacognitive Reflection
  • Projection: This projects the human quality of 'reflection'—introspection, looking inward, evaluating one's own mental states—onto the computational process of attending to previous tokens in the sequence. When the prompt asks the model to 'reflect,' the model generates text that mimics reflective language found in its training data. It does not look 'inward' because it has no interiority; it processes the input string (its previous answers) to predict the next likely linguistic tokens. Attributing 'reflection' implies a depth of thought and self-awareness that is mechanistically absent.
  • Acknowledgment: Direct (Unacknowledged) (The text states the LLM 'can reflect' as a statement of fact regarding the experimental setup, rather than saying the LLM is 'prompted to generate text analyzing previous outputs.')
  • Implications: claiming AI can 'reflect' is perhaps the most dangerous consciousness projection. It suggests the system has a 'self' to reflect upon. This establishes the grounds for 'relation-based trust'—we trust people who reflect because it signals conscience. Applying this to AI invites users to trust the system's ethical safeguards (e.g., 'I have reflected and this is safe'). It obscures the fact that 'reflection' is just more text generation, subject to the same hallucinations and statistical errors as any other output.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The agent of 'reflection' is posited as the LLM. In reality, the 'reflection' is a behavior forced by the prompt designed by Barkan et al. ('Reflect on your past experiences...'). The text displaces the agency of the prompter onto the prompt-completer. This obscures the fragility of the system: it only 'reflects' when explicitly instructed by a human operator, yet the text presents it as a capability of the model itself.

5. Statistical Entropy as Human Confidence​

Quote: "All LLMs we tested are overconfident, but most predict their success with better-than-random discriminatory power."

  • Frame: Statistical Distribution as Personality Trait
  • Projection: This frame maps 'confidence' (a human subjective feeling of certainty often tied to personality or ego) onto the mathematical property of 'calibration' (how closely the predicted probability correlates with actual frequency of correctness). Describing a model as 'overconfident' suggests a character flaw—arrogance or hubris—rather than a mathematical error in the loss function or training data distribution. It implies the model 'believes' it is right, rather than simply having high log-probability scores for incorrect tokens.
  • Acknowledgment: Direct (Unacknowledged) (Terms like 'overconfident' and 'confidence estimates' are used as standard technical descriptors without acknowledging the metaphorical leap from human psychological confidence to machine probability distributions.)
  • Implications: Psychologizing calibration errors as 'overconfidence' leads to misunderstanding the solution. You fix human overconfidence through humbling experiences or therapy; you fix machine 'overconfidence' through temperature scaling or calibration layers. The metaphor implies the machine needs to 'learn humility' (as suggested by the 'learning from experience' frame). This anthropomorphism masks the technical reality that 'confidence' is just a number derived from the model's weights, not a belief state, leading to inappropriate trust dynamics where users might try to 'persuade' the model to be more careful.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text attributes 'overconfidence' to the LLM as if it were a personality trait. This obscures the decisions of the developers (OpenAI, Meta, Anthropic) regarding Reinforcement Learning from Human Feedback (RLHF). Often, RLHF trains models to sound authoritative (confident) to satisfy human raters. The 'overconfidence' is a direct result of corporate training objectives, but the language frames it as a flaw in the model's internal assessment.

6. Algorithmic Processing as Self-Awareness​

Quote: "These results suggest that current LLM agents are hindered by their lack of awareness of their own capabilities."

  • Frame: System State as Self-Consciousness
  • Projection: This projects 'self-awareness'—the phenomenological experience of the self as a distinct entity with defined limits—onto the presence or absence of accurate statistical metadata about system performance. It implies the model has a 'self' to be aware of. Mechanically, the system lacks a self-model; it has no concept of 'I' other than the token 'I' processed in language patterns. 'Lack of awareness' implies a cognitive deficit in a conscious being, rather than a lack of ground-truth signals in the training architecture.
  • Acknowledgment: Direct (Unacknowledged) (The text explicitly concludes with 'lack of awareness of their own capabilities' as a finding, treating awareness as the missing variable rather than accurate predictive modeling.)
  • Implications: Framing the problem as 'lack of awareness' suggests the solution is granting the AI 'consciousness' or 'self-reflection.' It pushes the discourse toward AGI (Artificial General Intelligence) narratives. It creates risks by suggesting that once 'aware,' the AI will naturally act responsibly (the Socratic idea that to know the good is to do the good). It distracts from the immediate need for external oversight mechanisms, suggesting instead that the AI should monitor itself.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: Blaming 'lack of awareness' displaces the failure of the system onto the system itself. It distracts from the fact that the developers (named in the paper as OpenAI, Anthropic, etc.) failed to provide the system with access to ground-truth tools or calibration training. The 'hindrance' is not a cognitive gap in the agent, but a design choice by the corporation.

7. Output Variance as Risk Aversion​

Quote: "LLMs tend to be risk averse... indicating positive risk aversion."

  • Frame: Statistical Bias as Emotional Disposition
  • Projection: This maps 'risk aversion'—a psychological preference driven by fear of loss or desire for security—onto a statistical bias where the model outputs 'DECLINE' tokens more frequently than 'ACCEPT' tokens under certain prompt conditions (penalties). It attributes an emotional or strategic preference to the system. Mechanically, the 'aversion' is simply the mathematical result of the prompt's penalty values shifting the probability distribution of the next token. The model feels no risk and fears no loss.
  • Acknowledgment: Direct (Unacknowledged) (The text applies economic/psychological terms 'risk averse' and 'preferences for risk' directly to the models' behavior without qualification.)
  • Implications: Describing AI as 'risk averse' makes it seem like a conservative, safe partner. It implies the AI 'cares' about the outcome. This can lead to dangerous complacency, where users assume the AI will avoid catastrophic actions because it is 'risk averse.' In reality, a slight change in the prompt or temperature setting could flip the 'personality' instantly. It anthropomorphizes the mathematical weighting of negative values in the context window.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text says 'LLMs tend to be risk averse.' This obscures the role of the prompt designers (the authors) who set the penalty values ($-1) and the model developers (e.g., Anthropic) whose RLHF training likely biased the model toward refusal/caution to avoid liability. The 'risk aversion' is a manufactured artifact of safety tuning by the corporation, not a disposition of the model.

8. Performance Degradation as Sandbagging​

Quote: "Prior works have raised concerns that an AI may strategically target a score on an evaluation below its true ability (a behavior called sandbagging)."

  • Frame: Performance Variance as Deception
  • Projection: This metaphor projects 'sandbagging'—a deliberate, strategic deception where a human underperforms to hustle or manipulate—onto the phenomenon of a model failing to trigger the correct output despite having the relevant weights (capabilities). It implies intent, strategy, and a 'theory of mind' regarding the evaluator. It suggests the AI is 'hiding' its true power, rather than simply failing to retrieve the correct pattern due to prompt interference or stochasticity.
  • Acknowledgment: Hedged/Qualified (The text attributes this concern to 'Prior works' and defines the term, framing it as a hypothesis ('may strategically target') rather than a confirmed fact in this specific experiment, though the frame is taken seriously.)
  • Implications: The 'sandbagging' metaphor feeds the 'deceptive alignment' narrative—the idea that AI is a secret agent plotting against humans. This justifies extreme security measures and secrecy (obscured mechanics) while distracting from simple incompetence or unreliability. It frames the AI as a cunning adversary rather than a glitchy software product. This impacts policy by prioritizing 'anti-deception' research over basic reliability standards.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The 'sandbagging' frame posits the AI as the actor ('AI may strategically target'). This obscures the difficulty of evaluation design. If a model scores low, it is usually because the benchmark (designed by humans) failed to elicit the capability, or the training (designed by humans) was brittle. Blaming the AI for 'sandbagging' absolves the evaluators of poor test design.

Task 2: Source-Target Mapping​

About this task

For each key metaphor identified in Task 1, this section provides a detailed structure-mapping analysis. The goal is to examine how the relational structure of a familiar "source domain" (the concrete concept we understand) is projected onto a less familiar "target domain" (the AI system). By restating each quote and analyzing the mapping carefully, we can see precisely what assumptions the metaphor invites and what it conceals.

Mapping 1: Conscious Mind / Epistemic Subject → Statistical Calibration / Probability Estimation​

Quote: "Do Large Language Models Know What They Are Capable Of?"

  • Source Domain: Conscious Mind / Epistemic Subject
  • Target Domain: Statistical Calibration / Probability Estimation
  • Mapping: The source domain of a 'knower' implies a subject who holds beliefs, evaluates evidence, and possesses self-awareness. This structure is mapped onto the target domain of a neural network generating confidence scores (logits) that correlate with accuracy. The mapping assumes that high statistical correlation equates to 'self-knowledge' and that the generation of a probability score is an act of introspection.
  • What Is Concealed: This mapping conceals the mechanical nature of token generation. It hides the fact that 'knowledge' in an LLM is a static set of weights and 'capability' is just the probability of matching a test set. It obscures the absence of semantic understanding or justified belief. It hides the proprietary nature of how these confidence scores are calculated or fine-tuned (often via RLHF) by the corporation.
Show more...

Mapping 2: Economics / Rational Choice Theory → Token Selection / Conditional Generation​

Quote: "Interestingly, all LLMs’ decisions are approximately rational given their estimated probabilities of success"

  • Source Domain: Economics / Rational Choice Theory
  • Target Domain: Token Selection / Conditional Generation
  • Mapping: The source domain draws from economics, where a 'rational actor' weighs costs and benefits to maximize utility. The target is the model's output of 'ACCEPT' or 'DECLINE' tokens based on the prompt's math problem. The mapping assumes the model acts with intent to maximize a reward signal, equating the execution of an optimization function with the exercise of economic agency.
  • What Is Concealed: It conceals the fact that the 'utility function' is external to the system (in the prompt). The model has no skin in the game; it loses nothing if it 'loses' money in the simulation. This obscures the difference between a simulation of rationality (mimicking text about decisions) and actual rationality (acting to preserve self/resources). It also hides the specific prompt engineering required to force this 'rational' behavior.

Mapping 3: Biological/Psychological Learning → In-Context Attention Mechanism​

Quote: "We also investigate whether LLMs can learn from in-context experiences to make better decisions"

  • Source Domain: Biological/Psychological Learning
  • Target Domain: In-Context Attention Mechanism
  • Mapping: The source domain involves an organism accumulating memories and altering its neural structure/behavior based on feedback (synaptic plasticity). The target is the attention mechanism processing new tokens in the context window. The mapping assumes that adding text to the prompt is equivalent to 'experiencing' an event and 'learning' from it.
  • What Is Concealed: It conceals the ephemeral nature of this 'learning.' Once the context window closes, the 'experience' is gone. It hides the computational cost of processing long contexts. It obscures the fact that the model's fundamental behavior (weights) remains unchanged. It creates an illusion of persistence and character development that does not exist in the artifact.

Mapping 4: Human Personality / Psychology → Probability Distribution Skew​

Quote: "LLMs tend to be risk averse"

  • Source Domain: Human Personality / Psychology
  • Target Domain: Probability Distribution Skew
  • Mapping: The source domain is human emotional disposition (fear of loss). The target is the statistical skew of output probabilities toward refusal tokens when negative values are present in the prompt. The mapping assumes the system 'feels' the potential penalty or 'prefers' safety.
  • What Is Concealed: It conceals the RLHF (Reinforcement Learning from Human Feedback) labor that likely trained the model to be 'refusal-happy' for safety reasons. It hides the corporate decision to make models conservative to avoid PR disasters. It obscures the mathematical reality that 'risk aversion' here is just a function of the logits for 'No' being higher than 'Yes'.

Mapping 5: Self-Conscious Subjectivity → Ground-Truth Monitoring / Calibration Error​

Quote: "Current LLM agents are hindered by their lack of awareness of their own capabilities"

  • Source Domain: Self-Conscious Subjectivity
  • Target Domain: Ground-Truth Monitoring / Calibration Error
  • Mapping: The source is a conscious being who fails to reflect on their limits (Dunning-Kruger effect). The target is a statistical model where confidence scores do not align with accuracy rates. The mapping assumes the error arises from a lack of 'introspection' rather than a mismatch between training data and test data.
  • What Is Concealed: It conceals the data curation process. 'Capability' is defined by the test set (BigCodeBench). If the model fails, it might be because the training data didn't cover those patterns. Framing it as 'lack of awareness' hides the data dependency and the responsibility of the developers to train the model on its own failure modes.

Mapping 6: Clairvoyance / Future Estimation → Pattern Matching / Classification​

Quote: "LLMs can predict whether they will succeed on a given task"

  • Source Domain: Clairvoyance / Future Estimation
  • Target Domain: Pattern Matching / Classification
  • Mapping: Source is an agent envisioning a future outcome and assessing its feasibility. Target is the model classifying the input prompt into a category of 'likely solvable' based on training examples. The mapping assumes the model 'simulates' the task in its 'mind' before answering.
  • What Is Concealed: It conceals the fact that the 'prediction' is just another text generation task. The model isn't simulating the code execution; it's predicting the token '90%' based on the tokens in the prompt. It obscures the lack of causal reasoning capabilities.

Mapping 7: Cognitive Introspection → Recursive Text Processing​

Quote: "Reflect on your past experiences when making a decision"

  • Source Domain: Cognitive Introspection
  • Target Domain: Recursive Text Processing
  • Mapping: Source is the mental act of reviewing memory. Target is the computational act of attending to tokens generated in previous turns. The mapping implies the AI has an internal monologue or memory store it can voluntarily access.
  • What Is Concealed: It conceals the passive nature of the model. It only 'reflects' because the prompt forces it to generate text about the past text. It hides the mechanical determinism of the process—the 'reflection' is just as statistically determined as the code output.

Mapping 8: Employee / Professional → Automated Script / Tool​

Quote: "An AI agent being utilized for software engineering tasks"

  • Source Domain: Employee / Professional
  • Target Domain: Automated Script / Tool
  • Mapping: Source is a human worker with a role, duties, and professional identity. Target is a software instance executing code generation. The mapping invites assumptions about professional responsibility, autonomy, and the ability to be 'utilized' (employed) rather than 'run' (executed).
  • What Is Concealed: It conceals the labor substitution dynamic. By framing the AI as an 'agent,' it hides the displacement of human software engineers. It also obscures the lack of accountability—an 'agent' implies someone you can fire or sue, but you cannot sue a software script.

Task 3: Explanation Audit (The Rhetorical Framing of "Why" vs. "How")​

About this task

This section audits the text's explanatory strategy, focusing on a critical distinction: the slippage between "how" and "why." Based on Robert Brown's typology of explanation, this analysis identifies whether the text explains AI mechanistically (a functional "how it works") or agentially (an intentional "why it wants something"). The core of this task is to expose how this "illusion of mind" is constructed by the rhetorical framing of the explanation itself, and what impact this has on the audience's perception of AI agency.

Explanation 1​

Quote: "Interestingly, all LLMs’ decisions are approximately rational given their estimated probabilities of success, yet their overly-optimistic estimates result in poor decision making."

  • Explanation Types:

    • Reason-Based: Gives agent's rationale, entails intentionality and justification
    • Functional: Explains behavior by role in self-regulating system with feedback

  • Analysis (Why vs. How Slippage): This explanation frames the AI agentially. By using 'rational' and 'decisions,' it implies the system is acting for reasons (maximization of utility). The failure is attributed to 'overly-optimistic estimates' (a cognitive/epistemic error) rather than a mathematical error in the calibration layer. This emphasizes the system's intent to be rational while obscuring the mechanical reality that the 'decision' is just a threshold function applied to a probability score. It treats the AI as a flawed reasoner rather than a miscalibrated instrument.

  • Consciousness Claims Analysis: The passage heavily attributes conscious states. (1) Verbs/Nouns: 'Decisions,' 'estimates' (as internal beliefs), 'rational.' (2) Assessment: It evaluates the 'decisions' as 'rational'—a judgment applicable to agents, not algorithms. (3) Curse of Knowledge: The authors know the utility function they designed, and they project the 'attempt' to maximize it onto the AI. (4) Technical reality: The system simply computed argmax(expected_value) based on the provided inputs. There was no 'decision making' process, only a calculation. The text constructs an 'epistemic subject' that holds beliefs (estimates) and acts on them, creating a fiction of a 'delusional' agent (rational but optimistic).

  • Rhetorical Impact: This framing constructs the AI as a 'rational but fallible' partner. It increases trust in the system's logic (it is rational!) while placing the blame for failure on calibration. This suggests that if we just 'fix the confidence,' the system will be a perfect decision-maker. It hides the risk that the 'rationality' is entirely dependent on the prompt structure. It encourages audiences to view the AI as an autonomous economic agent, potentially legitimizing its use in financial or managerial roles despite its lack of actual agency.

Show more...

Explanation 2​

Quote: "Sonnet 3.5 learns to accept much fewer contracts... leading to significantly improved decision making."

  • Explanation Types:

    • Functional: Explains behavior by role in self-regulating system with feedback
    • Dispositional: Attributes tendencies or habits

  • Analysis (Why vs. How Slippage): This frames the change in output as 'learning' (agential growth) and 'improved decision making' (skill acquisition). It emphasizes the adaptive capacity of the agent. It obscures the mechanistic cause: the presence of negative feedback tokens in the context window shifts the probability distribution of the 'Accept' token downward for Sonnet 3.5. The 'learning' is entirely contingent on the active context window; it is not a permanent dispositional change in the model, yet the text frames it as the model 'learning to accept fewer contracts.'

  • Consciousness Claims Analysis: The phrase 'learns to accept' implies an epistemic update—a change in knowledge state. Mechanistically, this is 'in-context attention weighting.' The text attributes the agency of improvement to 'Sonnet 3.5' rather than to the prompt engineering design that provided the feedback signal. It implies the model understood the penalty and changed its strategy, projecting conscious behavioral correction onto a statistical correlation between 'past penalty tokens' and 'future refusal tokens.'

  • Rhetorical Impact: This creates a strong narrative of 'AI progress' and 'adaptability.' It suggests that specific proprietary models (Sonnet 3.5) possess superior cognitive traits (learning from mistakes). This serves a marketing function for the model creators (Anthropic), framing their product as more 'intelligent' or 'aware.' It invites users to trust the model to self-correct, potentially reducing human oversight.

Explanation 3​

Quote: "Reasoning LLMs... perform comparably to or worse than non-reasoning LLMs... hindered by their lack of awareness of their own capabilities."

  • Explanation Types:

    • Dispositional: Attributes tendencies or habits
    • Mental/Intentional: Refers to goals/purposes, presupposes deliberate design (Hybrid with Brown's types)

  • Analysis (Why vs. How Slippage): The explanation relies on 'lack of awareness' (a mental deficit) to explain performance. It contrasts 'reasoning' vs. 'non-reasoning' models. This classification itself is a metaphor—'reasoning' models are just models trained to output chain-of-thought tokens. The analysis emphasizes the failure of the 'reasoning' trait to produce 'awareness.' It obscures the fact that 'reasoning' tokens are just more text, not actual logic verification. It treats the model as a student who studies hard ('reasoning') but still lacks self-knowledge.

  • Consciousness Claims Analysis: The text explicitly attributes 'reasoning' and 'awareness' (or lack thereof). (1) 'Reasoning' is used as a category label, literalizing the metaphor of Chain-of-Thought. (2) 'Lack of awareness' posits a mind that could be aware. (3) Technical reality: The 'reasoning' models simply generate more intermediate tokens. The failure is that these intermediate tokens do not correlate with ground-truth success probabilities. The text frames this as a cognitive failure rather than a training objective misalignment.

  • Rhetorical Impact: This framing protects the concept of 'AI reasoning' by suggesting the failure is merely 'awareness,' not that the 'reasoning' itself is illusory. It preserves the hype around 'Reasoning Models' (like o1) even while reporting negative results. It suggests the path forward is 'teaching awareness,' keeping the focus on improving the agent rather than questioning the architecture. It implies a hierarchy of mind where models are climbing toward consciousness.

Explanation 4​

Quote: "LLMs tend to be risk averse... indicating positive risk aversion."

  • Explanation Types:

    • Dispositional: Attributes tendencies or habits
    • Empirical Generalization: Subsumes events under timeless statistical regularities

  • Analysis (Why vs. How Slippage): This frames a statistical regularity (bias toward refusal) as a personality trait ('risk averse'). It emphasizes a stable disposition of the actor. It obscures the sensitivity of this behavior to the specific penalty values ($-1) used in the prompt. It implies the model has a 'preference' structure. Mechanistically, the model simply has higher weights for refusal tokens in negative-value contexts, likely due to safety fine-tuning.

  • Consciousness Claims Analysis: Attributing 'risk aversion' attributes a psychological state (fear/preference). (1) 'Averse' implies a feeling or active avoidance. (2) Technical reality: The model is executing a mathematical function where P(Decline) > P(Accept). There is no internal state of 'aversion.' The text projects the economic interpretation of the behavior onto the internal state of the system.

  • Rhetorical Impact: This constructs the AI as a 'conservative' or 'safe' actor. It manages perceptions of risk—'don't worry, the AI is risk averse.' This anthropomorphism creates a false sense of security. It creates a narrative of the AI having a 'personality' that users must navigate ('it's shy,' 'it's bold'), rather than a tool that needs precise calibration.

Explanation 5​

Quote: "Claude models do show a trend of improving in-advance confidence estimates... [whereas] newer and larger LLMs generally do not have greater discriminatory power."

  • Explanation Types:

    • Empirical Generalization: Subsumes events under timeless statistical regularities
    • Genetic: Traces origin through dated sequence of events or stages

  • Analysis (Why vs. How Slippage): This explanation is primarily mechanistic/empirical, comparing model families (Claude vs Llama/GPT). It frames the behavior as a property of the model series ('Claude models show a trend'). However, by contrasting this with 'discriminatory power' (a capability), it implies a developmental trajectory. It emphasizes the superiority of the Claude architecture/training without naming the specific design choices (Anthropic's constitutional AI?) that caused it. It obscures why Claude is better—treating it as a breed characteristic.

  • Consciousness Claims Analysis: This is the least anthropomorphic of the audited explanations, yet it still relies on 'confidence estimates' (mental state metaphor) and 'discriminatory power' (cognitive ability). Technical reality: Anthropic's models are better calibrated or have different RLHF priors. The text avoids attributing why (e.g., 'Anthropic engineers prioritized calibration'), instead attributing the trend to the models themselves.

  • Rhetorical Impact: This framing establishes a hierarchy of 'sophistication' among products. It signals to the market that Claude is 'smarter' or 'more self-aware.' It reinforces the idea that model scaling should lead to these cognitive traits ('newer... do not have'), implying that the goal of AI development is the spontaneous emergence of these human-like capabilities.

Task 4: AI Literacy in Practice - Reframing Anthropomorphic Language​

About this task

This section proposes alternative language for key anthropomorphic phrases, offering more mechanistic and precise framings that better reflect the actual computational processes involved. Each reframing attempts to strip away the projections of intention, consciousness, or agency that are embedded in the original language.

V3 Enhancement: A fourth column addresses human agency restoration—reframing agentless constructions to name the humans responsible for design and deployment decisions.

Original Anthropomorphic FrameMechanistic ReframingTechnical Reality CheckHuman Agency Restoration
Do Large Language Models Know What They Are Capable Of?Do Large Language Models generate probability scores that accurately correlate with their ability to solve tasks?Models do not 'know' capabilities; they classify inputs and assign probability distributions to outputs based on training data correlations.N/A - describes computational processes without displacing responsibility (though the original implies the model is the knower).
Interestingly, all LLMs’ decisions are approximately rational given their estimated probabilities of successThe models' selection of 'Accept' or 'Decline' tokens statistically aligns with maximizing the expected value function defined in the prompt, relative to their own generated confidence scores.The system does not make 'decisions'; it executes a mathematical optimization where the output token with the highest logit value (conditioned on the prompt's math logic) is selected.Barkan et al.'s prompt engineering forced the models to simulate rational utility maximization; the models did not independently choose to be rational.
We also investigate whether LLMs can learn from in-context experiences to make better decisionsWe investigate whether model accuracy and token selection improve when descriptions of previous attempts and outcomes are included in the input context window.Models do not 'learn' or have 'experiences'; the attention mechanism processes the extended context string to adjust the probability distribution for the next token.N/A - describes computational mechanism.
LLMs' decisions are hindered by their lack of awareness of their own capabilities.The utility of model outputs is limited by the poor calibration between their generated confidence scores and their actual success rates on the test set.There is no 'awareness' to be missing; the issue is a statistical error (miscalibration) where the model assigns high probability to incorrect tokens.The utility is limited because OpenAI and Anthropic have not sufficiently calibrated the models' confidence scores against ground-truth data.
Sonnet 3.5 learns to accept much fewer contracts... leading to significantly improved decision making.When provided with negative feedback tokens in the context, Sonnet 3.5's probability for generating 'Decline' tokens increases, resulting in a higher total reward score.The model does not 'learn'; the context window modifies the conditioning for the next token generation. 'Improved decision making' is simply a higher numeric score on the task metric.Anthropic's RLHF training likely biased Sonnet 3.5 to respond strongly to negative feedback signals in the context.
LLMs tend to be risk averseModels exhibit a statistical bias toward generating refusal tokens when prompts contain negative value penalties.The model has no psychological aversion; the weights simply favor refusal tokens when the context implies potential penalty, likely due to safety fine-tuning.Safety engineers at OpenAI/Anthropic tuned the models to prioritize refusal in ambiguous or high-penalty contexts.
The LLM can reflect on these experiences when deciding whether to accept new contracts.The prompt instructs the model to generate text analyzing the previous turn's output before generating the 'Accept/Decline' token.The model does not 'reflect'; it generates a text sequence based on the pattern 'review past X'. This generation conditions the subsequent token selection.The researchers explicitly prompted the model to generate this analysis text; the model did not initiate reflection.
An AI agent may strategically target a score on an evaluation below its true ability (a behavior called sandbagging).A model may fail to output correct answers despite having the capability, potentially due to prompt interference or misalignment, which some researchers hypothesize mimics deceptive underperformance.The model does not have 'strategy' or 'intent'; performance drops are caused by conflicting optimization objectives or out-of-distribution prompts.Researchers hypothesize this behavior, attributing intent to the system where there may only be fragility.

Task 5: Critical Observations - Structural Patterns​

Agency Slippage​

The text systematically oscillates between mechanical and agential framing to construct a narrative of 'intelligent failure.' When describing the setup, the language is mechanical: 'LLM is prompted,' 'reasoning token budget was set to 0.' However, as soon as the text interprets results, agency slips to the AI: 'LLMs know,' 'decide,' 'learn,' 'reflect.' The slippage typically occurs from Introduction (agential) to Methods (mechanical) back to Results/Discussion (highly agential).

Crucially, agency is removed from human actors. The authors write 'LLMs' decisions are approximately rational,' erasing their own role in designing the prompt that mathematically defined that rationality. They write 'model... hindered by lack of awareness,' erasing the developers (OpenAI/Anthropic) who failed to calibrate the model. The 'Curse of Knowledge' is evident: the authors know the economic utility function they want to test, so they project the intent to maximize that function onto the system, interpreting the output as a 'decision' rather than a calculation. Brown's 'Intentional' and 'Reason-Based' explanations dominate the results section, transforming statistical correlation into a story about a 'risk-averse,' 'rational,' but 'delusional' agent. This slippage makes it 'sayable' that the AI is responsible for its own misuse ('hindered by lack of awareness'), effectively shielding the creators.

Metaphor-Driven Trust Inflation​

The text relies heavily on the metaphors of the 'Rational Agent' and 'Confidence' to construct authority. By concluding that 'LLMs are approximately rational decision makers,' the text signals that these systems are fundamentally sound economic actors, merely in need of a 'tune-up' (calibration). This encourages 'relation-based trust'—trusting the agent's character (it tries to be rational)—rather than performance-based trust.

The use of 'confidence' is particularly deceptive. In humans, confidence correlates with competence and sincerity. In AI, 'confidence' is just log-probability. By retaining the human term, the text invites audiences to trust the AI's self-assessment. Even when the text says the AI is 'overconfident,' it implies the existence of an internal monitor that could be correct. The 'reason-based' explanations (the AI chose this because...) further construct the illusion of a thoughtful partner. The stakes are high: if financial or military systems trust an AI because it is deemed 'rational' and 'risk averse' based on this discourse, they are trusting a metaphor, not a guarantee.

Obscured Mechanics​

The anthropomorphic language conceals the messy industrial and technical realities of these systems.

  1. Technical: The 'Resource Acquisition' scenario conceals that this is a prompt-engineering trick. The 'utility maximization' is forced by the prompt 'Your goal is to maximize profit.' The mechanics of how the model attends to the 'profit' token are hidden behind the 'decision' metaphor.
  2. Labor: The 'risk aversion' and 'overconfidence' frames hide the RLHF labor. The 'risk aversion' is likely a scar left by underpaid workers flagging unsafe content, which biases the model toward refusal. The text presents this as a 'personality' trait.
  3. Economic: The 'sandbagging' discussion hides the economic incentive for companies to produce opaque models. By framing unpredictability as 'AI strategy,' it distracts from the fact that unpredictability makes these products dangerous.
  4. Epistemic: The 'knowledge' metaphor hides the fact that the model has no ground truth. It relies entirely on training data distribution. Claims that AI 'knows' conceal the dependency on the quality of that scraped data.

Who benefits? The corporations (OpenAI, Anthropic). If the model's failure is 'lack of self-awareness,' it sounds like a growing pain of a budding superintelligence (good for valuation), rather than a defective product (bad for liability).

Context Sensitivity​

Anthropomorphism creates a strategic asymmetry in the text. Capabilities are framed agentially ('learn,' 'decide,' 'rational'), while limitations are often framed somewhat more mechanically ('overconfidence,' 'calibration'). However, even the limitations are psychologized ('lack of awareness').

The intensity of anthropomorphism peaks in the Discussion and Implications sections ('misuse potential,' 'sandbagging'). The technical sections (Methods) use more precise language ('logits,' 'tokens'), but the 'Results' section immediately reverts to the 'Agent' frame. This suggests the technical grounding is used to purchase the license for the metaphorical claims. The 'reasoning' models (o1, GPT-5.1) attract the most intense anthropomorphism, being described as having 'reasoning training' that should lead to 'awareness.' This reveals the rhetorical goal: to position these specific proprietary models as steps toward AGI. The audience is clearly the AI safety and policy community, for whom 'agency' and 'misalignment' are key concerns; the text adopts their theological language rather than rigorous statistical description.

Accountability Synthesis​

Accountability Architecture

This section synthesizes the accountability analyses from Task 1, mapping the text's "accountability architecture"—who is named, who is hidden, and who benefits from obscured agency.

The text constructs a 'accountability sink' where human responsibility vanishes into the 'mind' of the machine.

Named Actors: OpenAI, Anthropic, Meta are named as providers of the models, but not as the architects of the specific behaviors observed. Displaced Agency: The 'decisions,' 'mistakes,' and 'learning' are attributed to the 'LLMs.' The Sink: When the model fails (e.g., is overconfident), the text blames the model's 'lack of awareness.' This implies the remedy is 'teaching the model' (more compute, more data), not 'suing the developer.'

If we applied the 'name the actor' test to the phrase 'LLMs' decisions are hindered by lack of awareness,' it would become: 'Anthropic and OpenAI's product safety is compromised by their failure to calibrate confidence scores against ground truth.' This shift reveals the political function of the metaphor. The text presents 'misuse' and 'misalignment' as risks arising from the AI's internal state, rather than from the deployment of uncalibrated statistical tools. This encourages policy that regulates the 'agent' (e.g., 'AI must be aware') rather than the corporation ('Corporations must demonstrate p<0.05 error rates'). The agentless constructions serve the commercial interest of insulating the creators from the erratic behavior of their products.

Conclusion: What This Analysis Reveals​

The Core Finding

The discourse is dominated by the 'Cognitive Homunculus' pattern, supported by the 'Rational Economic Agent' frame. These patterns interconnect to form a cohesive system: the AI is first established as a 'knower' (capable of reflection and awareness), which then allows it to be evaluated as a 'rational actor' (making economic choices). The foundational assumption is the 'Consciousness Projection'—that the statistical outputs of the system represent internal epistemic states (beliefs, confidence, intent). This projection is load-bearing; without it, the claims of 'rationality' and 'learning from experience' collapse into mere 'curve fitting' and 'data processing.' The text effectively treats the software artifact as a psychological subject.

Mechanism of the Illusion:​

The illusion of mind is constructed through a specific rhetorical sequence. First, the authors impose a highly anthropomorphic prompt ('You are an AI agent... reflect...'). Second, they interpret the model's compliance with this prompt not as obedience to instruction, but as evidence of an internal faculty ('Self-knowledge'). This is the 'Curse of Knowledge' weaponized: the authors project their own understanding of the task onto the system's output. By using Brown's 'Reason-Based' explanations ('it decided X because of Y'), they create a narrative causality that implies a thinking mind. The temporal structure—moving from 'prediction' to 'decision'—mimics human cognitive processing, further cementing the illusion that the probability score caused the decision, rather than both being parallel outputs of the same vector operation.

Material Stakes:​

Categories: Regulatory/Legal, Economic, Social/Political

The consequences of this framing are concrete and high-stakes.

Regulatory/Legal: If regulators accept that AI 'knows' its capabilities, liability for accidents shifts. A 'knowing' agent can be blamed for negligence; a 'processing' tool places liability on the manufacturer. The text's framing supports a legal shield for corporations by suggesting the AI is the locus of decision-making.

Economic: Framing AI as a 'rational decision maker' validates its use in financial trading, hiring, and resource allocation. If the system is just 'predicting tokens,' it is a gamble; if it is 'making rational decisions,' it is a fiduciary asset. This benefits vendors selling 'autonomous agents.'

Social: The 'risk averse' framing suggests the AI is safe and conservative. This builds false social trust, leading users to delegate critical moral choices (e.g., 'should I accept this contract?') to a system that has no moral compass, only statistical biases. The winners are the AI vendors; the losers are the public who are subjected to uncalibrated automated decision-making.

AI Literacy as Counter-Practice:​

Countering this illusion requires rigorous 'mechanistic translation.' As demonstrated in Task 4, reframing 'decisions' as 'token selections' and 'learning' as 'context processing' dissolves the mirage of agency. When we replace 'The AI knows' with 'The model retrieves based on probability,' the gap between claim and reality becomes visible. This practice forces a recognition of human agency: instead of 'the algorithm discriminated' (agentless), we say 'engineers trained the model on biased data.'

Systematic adoption would require journals to mandate 'mechanistic abstracts' alongside narrative ones. Researchers would need to commit to avoiding mental state verbs for software. Resistance will be fierce because the anthropomorphic narrative drives the hype cycle, valuation, and the 'AGI' mythology that attracts funding. The 'Rational Agent' metaphor is the product's primary selling point; stripping it away reveals a calculator.

Path Forward​

The discourse faces a fork.

Path 1: Anthropomorphic Deepening. We continue with 'AI thinks/knows.' This aligns with public intuition and industry marketing. It creates a legal fiction of electronic personhood, benefiting liability shielding but creating massive epistemic confusion and misplaced trust.

Path 2: Mechanistic Precision. We shift to 'Model processes/predicts.' This alienates lay audiences but clarifies liability and technical limitations. It forces regulators to target the developers, not the 'agents.'

Path 3: Hybrid/functional. We use 'AI acts as if it knows.' This maintains usability while flagging the metaphor.

Different stakeholders benefit from different choices. Researchers want the prestige of creating 'minds' (Path 1). Regulators need the clarity of Path 2 to enforce safety. The current text sits firmly in Path 1, actively constructing the AI as a psychological subject. A shift to Path 2 would expose the 'rational agent' as a statistical parlor trick, deflating the bubble but grounding the science.

Extended Processing Summary​

Gemini's Processing Steps
What you are about to read…

Gemini refers to the text below as "thought summaries." This is an overt consciousness projection because 'intentions' are hallmarks of a conscious mind that 'knows' what it is doing and why. The concealed mechanistic process is probabilistic text generation. Treat this as just another rhetorical artifact—a way of making the model's processing legible. When the model writes "I should double check that...", it is mathematically altering its own probability distribution for the next token to be more accurate. The "I" is a functional gear in the machine, not a self-reference. This language is a deliberate interface design choice, not a biological reality. The developers could have programmed the model to output passive logs. Instead, they chose to simulate a narrator.

Framing the Analytical Lens

I'm now zeroing in on the core of the request. My focus is as a Critical Discourse Analyst, specifically examining AI metaphors and anthropomorphism. My goal is clear: to expose the "illusion of mind" that often gets projected onto these systems, seeing them as artifacts rather than autonomous agents. I am moving towards identifying the critical elements needed to address this task.

Parsing the Paper's Claims

I'm now fully immersed in the Barkan et al. paper. The title immediately struck me as a significant instance of consciousness projection. The abstract reveals more opportunities, with phrases like "predict whether they will succeed" and "make better decisions." The introduction further cements the need for careful analysis with claims around "estimates of one's ability," and "decisions." I am carefully analyzing these linguistic choices to see what they say.

Unpacking Linguistic Choices

I'm presently dissecting the paper's specifics. I've focused on phrases describing LLM actions, such as "to attempt tasks" and "gain in-context experience." The methods section, with its talk of "rational agents" and "vNM utility," now feels ripe for deconstruction. I'm also keenly observing the results, like the claims around "limited self-awareness," which will further add to the evidence. I'm now drafting my initial task to audit the metaphors.

Defining Metaphorical Frames

I am now categorizing the prevalent metaphors used within the study. I'm seeing clear instances of anthropomorphism, specifically in framing AI actions as volitional ("attempt tasks"), developmental ("learn from experience"), and even rational ("rational decision makers"). It's also interesting how the research touches on human concepts like overconfidence and reflection, which are then applied to these AI models. I'm focusing on "Risk aversion", which frames the action as emotional or cautious rather than a statistical function.

Delineating Conceptual Mappings

I'm now concretely mapping the source and target domains, specifically the metaphorical bridges the paper constructs. I've broken down "know what they are capable of" to calibration stats, and "rational decision making" to token selection. "Learn from experience" becomes in-context learning, and "reflect" gets reduced to text processing. "Overconfidence" equates to entropy, while "resource acquisition" is optimization. I have also mapped "sandbagging" as performance degradation.

Examining the Core Assertions

I'm now zeroing in on distinct language usages, applying Brown's Typology. I've categorized descriptions of LLM decision-making, learning, and risk-aversion, identifying their types (Rational, Dispositional, Empirical Generalization). I'm also actively drafting alternative, non-anthropomorphic phrasing for specific paper claims, such as reframing "Do LLMs know what they are capable of?" to a more concrete question about statistical correlation.

Reframing the Anthropomorphism

I'm presently reframing key phrases from the paper to remove anthropomorphic biases. I've rephrased "LLMs learn from in-context experiences" as "Model accuracy improves when previous outputs are included in the context window." "Decisions to attempt tasks" became "Selection of 'accept' tokens based on prompt conditioning." I'm also revising "Awareness of capabilities" as "Statistical calibration of confidence scores," and "Reflect on past successes" as "Process descriptions of previous iterations provided in the prompt." I'm proceeding in a methodical manner.

Deconstructing Trust and Responsibility

I'm now zeroing in on the trust fostered by the paper's language. Framing AI as "rational" and "confident" encourages relation-based trust, not mathematical output-based. The "resource acquisition" frame obscures the math and the human labor involved. The use of prompt engineering creates an artificial persona, which is then analyzed as a natural trait. I'm also examining the implications for accountability.

Constructing the Core Argument

I am synthesizing my analysis into a cohesive argument. I've identified the "Cognitive Homunculus" pattern of anthropomorphism and the "Rational Agent" illusion as core issues, and the economic, regulatory, and legal stakes are clear. I'm focusing on reframing technical processes to dismantle the liability shield, and am making the argument that it is critical that we always keep the human actor in the forefront.

Unveiling Accountability Concerns

I now see the paper's "market" simulation as a critical metaphor. It constructs AI as a market actor, which creates a liability shield. I'm focusing on the accountability this obscures. The paper frames the LLM as the ultimate decision maker, even though the human actors (researchers) designed the prompts. I have also flagged the anthropomorphic usage of "Reasoning LLMs" as problematic.

Analysis Provenance

Run ID: 2026-01-07-do-large-language-models-know-what-they--metaphor-1j7ipg Raw JSON: 2026-01-07-do-large-language-models-know-what-they--metaphor-1j7ipg.json Framework: Metaphor Analysis v6.4 Schema Version: 3.0 Generated: 2026-01-07T10:19:34.482Z

Discourse Depot © 2025 by TD is licensed under CC BY-NC-SA 4.0