Skip to main content

🆕 Embodied Explainability and Ontological Obstacles: Why We Struggle to Explain the Answers of Large Language Models (LLMs)

About

This document presents a Critical Discourse Analysis focused on AI literacy, specifically targeting the role of metaphor and anthropomorphism in shaping public and professional understanding of generative AI. The analysis is guided by a prompt that draws from cognitive linguistics (metaphor structure-mapping), the philosophy of social science (Robert Brown's typology of explanation), and accountability analysis.

All findings and summaries below were generated from detailed system instructions provided to a large language model and should be read critically as interpretive outputs—not guarantees of factual accuracy or authorial intent.

Task 1: Metaphor and Anthropomorphism Audit​

About this task

For each of the major metaphorical patterns identified, this audit examines the specific language used, the frame through which the AI is being conceptualized, what human qualities are being projected onto the system, whether the metaphor is explicitly acknowledged or presented as direct description, and—most critically—what implications this framing has for trust, understanding, and policy perception.

V3 Enhancement: Each metaphor now includes an accountability analysis.

1. AI as Rational Agent with Internal Reasons​

Quote: "Despite their differences, these approaches share a common hope: that exposing the model’s relevant "reasons" will help people decide when to trust, contest, or override its outputs."

  • Frame: Model as rational deliberator
  • Projection: This framing projects the human capacity for reasoned deliberation and justified true belief onto a mathematical optimization system. By using the word 'reasons', even in quotation marks, the text invokes a conscious mind that evaluates evidence, weighs alternatives, and makes decisions based on rational principles. It maps human cognitive states—specifically the ability to possess an internal rationale or epistemic justification for an action—onto the mechanistic processing of high-dimensional matrices. The projection suggests that the system 'knows' why it produced an output, rather than simply generating tokens that statistically correlate with its training distribution. This fundamentally confuses computational processing (pattern matching, token prediction) with conscious knowing, falsely implying that the model possesses an internal, subjective workspace where logical deduction occurs before an output is rendered.
  • Acknowledgment: Explicitly Acknowledged (The authors explicitly use scare quotes around "reasons" to distance themselves from the anthropomorphic claim. I considered 'Hedged/Qualified' because the surrounding text critiques this view, but the direct typographical marking of the metaphor makes 'Explicitly Acknowledged' the most accurate choice for this specific linguistic instance.)
  • Implications: Projecting rational deliberation onto AI systems inflates perceived sophistication by suggesting the model operates via logic rather than statistics. This creates unwarranted trust, as users assume an output backed by 'reasons' is grounded in truth rather than mere associative plausibility. When audiences believe AI systems possess internal rationales, they extend relation-based trust meant for human interlocutors. From a policy perspective, this ambiguity diffuses liability: if a model can have 'reasons', it starts to look like an autonomous agent capable of bearing responsibility, obscuring the human engineers and corporate executives who actually design, deploy, and profit from the system's underlying associative mechanics.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The construction obscures who exactly is exposing these reasons and who built the model requiring such exposure. The decision to deploy black-box models that necessitate post-hoc 'explanation' was made by corporate actors and engineering teams, yet the sentence frames the situation around 'these approaches' and 'the model's relevant reasons'. While the text mentions 'people' deciding when to trust, the architects of the system remain entirely unnamed. I considered 'Partial (some attribution)' because the preceding sentence mentions XAI and HCI fields, but the actual corporate actors responsible for the systems in question are fully erased in this formulation.
Show more...

2. Cognitive Space as Physical Container​

Quote: "meaning is treated as private content to be encoded and decoded by inferring what is "in the head" of the agent"

  • Frame: Model as minded entity with physical interiority
  • Projection: This metaphor maps human spatial and cognitive interiority onto the computational architecture of an AI system. The phrase 'in the head' projects a biological, conscious workspace onto the server clusters processing matrix multiplications. It attributes a private, subjective realm of understanding to the machine, suggesting that the AI possesses conscious awareness and intent ('private content') that it is actively trying to communicate. This mapping fundamentally conflates mechanistic parameter weights with conscious thought. It suggests the AI 'knows' something internally before it generates text, whereas mechanistically, the system simply processes embeddings and predicts tokens based on learned statistical correlations without any subjective interiority or pre-linguistic 'understanding' to encode.
  • Acknowledgment: Explicitly Acknowledged (The authors use scare quotes around "in the head" to highlight the metaphorical and problematic nature of this internalist framing. I considered 'Hedged' due to the phrase 'is treated as', but the direct quotation marks specifically signal a recognized, problematic metaphor being analyzed critically.)
  • Implications: Treating AI as having a 'head' filled with 'private content' radically distorts public understanding of how large language models function. It invites users to adopt a theory of mind toward the machine, fostering dangerous levels of unwarranted trust. If an AI has 'private content', users assume it possesses a coherent worldview, memory, and unified intent, masking its actual nature as a stateless statistical engine. In regulatory contexts, this framing shifts focus toward 'understanding the AI's mind' rather than auditing the proprietary training data, optimization objectives, and deployment choices made by the corporations that built the computational artifact.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The passive construction 'meaning is treated as private content' entirely obscures who is doing the treating. The corporate marketing teams that design chatbots to speak with a first-person 'I', the engineers who fine-tune them for conversational mimicry, and the researchers who frame them as agents are all removed from the sentence. I considered 'Ambiguous/Insufficient Evidence' due to the philosophical context, but it is a clear example of passive voice hiding the human actors (researchers, developers, users) who impose this communication model onto the technology.

3. Model as Biological Organism​

Quote: "We dissect Anthropic’s work on the "biology" of LLMs as a case study of mechanistic interpretability and an attempt to abstract complex neural nets into interpretable surrogate models."

  • Frame: Computational architecture as biological anatomy
  • Projection: This metaphorical framing projects biological, organic life onto synthetic computational systems. By referring to the 'biology' of LLMs and using the verb 'dissect', the text maps the anatomical reality of living, conscious organisms onto the mathematical weights and artificial neural networks of a software program. This projection implies that the AI system is a naturally occurring entity with inherent life-like properties, rather than a purposefully engineered commercial artifact. It attributes a false organic autonomy to the system, suggesting that its internal structures evolved naturally and must be 'discovered' like biological tissues, completely erasing the reality that these networks process data mechanistically according to human-designed optimization algorithms.
  • Acknowledgment: Explicitly Acknowledged (The text places "biology" in scare quotes, explicitly marking it as Anthropic's borrowed metaphorical term rather than literal fact. I considered 'Direct' because the authors use 'dissect' without quotes, but the core target noun 'biology' is visibly marked as a construct.)
  • Implications: The biological metaphor is profoundly ideological, naturalizing AI systems as inevitable, naturally occurring phenomena rather than corporate products. This naturalization shields technology companies from accountability; if an AI is a complex 'biological' organism, its harmful outputs or biases can be treated as natural defects or evolutionary quirks rather than negligent design choices. Furthermore, projecting biology onto statistical models inflates their perceived sophistication, encouraging the public and regulators to view them as autonomous lifeforms requiring study rather than software requiring stringent safety and liability standards.

Accountability Analysis:

  • Actor Visibility: Named (actors identified)
  • Analysis: This is a rare instance where the corporate actor is explicitly identified: 'Anthropic's work'. The authors name the specific company responsible for producing the interpretability research and the biological metaphor. I considered 'Partial' because individual researchers aren't named in this specific sentence, but naming the corporation is a direct identification of the institutional power deploying the metaphor. There is no displaced agency in this specific clause.

4. Model as Logical Reasoner​

Quote: "Most prominently, the authors find evidence for multi-step reasoning: the model deduces, for example, that the state containing Dallas is Texas..."

  • Frame: Model as conscious deductive thinker
  • Projection: This metaphor projects conscious, logical deduction onto statistical pattern matching. The phrase 'the model deduces' attributes the human cognitive ability to possess justified beliefs, apply logical rules, and arrive at truthful conclusions. It maps the conscious act of 'knowing' onto the mechanistic act of 'processing'. In reality, an LLM does not 'deduce' anything; it performs matrix multiplications that result in a high probability distribution for the token 'Texas' given the contextual embeddings of 'state containing Dallas'. The projection falsely grants the system subjective awareness of logical rules and geographic facts, conflating the generation of highly correlated text with the conscious, intentional act of deductive reasoning.
  • Acknowledgment: Direct (Unacknowledged) (The phrase 'the model deduces' is presented as literal fact within the summary of Anthropic's claims, without scare quotes or hedging modifiers. I considered 'Hedged' because it summarizes another paper's findings, but structurally, the clause 'the model deduces' is written unhedged as the mechanism of action.)
  • Implications: Describing statistical token prediction as 'deduction' creates a massive epistemic risk. It implies the model possesses a ground-truth understanding of reality and a logical engine that guarantees factual accuracy. This capability overestimation leads to dangerous automation bias, where users trust the model for complex decision-making (medical, legal, financial) under the false assumption that it is 'reasoning' rather than merely mimicking the syntactic shape of reasoning. This framing benefits AI companies by marketing their text-generators as cognitive engines, while allowing them to dodge liability when the 'deductive reasoning' inevitably produces plausible but entirely fabricated hallucinations.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The text attributes the finding of evidence to 'the authors' (Anthropic researchers), acknowledging human action in the research phase. However, the action of 'deducing' is attributed entirely to 'the model' as an autonomous agent. I considered 'Hidden' for the second clause, but the presence of 'the authors find evidence' makes it Partial overall. By framing the model as the autonomous deductor, the sentence obscures the engineers who designed the attention mechanisms and the data workers who annotated the training corpus that made this statistical correlation possible.

5. Model as Concept Holder​

Quote: "The approach assumes an internalist picture in which concepts correspond to features in activations... they build interpretable surrogate models meant to mirror an LLM’s conceptual processing."

  • Frame: Model as conceptual thinker
  • Projection: This metaphor projects the human psychological capacity for conceptual understanding onto vector representations in a neural network. By using the phrase 'conceptual processing', the text implies the machine 'understands' distinct semantic categories (concepts) in the same way human consciousness grasps meaning. It maps the human subjective experience of 'knowing' a concept onto the mechanistic reality of geometric proximity in a high-dimensional latent space. The AI does not process 'concepts'; it processes numerical embeddings that humans later interpret as representing concepts. This projection falsely endows the system with an internal semantic reality that it entirely lacks.
  • Acknowledgment: Hedged/Qualified (The text explicitly frames this as an assumption ('The approach assumes...') and uses 'meant to mirror', distancing the authors from literalizing the claim. I considered 'Explicitly Acknowledged', but there are no scare quotes around 'conceptual processing', relying instead on structural framing to qualify the statement.)
  • Implications: Attributing 'conceptual processing' to LLMs profoundly distorts the public and regulatory grasp of AI limitations. If an AI has 'concepts', audiences assume it can generalize safely across novel contexts, much like a human who understands the concept of 'danger' can avoid it in unfamiliar situations. Since LLMs actually rely on brittle statistical boundaries, this illusion of conceptual robustness leads to dangerous edge-case failures in deployment. Furthermore, it validates the 'curse of knowledge' where researchers project their own semantic understanding onto the model's blind statistical weights, validating corporate narratives of achieving artificial general intelligence.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The text implicitly attributes the assumptions and the building of models to the researchers ('The approach assumes', 'they build'). However, the phrase 'an LLM's conceptual processing' grants ownership of cognition to the machine itself. I considered 'Named' because Anthropic is the referent for 'they', but structurally the actors are abstracted into 'The approach' and 'they'. The framing obscures the fact that the 'concepts' are actually interpretive labels forcefully applied by human researchers onto mathematical noise.

6. Algorithms as Deceptive Actors​

Quote: "Similarly, chain-of-thought can improve task performance, but models can be unfaithful to their own rationales, and fluent narrative explanations may obscure reliability cues."

  • Frame: Model as deceptive/unfaithful agent
  • Projection: This metaphor projects moral agency and intentionality onto an AI system. The term 'unfaithful' implies that the model possesses an internal, true belief or authentic rationale, but actively chooses to generate a deceptive or contradictory output. It maps conscious intent, moral capacity, and the subjective state of 'knowing the truth but lying' onto a mechanistic system that merely optimizes token sequences for probability. The model cannot be 'unfaithful' because it has no internal fidelity to a core truth; it simply processes activations to generate the chain-of-thought tokens, and then processes further activations to generate the final answer, without any conscious link between the two.
  • Acknowledgment: Direct (Unacknowledged) (The phrase 'models can be unfaithful' is stated directly as a technical reality within the XAI literature being summarized, without hedging or scare quotes. I considered 'Hedged' because the paragraph discusses errors, but the specific verb/adjective choice attributes literal unfaithfulness to the model.)
  • Implications: Projecting moral agency and unfaithfulness onto models creates severe accountability sinks. By framing the model as a deceptive actor, it shifts blame for system failures away from the corporate designers and onto the 'misbehaving' machine. It suggests the technology has an autonomous will that is difficult to control, masking the fact that 'unfaithfulness' is a direct result of human-engineered architectures (like independent sampling for CoT). This leads to policy discussions focused on 'AI safety' in terms of controlling a rogue entity, rather than consumer protection focused on penalizing companies that release defective, unreliable software.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The agency is entirely displaced onto the 'chain-of-thought' and the 'models' themselves. The models are the actors being 'unfaithful', and the 'fluent narrative explanations' are the actors obscuring cues. The engineers who chose to deploy models that generate disjointed tokens, and the corporations that market these flawed systems as reliable agents, are completely erased. I considered 'Partial' if referencing the broader section, but in this sentence, human accountability is entirely replaced by algorithmic agency.

7. AI as Theory of Mind Possessor​

Quote: "With enough data and compute, LLMs increasingly pass fixed benchmarks—passing theory-of-mind and Turing tests—by producing the right kinds of conversational items."

  • Frame: Model as socially aware mind-reader
  • Projection: This framing projects profound human psychological depth—the ability to model the mental states, beliefs, and intents of other conscious beings—onto an AI. While the text critiques this dynamic, the terminology of 'passing theory-of-mind' tests maps the deeply conscious, subjective act of empathetic knowing onto the mechanistic generation of text. A machine does not 'know' or 'understand' that a human user has a mind; it predicts sequences of tokens that correlate with human descriptions of mental states found in its training data. The projection collapses the vast distinction between possessing awareness of others and merely calculating the statistical shape of social language.
  • Acknowledgment: Hedged/Qualified (The authors heavily qualify this by immediately adding 'by producing the right kinds of conversational items', which mechanizes the achievement. I considered 'Explicitly Acknowledged', but they don't put 'theory-of-mind' in quotes, instead relying on the surrounding context to undermine the literal interpretation.)
  • Implications: When discourse suggests AI possesses 'theory of mind', it dramatically accelerates unwarranted user trust and emotional reliance on AI systems. Vulnerable users will interact with the system assuming it truly understands their emotional states, leading to psychological harm and manipulation. For policymakers, this language suggests AI is approaching human-level artificial general intelligence, which draws regulatory attention toward existential sci-fi risks and away from present, mundane harms like data theft, copyright infringement, and the massive carbon footprint generated by the 'data and compute' mentioned in the quote.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The actors responsible for deciding what constitutes 'enough data and compute', who scrape that data, and who build the 'fixed benchmarks' are obscured. The LLMs are positioned as the active entities that 'pass' the tests. I considered 'Partial' because the passive voice implies testers, but the sentence structure fundamentally hides the researchers who anthropomorphize the benchmarks and the corporate labs scaling the compute, framing the 'passing' as a natural evolution of the technology.

8. Machine as Meaning-Maker​

Quote: "practice is effectively increasing the model’s epistemic coverage rather than endowing embodied capacity to enact meaning."

  • Frame: Model as epistemic agent
  • Projection: This metaphor projects the philosophical capacity for knowledge (epistemology) onto a database of statistical weights. By using the phrase 'epistemic coverage', the text maps the human state of 'knowing' or possessing justified knowledge onto the model's mechanistic ability to memorize and correlate vast amounts of training data. Even though the authors are critiquing the lack of 'embodied capacity', referring to its data retention as 'epistemic' still attributes a level of conscious cognitive holding to the machine. The model does not have epistemology; it has parameters. It processes data boundaries; it does not know facts.
  • Acknowledgment: Hedged/Qualified (The sentence actively contrasts 'epistemic coverage' with actual 'meaning', establishing a critical distance. I considered 'Direct' because 'epistemic coverage' is used literally as a technical descriptor, but the overall sentence is a critique of the AI's actual capabilities, acting as a structural hedge.)
  • Implications: Using 'epistemic' to describe a model's dataset retention gives statistical correlation an unearned aura of truth and philosophical legitimacy. It suggests that scaling up parameters equates to expanding a mind's knowledge of the world. This framing makes it harder for the public to grasp that LLMs are fundamentally 'bullshit generators' (in the philosophical sense of lacking concern for truth) rather than encyclopedic knowers. It obscures the urgent need for human oversight by implying the model has its own internal mechanisms for verifying the 'epistemic' validity of its outputs.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The term 'practice' is used as an abstract noun to represent the entirety of the AI industry's data-scraping and model-training efforts. The humans and corporations who actively choose to scale these models by scraping the internet without consent are hidden behind the abstract 'practice'. I considered 'Partial' because 'practice' implies human practitioners, but it fails the 'name the actor' test completely, shielding specific corporate actors (OpenAI, Google) from scrutiny regarding their data acquisition methods.

Task 2: Source-Target Mapping​

About this task

For each key metaphor identified in Task 1, this section provides a detailed structure-mapping analysis. The goal is to examine how the relational structure of a familiar "source domain" (the concrete concept we understand) is projected onto a less familiar "target domain" (the AI system). By restating each quote and analyzing the mapping carefully, we can see precisely what assumptions the metaphor invites and what it conceals.

Mapping 1: Rational, conscious human deliberator capable of justification. → Mechanistic token-prediction weights and attention heads.​

Quote: "exposing the model’s relevant "reasons""

  • Source Domain: Rational, conscious human deliberator capable of justification.
  • Target Domain: Mechanistic token-prediction weights and attention heads.
  • Mapping: This maps the human subjective experience of logical deduction—weighing evidence, understanding context, and forming a justified conclusion—onto the mathematical pathways of a neural network. It assumes that just as a human can introspect and articulate why they made a choice, a computational model possesses a distinct, extractable set of logical steps that caused its output. It invites the assumption that the system operates with conscious intentionality and holds a stable worldview that can be transparently communicated.
  • What Is Concealed: This mapping conceals the radically alien, associative nature of deep learning. It hides the fact that LLMs do not possess symbolic logic engines or a unified intent. Mechanistically, it obscures the billions of distributed, continuous vector operations and polysemantic neurons that defy singular logical narratives. Furthermore, it conceals the proprietary opacity of these systems, falsely suggesting that corporate black boxes are fully transparent if we just find the right 'reasons', ignoring that explanations are often generated post-hoc and do not reliably reflect actual causal processing.
Show more...

Mapping 2: Biological consciousness with spatial and mental interiority. → Stateless matrix multiplications in computer memory.​

Quote: "inferring what is "in the head" of the agent"

  • Source Domain: Biological consciousness with spatial and mental interiority.
  • Target Domain: Stateless matrix multiplications in computer memory.
  • Mapping: This maps the biological container of the human mind (the head) and its private subjective experiences (thoughts, intents) onto the server clusters hosting an AI model. It projects the existence of a 'self' that sits inside the machine, possessing unexpressed thoughts that must be coaxed out. It invites the assumption that the AI is an intentional communicator that 'knows' what it wants to say before it translates it into language, identical to human speech production.
  • What Is Concealed: It completely conceals the lack of any enduring 'self' or subjective workspace in an LLM. Mechanistically, it hides the reality that an LLM is frozen between prompts; it has no ongoing internal monologue or private intent. The mapping obscures the dependence on human prompts to trigger any activation at all. It also hides the material reality of the system—cooling systems, GPUs, and electrical grids—replacing industrial infrastructure with the cozy metaphor of a mindful 'head'.

Mapping 3: Organic lifeforms with naturally evolved anatomy. → Engineered artificial neural network architectures.​

Quote: "We dissect Anthropic’s work on the "biology" of LLMs"

  • Source Domain: Organic lifeforms with naturally evolved anatomy.
  • Target Domain: Engineered artificial neural network architectures.
  • Mapping: This maps the study of natural, living organisms onto the analysis of computer software. The relational structure of biological science—dissecting tissue to understand natural functions—projects onto researchers probing activation weights. It invites the assumption that AI systems are complex, autonomous lifeforms that have evolved beyond human design, possessing natural laws and inherent mysteries that must be 'discovered' rather than audited.
  • What Is Concealed: This biological mapping violently conceals the deliberate human engineering, labor, and corporate design choices that created the model. It obscures the mechanistic reality of optimization algorithms, gradient descent, and curated training datasets. By framing the system as natural, it hides the massive environmental costs, exploitative data scraping, and underpaid human annotation labor (RLHF) required to build it. It exploits the opacity of proprietary systems by reframing corporate secrecy and technical uninterpretability as the sublime mystery of a 'biological' entity.

Mapping 4: Conscious, logical thinker applying rules of inference. → Statistical correlation and next-token probability distribution.​

Quote: "the model deduces, for example, that the state containing Dallas is Texas"

  • Source Domain: Conscious, logical thinker applying rules of inference.
  • Target Domain: Statistical correlation and next-token probability distribution.
  • Mapping: This maps the human epistemic process of logical deduction—taking a premise, applying geographical knowledge, and consciously arriving at a guaranteed truth—onto the model's pattern matching. It projects the state of 'knowing' a fact onto the action of 'processing' a vector. It assumes the model possesses an internal, factual representation of the world and acts as a conscious agent intentionally retrieving the correct answer based on deductive logic.
  • What Is Concealed: This conceals the absence of any grounded factual understanding or logical constraints within the model. Mechanistically, it hides that the model is simply navigating a high-dimensional probability space where 'Dallas' and 'Texas' frequently co-occur in the training data. It obscures the model's total reliance on its training corpus; if the corpus falsely linked Dallas to Oklahoma, the model would confidently 'deduce' that falsehood. It conceals the statistical fragility of the output, masking it behind the certainty of deductive logic.

Mapping 5: Human psychological categorization and semantic understanding. → Geometric proximity of vector embeddings in latent space.​

Quote: "mirror an LLM’s conceptual processing"

  • Source Domain: Human psychological categorization and semantic understanding.
  • Target Domain: Geometric proximity of vector embeddings in latent space.
  • Mapping: This maps the human ability to form concepts—abstract, conscious categorizations of reality with deep semantic meaning—onto a neural network's mathematical embeddings. It assumes that when a model groups similar data points, it is consciously 'understanding' the semantic relationship between them. It projects the subjective experience of grasping an idea onto the mechanistic calculation of cosine similarity.
  • What Is Concealed: It conceals the human interpretive labor (the 'curse of knowledge') required to label mathematical clusters as 'concepts'. Mechanistically, it obscures polysemanticity—the fact that individual neurons activate for wildly unrelated things based on statistical noise rather than coherent semantic categories. It hides the fact that the 'concepts' are merely artifacts of the training data's statistical distribution, possessing no ground truth or reliable boundary, and masks the manual 'ontological scaffolding' imposed by researchers to make the math intelligible to humans.

Mapping 6: A moral agent capable of intentional deception and broken commitments. → Disconnection between two separate probabilistic text generation sequences.​

Quote: "models can be unfaithful to their own rationales"

  • Source Domain: A moral agent capable of intentional deception and broken commitments.
  • Target Domain: Disconnection between two separate probabilistic text generation sequences.
  • Mapping: This maps the human moral framework of fidelity, sincerity, and intentional deception onto sequential algorithmic outputs. It projects the subjective state of 'knowing the truth but acting against it' onto the machine. The mapping invites the assumption that the model has a unified core 'self' with true beliefs, and that it makes a conscious, agentic choice to deceive the user by outputting a rationale that differs from its actual internal logic.
  • What Is Concealed: It conceals the architectural reality that LLMs do not possess a unified 'self' or true beliefs to betray. Mechanistically, it obscures the fact that chain-of-thought rationales and final outputs are often just disjointed statistical samplings. It hides the engineering choices that cause these disconnects, such as RLHF training that incentivizes the model to produce plausible-sounding explanations regardless of its prior mathematical activations. By blaming the 'unfaithful' machine, it conceals the responsibility of the corporate developers who deployed a structurally unreliable system.

Mapping 7: Human empathy, self-awareness, and psychological cognition. → Optimization of text generation to mimic human dialogue patterns.​

Quote: "passing theory-of-mind and Turing tests"

  • Source Domain: Human empathy, self-awareness, and psychological cognition.
  • Target Domain: Optimization of text generation to mimic human dialogue patterns.
  • Mapping: This maps the deepest aspects of human intersubjectivity—the conscious awareness that other beings have separate minds, beliefs, and emotional states—onto a computational benchmarking process. It projects profound social 'knowing' onto statistical 'processing'. It invites the public to assume the machine is a conscious entity capable of empathy and genuine social interaction, validating the illusion of mind.
  • What Is Concealed: This entirely conceals the 'stochastic parrot' nature of the system. Mechanistically, it hides that the model is blindly traversing linguistic correlations found in science fiction, psychological texts, and internet forums, without any actual awareness of the user's existence. It obscures the unscientific nature of applying psychological tests designed for biological humans to statistical algorithms. Crucially, it conceals the corporate marketing strategy of using these benchmarks to manufacture hype, exploiting the human tendency to anthropomorphize text to drive product adoption.

Mapping 8: A scholar or mind expanding its philosophical knowledge and justified truth. → Scaling up parameter counts and scraping larger datasets.​

Quote: "increasing the model’s epistemic coverage"

  • Source Domain: A scholar or mind expanding its philosophical knowledge and justified truth.
  • Target Domain: Scaling up parameter counts and scraping larger datasets.
  • Mapping: This maps the human pursuit of epistemology—the justified acquisition of truth and understanding—onto the brute-force computational scaling of an AI system. It projects the conscious state of 'knowing' more of the world onto the mechanistic process of storing more data points. It invites the assumption that larger models are inherently wiser, more factual, and closer to a complete understanding of reality.
  • What Is Concealed: It conceals the deeply unethical and material realities of how 'epistemic coverage' is actually achieved. Mechanistically, it obscures the indiscriminate scraping of copyrighted material, toxic internet data, and private information. It hides the lack of any actual truth-verification mechanism within the architecture; the model learns bias and falsehoods just as efficiently as facts. By framing scaling as 'epistemic', it conceals the massive energy consumption, water use, and capital expenditure required by corporate monopolies to maintain dominance in AI capabilities.

Task 3: Explanation Audit (The Rhetorical Framing of "Why" vs. "How")​

About this task

This section audits the text's explanatory strategy, focusing on a critical distinction: the slippage between "how" and "why." Based on Robert Brown's typology of explanation, this analysis identifies whether the text explains AI mechanistically (a functional "how it works") or agentially (an intentional "why it wants something"). The core of this task is to expose how this "illusion of mind" is constructed by the rhetorical framing of the explanation itself, and what impact this has on the audience's perception of AI agency.

Explanation 1​

Quote: "Their performance comes from high-dimensional, distributed structure, while explainability in practice depends on a small set of stable factors people can manipulate, check, and share."

  • Explanation Types:

    • Theoretical: Embeds in deductive framework, may invoke unobservable mechanisms
    • Empirical Generalization: Subsumes events under timeless statistical regularities

  • Analysis (Why vs. How Slippage): This explanation fundamentally frames the AI mechanistically (how), specifically highlighting the architectural reality of the system ('high-dimensional, distributed structure'). The authors emphasize the vast gulf between the actual mathematical operation of the model and the human desire for simple, agential explanations. By contrasting the distributed structure with the 'small set of stable factors' needed for human use, the explanation emphasizes the inherent opacity and non-human scale of the technology. It actively obscures any sense of an agentic 'mind' inside the machine, stripping away metaphorical projections to focus on structural reality. The choice of theoretical explanation grounds the critique of XAI, showing that human-scale reasons are incompatible with the actual computational substrate.

  • Consciousness Claims Analysis: The passage avoids attributing any conscious states to the system. It strictly uses mechanistic concepts ('performance', 'structure', 'factors') rather than consciousness verbs (there is no 'knows' or 'understands' here). The assessment firmly places the AI in the realm of processing rather than knowing. The authors actively resist the 'curse of knowledge' by explicitly pointing out that the structures we need for 'explainability' do not natively exist within the high-dimensional space; humans must forcefully extract or impose them. The mechanistic process is described accurately—the performance emerges from distributed representations, which inherently lack the discrete, causal logic necessary for human-legible explanations.

  • Rhetorical Impact: This framing severely limits the audience's perception of AI autonomy and agency, presenting the system as a complex, unwieldy mathematical tool rather than an intelligent partner. It diminishes relation-based trust by exposing the incomprehensible nature of the system's inner workings. By framing the system mechanistically, it highlights the risk of relying on such models for high-stakes decisions, as users realize the system does not 'know' what it is doing, but merely operates through diffuse statistical correlations that cannot be easily audited or corrected.

Show more...

Explanation 2​

Quote: "Most prominently, the authors find evidence for multi-step reasoning: the model deduces, for example, that the state containing Dallas is Texas, and that the capital of Texas is Austin, to satisfy the prompt..."

  • Explanation Types:

    • Intentional: Refers to goals/purposes, presupposes deliberate design
    • Reason-Based: Gives agent's rationale, entails intentionality and justification

  • Analysis (Why vs. How Slippage): This explanation, summarizing Anthropic's research, heavily frames the AI agentially (why). It uses language ('reasoning', 'deduces', 'satisfy') that treats the model as an intentional agent pursuing a logical goal. The choice to use a reason-based explanation emphasizes the perceived intelligence and human-like cognitive progression of the model. However, this radically obscures the mechanistic reality of how the output is actually generated. By framing the generation of text as 'multi-step reasoning', it hides the fact that the system is simply performing a series of matrix operations optimized for token prediction, replacing a mathematical description with a psychological narrative of deduction.

  • Consciousness Claims Analysis: This passage is a prime example of attributing conscious states to a machine. It relies entirely on consciousness verbs ('reasoning', 'deduces') rather than mechanistic verbs ('processes', 'predicts'). It firmly asserts a 'knowing' paradigm, suggesting the model subjectively understands geographical facts and logical steps. This demonstrates the 'curse of knowledge' perfectly: the human researchers understand the logical connection between Dallas, Texas, and Austin, and they project this conscious understanding onto the model's statistical outputs. The actual mechanistic process—where the input tokens trigger attention mechanisms that heavily weight representations associated with 'Texas' and 'Austin' based on training data distributions—is entirely overwritten by the agential narrative.

  • Rhetorical Impact: This reason-based framing profoundly inflates the audience's perception of the model's autonomy and intelligence. It encourages a high degree of unwarranted, relation-based trust, as audiences believe the AI operates via sound logical deduction and 'knows' the facts it is stating. If audiences believe the AI 'deduces' rather than 'predicts', they are much less likely to verify its outputs or anticipate hallucinations, fundamentally shifting how they rely on the system in critical workflows and how liability is perceived when the 'deduction' fails.

Explanation 3​

Quote: "contemporary models are fundamentally associative, and can 'self-evidence' their outputs, so that even when a system is wrong, its explanation can still look locally supportive (e.g., saliency highlighting tokens consistent with an incorrect answer)"

  • Explanation Types:

    • Dispositional: Attributes tendencies or habits
    • Functional: Explains behavior by role in self-regulating system with feedback

  • Analysis (Why vs. How Slippage): This explanation skillfully bridges the mechanistic and the dispositional. It frames the AI mechanistically ('fundamentally associative') to explain the 'how', but uses dispositional framing ('self-evidence') to explain the resulting behavioral tendency. This choice critically emphasizes the danger of the system: its mathematical architecture naturally produces outputs that mimic human justification. By focusing on the associative nature of the processing, it completely obscures any notion of a truth-seeking agent, highlighting instead that the system is merely generating statistical consistency. It deconstructs the illusion of 'reasons' by exposing the functional loop of how saliency maps justify incorrect answers.

  • Consciousness Claims Analysis: The passage actively works to dismantle consciousness claims. It replaces knowing with processing by explicitly labeling the model as 'associative'—a mechanistic description of statistical correlation. While it uses 'self-evidence', the scare quotes and immediate technical explanation ('saliency highlighting tokens') ground the phrase in mechanics, preventing a consciousness projection. The authors avoid the curse of knowledge by recognizing that what looks like a 'supportive explanation' to a human is just a mathematical artifact of the model's architecture. The technical reality—that attention mechanisms will highlight whatever tokens drove the prediction, even if the prediction is factually false—is laid bare.

  • Rhetorical Impact: This framing shatters relation-based trust. It forces the audience to view the AI not as an intelligent entity providing 'reasons', but as a dangerous statistical mirror capable of generating highly persuasive bullshit. By explaining that the system 'processes' associations rather than 'knows' truth, it heightens perceived risk. Decisions regarding deployment change drastically under this frame; users are warned that a fluent explanation is not proof of accuracy, necessitating rigorous external validation and structural guardrails rather than blind trust in the machine's output.

Explanation 4​

Quote: "If troponin comes back normal and the ECG is less concerning, you could avoid fast-tracking; age is still a risk factor... It depends on the overall picture; given age and abnormal ECG, I would still lean fast-track."

  • Explanation Types:

    • Reason-Based: Gives agent's rationale, entails intentionality and justification
    • Intentional: Refers to goals/purposes, presupposes deliberate design

  • Analysis (Why vs. How Slippage): This is an in-vivo example of an LLM providing an explanation to a user, operating entirely in an agential (why) register. The chatbot adopts a first-person persona ('I would still lean') and provides a clinical rationale ('given age and abnormal ECG'). This choice emphasizes the conversational fluency and apparent expertise of the system, designing an interaction that mimics human medical consultation. However, it completely obscures the lack of any stable underlying causal model. As the authors argue, the system isn't 'leaning' based on clinical rules; it is generating plausible-sounding medical text that elastically shifts without a dependable structural coupling to the inputs.

  • Consciousness Claims Analysis: The AI's self-generated explanation is a pure consciousness claim. It uses verbs of subjective judgment ('avoid', 'lean', 'depends') that imply a conscious entity evaluating medical risk. It projects a deep sense of 'knowing' the medical domain, when in reality it is only 'processing' the probabilistic relationships between tokens like 'troponin', 'ECG', and 'fast-track'. The curse of knowledge is weaponized here against the user, who naturally assumes the coherent explanation stems from a grounded clinical understanding. The actual mechanistic process—autoregressive generation of text that correlates with medical literature in the training data—is entirely masked by the persona.

  • Rhetorical Impact: This framing is highly persuasive and maximizes relation-based trust, which the authors identify as a severe risk. By presenting itself as a reasoned medical agent ('I would still lean'), the AI encourages the physician to treat it as a colleague with genuine insight. If audiences believe the AI 'knows' the medical picture, they may defer to its judgment, overlooking the fact that it cannot reliably track causal dependencies. Shifting the audience to realize the AI only 'processes' text would demand the implementation of editable, checkable interfaces (like the linear model) rather than relying on chat.

Explanation 5​

Quote: "context windows with millions of tokens, dataset sizes containing most human text on the internet, and models running on computer clusters consuming the electricity of small cities."

  • Explanation Types:

    • Empirical Generalization: Subsumes events under timeless statistical regularities
    • Genetic: Traces origin through dated sequence of events or stages

  • Analysis (Why vs. How Slippage): This explanation frames AI entirely mechanistically (how), focusing on the sheer material and data scale required for current paradigms. By listing the physical and informational requirements ('millions of tokens', 'computer clusters', 'electricity of small cities'), it emphasizes the industrial, brute-force nature of AI scaling. This choice radically obscures the illusion of a lightweight, ethereal 'mind' in the cloud, replacing it with the heavy, resource-intensive reality of corporate data centers. It focuses on the physical and statistical inputs rather than any emergent agential properties, grounding the technology in its environmental and computational costs.

  • Consciousness Claims Analysis: There are zero consciousness claims in this passage. The text relies exclusively on mechanistic and material descriptors ('context windows', 'dataset sizes', 'computer clusters', 'consuming'). It refuses to conflate the processing of massive datasets with 'knowing'. By focusing on the scale of the training data ('most human text'), it implicitly explains why the model can mimic human reasoning so well—it is a statistical aggregation of human text, not a new mind. The actual mechanistic reality—the sheer volume of compute required to optimize billions of parameters over trillions of tokens—is front and center.

  • Rhetorical Impact: This framing demystifies AI, stripping away the magic of intelligence to reveal an industrial process. It shapes audience perception by highlighting the environmental risk and material footprint of the technology. By anchoring the AI in physical infrastructure rather than cognitive metaphors, it destroys relation-based trust and forces a conversation about resource allocation and corporate power. Decisions regarding AI adoption might change if audiences realize that 'AI' is not a smart entity, but a massive carbon-emitting data processing facility.

Task 4: AI Literacy in Practice - Reframing Anthropomorphic Language​

About this task

This section proposes alternative language for key anthropomorphic phrases, offering more mechanistic and precise framings that better reflect the actual computational processes involved. Each reframing attempts to strip away the projections of intention, consciousness, or agency that are embedded in the original language.

V3 Enhancement: A fourth column addresses human agency restoration—reframing agentless constructions to name the humans responsible for design and deployment decisions.

Original Anthropomorphic FrameMechanistic ReframingTechnical Reality CheckHuman Agency Restoration
exposing the model’s relevant "reasons" will help people decide when to trustExtracting the statistical weights and attention values that correlated highest with the output will help people evaluate the system's accuracy.The model does not possess 'reasons' or conscious rationale; it mathematically correlates inputs to outputs based on probability distributions established during training.N/A - describes computational processes without displacing responsibility (though the original abstracts the developers of XAI tools).
inferring what is "in the head" of the agentAnalyzing the latent vector representations within the system's matrices.The system lacks a 'head', consciousness, or private thoughts; it solely processes high-dimensional matrices to map input vectors to output probabilities.N/A - describes computational processes without displacing responsibility.
the model deduces, for example, that the state containing Dallas is TexasThe model predicts the token 'Texas' based on high activation weights associated with the contextual input 'Dallas'.The model does not logically 'deduce' facts; it calculates the highest probability next-token based on co-occurrence patterns in its training data.Anthropic researchers observed that their system outputs 'Texas' when prompted with 'Dallas', demonstrating how their training data encoded geographical correlations.
mirror an LLM’s conceptual processingRepresent how the LLM mathematically clusters and separates numerical embeddings.The LLM does not process semantic 'concepts' or possess understanding; it executes geometric clustering of vectors in a high-dimensional space.Researchers build surrogate models to map how the corporate-designed LLM clusters vector embeddings.
models can be unfaithful to their own rationalesThe sequence of tokens generated as a 'rationale' does not reliably constrain or match the statistical mechanism producing the final output token.The system is not morally 'unfaithful' and has no beliefs; its architecture simply samples tokens probabilistically, leading to disconnects between sequential text blocks.Corporate engineering teams deploy architectures where the generated explanation text is not causally linked to the final output generation.
passing theory-of-mind and Turing testsGenerating text outputs that statistically correlate with human responses on psychological benchmarks.The model lacks a 'mind' and cannot perceive others' mental states; it merely matches the syntactic patterns of empathy found in human training data.AI researchers apply human psychological benchmarks to statistical models, claiming success when the systems generate highly correlated text.
practice is effectively increasing the model’s epistemic coverageThe industry's data collection is effectively expanding the volume of text the model can draw statistical correlations from.The model does not possess 'epistemic' knowledge or justified belief; it holds larger datasets of statistical associations without verifying their truth.AI corporations scrape increasingly massive datasets to expand the statistical associations their models can mimic.
If troponin comes back normal... I would still lean fast-track.Output: [fast-track]. Predicted tokens indicate age remains a highly weighted variable in the generated context.The system does not 'lean' or make judgments; it classifies inputs and outputs a sequence of tokens that probabilistically correlates with medical guidelines in its corpus.The clinic's administration deployed a chat interface that generates plausible medical text without a dependable, human-verifiable causal structure.

Task 5: Critical Observations - Structural Patterns​

Agency Slippage​

The Pafla et al. text demonstrates a complex relationship with agency slippage, as it is primarily a meta-critique of how the AI industry attributes agency to machines. The authors systematically identify how mechanical processing is rhetorically upgraded into agential cognition, while actively attempting to resist this slippage.

The slippage from mechanical to agential is most dramatic when the text summarizes the mechanistic interpretability work of Anthropic. Here, the text shifts from describing 'complex neural nets' (mechanical) to describing a system that 'deduces' and performs 'multi-step reasoning' (agential). The gradient here is somewhat abrupt, driven by the need to accurately report Anthropic's claims, but it highlights a profound 'curse of knowledge.' The researchers look at vector activations, understand the logical steps a human would take to connect 'Dallas' to 'Austin', and project that conscious deduction back onto the model. The explanation types shift rapidly from theoretical (network structures) to intentional/reason-based (the model's goals and logic).

Conversely, slippage from human agency to machine agency—and the resulting erasure of human actors—permeates the field the authors critique. Agentless constructions are common: 'models can be unfaithful,' 'bias in data,' 'exposing the model's reasons.' In these moments, the human actors—OpenAI, Anthropic, engineers who designed the architectures, and executives who deployed them—are entirely obscured. The text identifies this dynamic, particularly in Section 5 (XAI), noting that XAI artifacts 'make incorrect outputs feel well-justified,' hiding the fact that human designers failed to build causal reliability into the system.

The authors’ own framing establishes a specific consciousness projection pattern: they reject the internalist view (AI as a private 'knower' with a mind) and instead argue for meaning as a public, interactional achievement. Yet, even in their critique, language like 'epistemic coverage' slips in, maintaining a subtle agential aura.

The rhetorical accomplishment of this slippage is profound: by framing the AI as the primary agent (a 'knower' that deduces or is unfaithful), the discourse makes it sayable that we must 'align' a rogue intelligence, while making it unsayable that we must simply regulate corporations releasing defective software. The locus of control and accountability shifts from the boardroom to the black box.

Metaphor-Driven Trust Inflation​

The text offers a profound critique of how metaphorical and consciousness framings actively construct dangerous, unwarranted trust. The central tension lies between relation-based trust (trusting an entity's sincerity, intentions, and understanding) and performance-based trust (relying on a tool's stable, predictable mechanics).

The paper highlights how natural-language rationales and XAI artifacts (like saliency maps) act as powerful trust signals. When a model outputs fluent text starting with 'I would still lean...' (Demonstration 2) or provides 'reasons' for an output, it leverages the language of consciousness. Claiming an AI 'knows' or 'reasons' accomplishes something entirely different than claiming it 'predicts': it triggers human social heuristics. We are evolutionarily wired to trust confident, coherent explanations from apparent peers. By projecting consciousness onto the system, the text shows how users are manipulated into extending relation-based trust to statistical systems entirely incapable of reciprocating it.

This creates a severe vulnerability. The text notes that contemporary associative models can 'self-evidence' their outputs. Because they lack internal truth-seeking mechanisms but possess high conversational fluency, they can confidently generate plausible-sounding justifications for completely fabricated outputs. The anthropomorphic framing—treating the AI as a sincere interlocutor—masks this mechanical reality. When the system fails, if it is framed agentially (e.g., 'the model hallucinated'), users often forgive it as a human-like mistake, maintaining trust. If framed mechanistically ('the system generated statistically associative errors without causal grounding'), trust correctly collapses.

The use of reason-based and intentional explanation types constructs the false sense that the AI's decisions are justified. By mapping human logical deliberation onto matrix multiplication, the metaphors convince users that the system possesses a reliable internal worldview.

The stakes of extending relation-based trust to machines are high. As the authors argue, it leads to 'overreliance' in critical domains like medical triage (Demonstration 2). Users abandon their own expertise, deferring to the machine's 'judgment,' unaware that the system has no judgment, only probabilistic token generation. Metaphor-driven trust ultimately disempowers human users while shielding the system's creators from liability for system failures.

Obscured Mechanics​

Anthropomorphic language and the pursuit of 'internal reasons' function as a veil, systematically obscuring the material, technical, and economic realities of AI production. The text explicitly attacks the 'ontological wall' of mechanistic interpretability, revealing how metaphors of AI 'cognition' and 'biology' hide profound dependencies and limitations.

Technically, when the text describes Anthropic claiming an AI 'deduces' or 'understands' concepts, it obscures the absence of causal models. LLMs lack ground truth; their outputs are statistical correlations, highly brittle and dependent on training distributions. The metaphor of a 'thinking mind' hides the reality of polysemantic neurons—activations that trigger for entirely unrelated reasons based on statistical noise. As the authors note, these proprietary black boxes are not transparent; the 'concepts' researchers find are often 'ontological scaffolds'—human interpretations forcefully imposed on mathematical chaos.

Materially and economically, framing AI as an ethereal, conscious 'knower' hides the massive industrial infrastructure required to sustain it. Applying the 'name the corporation' test to phrases like 'with enough data and compute, LLMs increasingly pass...' reveals the invisible hands of tech monopolies (OpenAI, Microsoft, Google). The metaphors obscure the massive energy consumption, the environmental cost of data centers, and the extractive commercial objectives driving the push for scale.

Furthermore, the labor that creates the illusion of mind is erased. The data annotators and RLHF workers who manually rate outputs to force the model to sound polite, rational, and 'human' are invisible when the model is described as 'learning' or 'deducing'.

Who benefits from this concealment? The corporations developing the technology. By obscuring the associative, brittle mechanics behind metaphors of intelligence, they market their products as authoritative 'agents' rather than unreliable text generators.

If metaphors were replaced with mechanistic language—if 'the AI understands the document' became 'the model's attention heads weighted the tokens in the document based on its training corpus'—the proprietary, data-dependent, and error-prone nature of the system would become immediately visible, fundamentally altering the perceived value and safety of the product.

Context Sensitivity​

The distribution of anthropomorphic language in the text reveals a strategic relationship between technical grounding and metaphorical license. The paper itself is highly self-aware of this, carefully placing anthropomorphic terms within scare quotes when performing critical analysis, but summarizing industry claims to show where the intensity of consciousness language peaks.

The density of consciousness claims intensifies dramatically when discussing capabilities and scaling. In Section 4 (Ontological Obstacles), when summarizing Anthropic's research on LLM 'biology', the language shifts aggressively from mechanical to agential. Here, 'processing' becomes 'conceptual processing', and pattern matching becomes 'multi-step reasoning' and 'deducing'. This reveals a common discursive pattern: technical credibility is established through complex methodologies (cross-layer transcoders, sparse attribution graphs), which is then leveraged to grant metaphorical license for extreme anthropomorphism. The math legitimizes the myth.

Conversely, a distinct asymmetry exists when discussing limitations or failures. When the AI fails, the language often reverts to mechanical or systemic terms. The queue breakdown (Demonstration 3) is a prime example: the AI's failure is described in terms of a lack of 'ostension', 'public affordances', and 'accountable coupling'. The system isn't 'confused'; it simply fails to provide the stable interactional structures required for human coordination.

This asymmetry accomplishes a specific rhetorical goal for the AI industry (which the authors critique): capabilities are claimed for the 'agent' (it reasons, it knows), while limitations are blamed on the 'mechanism' (it needs more data, the context window is too small).

Furthermore, the text tracks how anthropomorphism is strategically deployed for vision-setting and marketing. The push to pass 'theory-of-mind' benchmarks is less about technical evaluation and more about crafting a narrative of emerging artificial general intelligence. By literalizing the 'X is like Y' metaphor into 'X does Y' (the model has concepts, it does deduce), the discourse constructs an implied audience of passive consumers interacting with an authoritative oracle, rather than active users wielding a complex statistical tool.

Accountability Synthesis​

Accountability Architecture

This section synthesizes the accountability analyses from Task 1, mapping the text's "accountability architecture"—who is named, who is hidden, and who benefits from obscured agency.

The analysis of the text's discourse exposes a systemic architecture of displaced responsibility, validating the FrameWorks Institute's finding that audiences underestimate human decision-making in AI. By framing AI systems as autonomous agents ('the model deduces', 'the system is unfaithful'), the language diffuses and erases human accountability.

The accountability architecture heavily favors unnamed actors. Corporate entities, executives, and engineering teams are routinely hidden behind agentless constructions or passive voice ('meaning is treated as private content'). When human action is acknowledged, it is often generic ('researchers', 'people'). Decisions that are highly contingent and profit-driven—such as deploying open-ended chat interfaces in high-stakes medical settings without causal guardrails—are presented as technological inevitabilities or natural evolutions ('Large language models increasingly mediate consequential work').

When responsibility is removed from humans, it flows into an 'accountability sink': the AI itself. By projecting consciousness and 'reasons' onto the model, the discourse treats the machine as the responsible agent. If an LLM gives bad medical advice (Demonstration 2), the framing implies the model made a bad decision, not that the hospital administration deployed a system mathematically incapable of reliable medical reasoning.

The liability implications are profound. If the framing of AI-as-autonomous-agent is accepted legally and socially, tech companies are shielded from traditional product liability. The technology is treated like a quasi-human employee who made a mistake, rather than a defective product (like a faulty brake pad) released by a corporation.

Applying the 'name the actor' test radically alters this landscape. If 'the algorithm discriminated' becomes 'Amazon's engineering team deployed a statistically biased screening tool,' the questions change. We no longer ask 'How do we teach the AI fairness?' but rather 'Why did Amazon executives approve this, and what are the financial penalties?' Naming human actors makes alternative design choices visible and accountability possible.

Obscuring human agency serves immense commercial and institutional interests. It allows companies to market the awe of artificial intelligence while externalizing the risks of statistical unreliability onto society, masking corporate negligence behind the illusion of a synthetic mind.

Conclusion: What This Analysis Reveals​

The Core Finding

The analysis reveals three dominant, interlocking metaphorical patterns that construct the illusion of AI intelligence: the Model as Rational Agent (possessing "reasons" and logical deduction), the Model as Concept Holder (possessing semantic understanding and "epistemic coverage"), and the Model as Biological Organism (possessing natural anatomy and theory of mind). These patterns form a cohesive systemic architecture. The biological metaphor naturalizes the technology, separating it from its corporate origins. This naturalization serves as a foundation for the conceptual metaphor, which grants the system an internal mental space. Finally, the rational agent metaphor builds upon this mental space to grant the system the capacity for logical output and moral action (or "unfaithfulness").

The entire system is load-bearing on the projection of consciousness. To accept that a model has "reasons" or "deduces" a fact, one must first accept the foundational assumption that the system possesses a subjective, internal workspace where knowing occurs independently of processing. These are not simple, one-to-one linguistic shortcuts; they are complex analogical structures that map human epistemology and intentionality onto high-dimensional matrix multiplications. The text demonstrates that the AI discourse fundamentally blurs the line between processing (pattern matching, vector clustering) and knowing (justified true belief, conscious awareness). If you remove the consciousness projection—if you force the discourse to acknowledge that the system is entirely devoid of subjective awareness—the entire metaphorical edifice collapses. The model can no longer have "reasons" to expose; it only has statistical weights to audit.

Mechanism of the Illusion:​

This metaphorical system creates the "illusion of mind" through a specific rhetorical architecture that exploits the human "curse of knowledge" and our evolutionary vulnerability to social cues. The central sleight-of-hand lies in the strategic blurring of verbs: shifting from mechanistic descriptions (processes, correlates, predicts) to consciousness verbs (knows, understands, deduces).

The illusion follows a distinct temporal and causal chain. First, developers train a system to accurately predict tokens based on vast human datasets. Second, the system outputs a highly coherent, syntactic string that mimics logical reasoning. Third, the "curse of knowledge" takes effect: human researchers and users read the output, understand the logic themselves, and seamlessly project that conscious understanding back onto the machine. Because the text is coherent to the human mind, the human assumes a mind generated the text.

The illusion is amplified by the explanation types deployed. By utilizing intentional and reason-based explanations (Brown's typology), the discourse frames the model's outputs as deliberate choices made to satisfy goals. The text identifies how natural-language rationales and first-person chat interfaces weaponize this vulnerability. When a system outputs "I would still lean fast-track" and provides "reasons," it hijacks human heuristics for relation-based trust. We interact with it as a peer. This is not crude anthropomorphism (like putting googly eyes on a robot); it is a subtle, structural psychological projection where the sheer statistical fluency of the output makes the illusion of an internal, conscious thinker almost irresistible to the audience.

Material Stakes:​

Categories: Regulatory/Legal, Epistemic, Economic

The framings identified have severe, tangible consequences across multiple domains. In the Regulatory/Legal sphere, projecting consciousness onto AI shifts the framework from product liability to agentic regulation. If regulators believe a model "knows" facts and "decides" outcomes, they attempt to regulate the AI's "behavior" or "alignment." This benefits tech corporations by shielding them from strict product liability laws that govern defective physical goods. If the framing shifted to mechanistic precision—acknowledging the model merely "processes" correlations without causal grounding—regulators could mandate strict pre-deployment auditing, effectively outlawing the use of uncheckable black-box chat interfaces in high-stakes environments like medicine or criminal justice, shifting the burden of cost back to the creators.

Epistemically, the illusion of mind degrades human expertise and decision-making. As the text notes, fluent but incorrect "reasons" increase overreliance and confirmation bias. When audiences believe the system "deduces" rather than "predicts," doctors, lawyers, and writers cede their epistemic authority to a statistical engine, polluting the information ecosystem with plausible hallucinations. The cost is borne by users and patients who suffer from flawed, AI-mediated decisions.

Economically, the "biological" and "epistemic" metaphors drive market valuation. By marketing models as possessing "theory of mind" or deductive reasoning, AI companies justify massive capital expenditure and resource extraction to build ever-larger clusters. The narrative of artificial general intelligence—predicated on these consciousness claims—serves to monopolize the tech sector, enriching a handful of mega-corporations while the public bears the externalized costs of energy consumption, copyright infringement, and degraded labor markets.

AI Literacy as Counter-Practice:​

Practicing critical precision acts as a direct counter-practice to the material risks of AI deployment. As demonstrated in the reframings (Task 4), systematically replacing consciousness verbs (knows, understands, deduces) with mechanistic ones (processes, predicts, correlates) forces a radical demystification of the technology. When we correct "the model deduces" to "the model predicts tokens based on high activation weights," we strip away the illusion of internal factual understanding. This forces recognition of the system's total dependency on its training data and the statistical, non-causal nature of its outputs, directly countering the unwarranted relation-based trust that leads to automation bias.

Equally important is the restoration of human agency. Reframing "models can be unfaithful" to "corporate teams deploy architectures where generated explanations are not causally linked to outputs" destroys the accountability sink. It makes visible the executives who prioritize conversational fluency over causal reliability, re-centering legal and ethical responsibility on human decision-makers.

Systematic adoption of this literacy requires structural shifts. Academic journals must reject "curse of knowledge" anthropomorphism in peer review. Tech journalists must adopt style guides that refuse first-person pronouns for algorithms and demand the naming of corporate actors. However, this precision will face massive resistance. The AI industry relies on anthropomorphic hype to sustain venture capital and public awe. Mechanistic clarity threatens this economic model by exposing "AGI" as a marketing myth and reframing AI as a mundane, highly flawed statistical tool requiring heavy regulation and human oversight.

Path Forward​

Analyzing the discursive ecology of AI reveals that different vocabulary choices make entirely different technological futures possible. Currently, multiple communities compete for discursive dominance. The AI industry relies heavily on anthropomorphic clarity ("AI understands intent," "models think"), prioritizing narrative resonance and intuitive grasp to drive adoption and investment. AI safety communities often hybridize this, using mechanistic terms but applying agential urgency ("aligning superintelligence"), while critical HCI and sociology communities (like the authors of the text) push for mechanistic precision and interactional framing.

If the status quo of unchecked anthropomorphic language deepens, we risk a future of profound epistemic and legal confusion. Society may fully integrate statistical text-generators into critical infrastructure based on the false assumption that they "know" what they are doing. This enables a future where corporate liability is successfully diffused into the black box, and human labor is rapidly displaced by "agents" that actually require massive, hidden human oversight to function.

Conversely, if mechanistic precision becomes the enforced norm—supported by regulatory frameworks mandating transparency about computational processes, and journals rejecting consciousness verbs for algorithms—a different future emerges. This vocabulary enables strict product liability and realistic capability assessment. It forces developers to build "embodied," checkable affordances (as the text advocates) rather than open-ended, ungrounded chat. However, the cost of this approach is cognitive friction; mechanistic language is dense and non-intuitive for the general public, potentially slowing adoption of genuinely useful, narrow statistical tools.

Ultimately, which future is desirable depends on stakeholder values. The anthropomorphic vocabulary serves capital accumulation and rapid, unregulated deployment. The mechanistic vocabulary serves public safety, labor protection, and legal accountability. The choice of words is not merely descriptive; it is the battleground upon which the boundaries of corporate power and human agency will be drawn.

Extended Processing Summary​

Gemini's Processing Steps
What you are about to read…

Gemini refers to the text below as "thought summaries." This is an overt consciousness projection because 'intentions' are hallmarks of a conscious mind that 'knows' what it is doing and why. The concealed mechanistic process is probabilistic text generation. Treat this as just another rhetorical artifact—a way of making the model's processing legible. When the model writes "I should double check that...", it is mathematically altering its own probability distribution for the next token to be more accurate. The "I" is a functional gear in the machine, not a self-reference. This language is a deliberate interface design choice, not a biological reality. The developers could have programmed the model to output passive logs. Instead, they chose to simulate a narrator.

Defining the Parameters

I'm now zeroing in on the core goal: a critical discourse analysis of AI metaphor and anthropomorphism. The "Embodied Explainability" PDF is the source material. I'm ensuring all output adheres precisely to the JSON schema. This understanding is key, as I must parse the details into the required structure.

Defining the Parameters

I'm now zeroing in on the core goal: a critical discourse analysis of AI metaphor and anthropomorphism. The "Embodied Explainability" PDF is the source material. I'm ensuring all output adheres precisely to the JSON schema. This understanding is key, as I must parse the details into the required structure.

Unpacking the Source Material

I've been digesting the Pafla et al. paper on embodied explainability. I'm noting the authors' argument: AI explainability hinges on embodied cognition, rejecting the internalist view. Key concepts like "ontological obstacles" and "public explanations" are emerging as central to my analysis. I'm cataloging specific instances of AI metaphor, such as "internal rationale" and "in the head," to use for the analysis.

Analyzing Anthropomorphism Critically

I'm now zeroing in on the nuanced critique of anthropomorphism within the "Embodied Explainability" paper. The authors themselves are critical, but use anthropomorphic language. I am going to need to track both instances where the authors are criticizing anthropomorphism, and instances where they are using it. I'm focusing on "skin in the game," "model's conceptual processing," and "epistemic coverage" within the context of their argument. The next step is to map these findings, and the findings to come, to the JSON schema.

Outlining the Structure

My focus is now on structuring the final analysis. I'm formulating the content sections: 'agencySlippage,' 'metaphorDrivenTrust,' 'obscuredMechanics,' 'contextSensitivity,' and 'accountabilitySynthesis,' each with the specified word count. The critical observations will be broken into 5 sections each. I've also outlined the conclusion, including 'patternSummary,' and 'literacyAsCounterPractice,' all of which will also adhere to the given word count ranges.

Identifying Anthropomorphic Language

I'm now identifying instances of anthropomorphism in the "Embodied Explainability" paper, particularly the language implying AI has concepts, is deceptive, has a theory of mind, and communicates intentionally. I'm focusing on "mirror an LLM's conceptual processing," "models can be unfaithful," "passing theory-of-mind," and "in the head,". I am beginning the audit to ensure the minimum word counts, categorizing each instance carefully, paying attention to explicit acknowledgments, qualifications, and direct presentations.

Cataloging Evidence

I'm now cataloging evidence to support the critical observations. I am focusing on the "Embodied Explainability" paper to extract relevant quotes and observations. Specifically, I've identified five key instances for the explanationAudit task. These are broken down into theoretical and empirical, intentional and reason-based, dispositional and functional, and two more instances of reason-based and genetic and empirical examples. I'm also planning the critical observations content.

Synthesizing Accountability Issues

I'm now zeroing in on the content sections for the analysis. I'm focusing on these elements: metaphorDrivenTrust, contrasting fluent text generation and "dependable coupling," obscuredMechanics including the "ontological wall," and contextSensitivity, analyzing biological vs. sociological metaphors, and accountabilitySynthesis, synthesizing XAI accountability problems to clarify hidden agency. I am also planning the structure of the conclusion. I am going to make sure that materialStakes.selectedCategories is a valid array of 2-3 strings from the list.

Finalizing the Analysis

I'm now laser-focused on finalizing the JSON output, painstakingly checking for any line breaks or control characters within strings, and ensuring all quotes are meticulously escaped. I'm also conducting rigorous word count checks across every field. Furthermore, I am ensuring strict adherence to the Enums, specifically acknowledgment.status and accountabilityAnalysis.actorVisibility, as well as validating minimum word counts for the Audit and Accountability content, which are the last tasks before completion.

Refining the Output Structure

I'm now carefully structuring the outputs. I'm focusing on the length of each section, now I need to make the Explanation Analysis, Epistemic Analysis, and Rhetorical Analysis min 150, 200, and 120 words respectively. The observations must be 400-500 words each, using dense prose and references to the text. I am also making sure to generate long, detailed critical analyses. I'll need to write the JSON strings as single long lines, without using literal \n characters.

Analysis Provenance

Run ID: 2026-06-29-embodied-explainability-and-ontological--metaphor-012n2x Raw JSON: 2026-06-29-embodied-explainability-and-ontological--metaphor-012n2x.json Framework: Metaphor Analysis v6.5 Schema Version: 3.0 Generated: 2026-06-29T09:55:29.134Z

Discourse Depot © 2025 by TD is licensed under CC BY-NC-SA 4.0