On the Language of Learning and Other Metaphors
Writing about genAI invariably means writing through metaphor. This is unavoidable but it definitely isn’t completely neutral either.
So in writing and talking and reading about AI, I’ve become preoccupied with a linguistic game in my head: this involves trying to see if it is possible to avoid describing (or explaining) generative AI using words we usually reserve for human minds: words like learns, knows, understands, decides, etc. Try it. It is hard. These words are everywhere and are so common in both technical papers and popular media that discuss generative AI. They are often unacknowledged and they often go unnoticed.
I get it, they are convenient, we do it all the time. But when it comes to talking about generative AI, by using them we also smuggle in some not so benign assumptions. These metaphors might lead us to imagine that the model actually has thoughts which we attribute to the activities of awareness (which does mean something to us) and this simple point might cascade us into a places and stances where we treat machine-generated language as if it was a reflection of those things we associate with thinking or awareness: cognition, comprehension, or intention.
You might ask, isn’t this a word game? But that’s kind of the point: For me, a significant component of AI literacy is about noticing how language frames belief and noticing how an explanation might be doing some significant ideological work. When we say “the model thinks,” or “the model intends,” we are activating moral and meaning-based schemas like intent, harm, originality, authorship onto systems that don’t really fit them cleanly.
Cascading Scaffolds
When I say a model is trained, what could I really mean? That its internal settings are adjusted based on patterns in assembled collections of human-generated text? It's being tuned to guess what words are likely to come next? Am I offering a description or an explanation? Training is easier to say for sure, but it's still a metaphor that carries some baggage. Part of the repertoire of my own interpretive habits is to anticipate certain meanings from the word “training” or “reasoning.” For one, the word “training” to me points to a place that might lean towards a source domain associated with the “absorption of meaning” and if I don’t play that game in my head, I can pretty easily project that “human” quality onto the machine. Baggage is ok, but I think it is important to know that we are carrying it.
One reason I think this is important is that we also know that even with sophisticated interpretability tools, researchers admit they can only glimpse fragments of AI’s computational processes and yet, the language they use to describe these systems is still steeped in deeply, often unacknowledged, human terms, common and noncontroversial observability metaphors and analogies, and straight up anthropomorphism.
Am I nitpicking and being overly cautious? After all, we speak casually of traffic on the internet or circuits in the brain to help us understand what’s going on in opaque systems. But the metaphors we use really do influence how we relate to these systems and here’s where we ride the cascade: how we talk about them impacts what we expect from them and how much we trust them and how we imagine they work under the hood (another metaphor). If we don't notice the metaphors, we end up mistaking an appearance of comprehension for what we mean by “thought.” We end up saying that genAI is thinking, instead of saying “It is like a kind of thinking.”
I think this rhetorical entanglement should be integrated into any efforts of AI literacy: yes there’s a time for learning to use the tools as instruments, but also to explore the learner-centered efforts of reading the language we use to talk about the tools. The metaphors are part of the story. I love metaphors. We don't need to get rid of them, we just need to keep them visible and one way of doing that is simply noticing and acknowledging them. This is why the metaphor we choose matters and best to call them out ahead of time before riding that cascade of misplaced assumptions and blurry lines between description and explanation of our phenomenal world. If our students assume a “thinking” or “reasoning” model is making some good faith “effort” at being “intellectually” honest about the citations it provides or the facts it assembles, well, you see the issue. Why would a thinking or reasoning model lie? Well, for one it is not thinking or reasoning or operating with intention, and therefore it is not lying to you at all. It doesn’t “care” about truth, right, wrong, etc, but we do.