This essay is the fourth in a series that examines and explains the mechanics of AI, the challenges in discussing it with clarity, and the harms it presents. This concluding essay uses recent public communication from an AI researcher to examine in detail how the misuse of language contributes to fear and misunderstanding of AI. It’s important to be clear about communication because there is a very dangerous risk of becoming distracted by less important issues. AI is dangerous, but not for the commonly discussed reasons.
Although I am not an expert in AI or computer science, as I’ve outlined in the previous essays, the issues around AI can be understood by a motivated lay person. My interest in warning on this topic is because of the harms to children specifically, and the economy generally. Everyone should be informed, and have opinions about, AI. It’s too big to ignore.
The example analyzed below is a letter posted by Daniel Selsam online on September 14, 2026. The full text can be accessed here:
I do not know Daniel Selsam, and am not picking on him in particular. He is a senior manager at Open AI. From all appearances, he seems an extremely intelligent, reputable, and accomplished AI professional who is genuinely concerned with the ramifications of the work he does. The problematic ways he talks about AI are pervasive in the industry and the media. The point here is that if an experienced, accomplished, software engineer speaks and thinks like this, it’s no wonder the media and broader population misunderstand what AI is and can do.
The examples I highlight may seem like nit-picky hairsplitting, but the sloppy, ambiguous use of these words is where much of our misunderstandings of AI arise. I will leave to others the speculations about the motivations behind the ways the industry talks about AI, other than the obvious financial incentives.
Early on in his letter, Selsam uses the phrase, “neural networks learning to reason.” It’s a great jumping off point for examining how word choice creates confusion. First, the neural networks referenced have very little in common with living neurons, beyond the use of feedback loops to modify themselves. The evocation of neurons is a common practice in computer science that encourages the misgeneralization of the physiology of living systems to computers. This is the latest manifestation of a practice that misuses analogies of whatever the current technology is to explain the workings of living systems, a longstanding practice that goes back hundreds of years.
A distinction also has to be made between actual deductive reasoning and outputs from the software that look like reasoning. It’s possible the “reasoning” output could be the result of a retrieval of a similar pattern of verbiage from it’s training data, or that the “reasoning” is the result of a statistical inference rather than a true chain of deduction. These are topics of vigorous discussion and debate in the AI world. Both the words “learning” and “reason” are doing a lot of hidden, extra work here, which implicitly suggest the software output is the work of a conscious intellect.
The next several sentences articulate a thoughtful, prudent, concern about how well the AI industry is regulating itself and the potential risks failing at this may pose. These are human problems, though, that require human oversight in the form of regulations and penalties. Ruinous fines and jail time for corporate leaders will focus attention quickly on prudent behaviors.
Then we get to this sentence:
“The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled.” (bold mine)
As I’ve said before, there is no “I” in software. Software is not “aware”, and software does not “believe”. It’s possible that a very broad definition of aware could be applied to software that updates it’s environmental state in real time could be considered to have “awareness”, just like a thermostat is “aware” of the temperature in the living room, but again, that usage is doing a lot of hidden work.
The next sentence says, “…how they would behave…” Behave is a word that is likely the least inappropriate in AI usage, if only for historical reasons. There is a long track record of using the word behavior to describe the performance of mechanical systems in various circumstances, even though the origins of behavior are in living things. But again, the usage in AI unconsciously biases the reader/listener to begin assuming there is something more going on than merely software operations.
A few sentences later, this important statement appears:
“Both types of processes may be described as AI or ASI, but “AI” is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered “AI” matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues.”
Here he begins acknowledging the imprecision of language often used in AI circles, and the very real phenomenon of humans investing incompletely understood things with imagined capabilities until later it is better understood. That is absolutely occurring today, where the opacity of these complex systems is mistaken for sentience, cognition, and agency. These are well understood byproducts of human cognitive biases and shortcomings, projection and pareidolia (misperceiving human attributes in inanimate objects) being the main ones.
After a great discussion of the limitations of current AI models and how they fail at important basic measures of cognition, this sentence follows:
“However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase.”
First, and I know this is sounding like more pedantic quibbling, but software doesn’t have amnesia, living things do. Software can have data continuity issues, or state persistence issues, but calling these phenomena “amnesia” once again implies consciousness that isn’t there. Second, the ONLY way AI models can “steer the world” is if they are connected to, or are permitted to connect to, external control systems. Those are human decisions, or human failures.
There follows a detailed discussion of iterative improvements of the software, initiated by the software. Astonishing and powerful capability to be sure, but it can only happen if humans allow it to happen. There is nothing foreordained or deterministic about it. Every capability, every risk, is the result of a conscious, human design decision. If the software has the ability to amend itself, it’s because humans allowed it. The next highlighted sentence captures the abandonment of responsibility so common in AI industry apocalypticism.
“If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process.”
The “if” in this sentence is doing a lot of work. We only approach that threshold if AI engineers, and the company executives that pay them, make the explicit decisions to enable the software to get there. But the next paragraph is the most concerning (bold mine):
“Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model’s explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases.”
This gives up the game. Here we have the admission of a highly trained expert at the cutting edge of a transformational technology admitting to his own cognitive surrender. This is a choice, an admission of weakness and despair, not a diagnosis of the capabilities of the software.
Throughout the rest of the document, casual usage of words like “models know”, “models understand”, “realize”, “care”, “perceive,” all add to the pervasive assumption that there is a nascent consciousness stirring somewhere inside the mountains of code.
This all culminates in his conclusion where he states, “If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek.”
He concludes with “I do not have answers, but as a first step, I wanted to share my present concerns.”
He does in fact have answers, embedded throughout his essay. “If progress continues…” sums it up. Strip away the anthropomorphizing, the cognitive surrender, and the despair, and what is left are words and ideas he himself uses:
1) the industry is suffused with “hype” and “confusion”;
2) industry leaders have begun considering self-restraint and regulations, but it’s not enough;
3) the human responses to the outputs of the software are the result of explicit, designed, coded programming choices manipulating deeply entrenched, known human cognitive shortcomings and biases. The software using first person pronouns, the reinforcement patterns (learned from the social media engagement engineering) are all the result of engineering choices to maximize human engagement in the service of revenue goals;
4) because of these financial incentives, affecting EVERYONE associated with the industry, all claims must be evaluated with a pitiless “cui bono?” filter, and treated with the appropriate amount of skepticism. Never lose sight of the massive infusions of venture capital that must be repaid; the inflation of AI valuations and the hype needed to keep them inflated are intimately connected to the financial incentives;
5) the human capacity for self-deception, projection, anthropomorphizing, and irrational delusions are an unfortunate complication of work that attempts to replicate human cognition. Even the most accomplished scientists in human history have held wildly irrational beliefs alongside their greatest scientific insights; no opinion, diagnosis, or forecast emanating from any sector of the AI industry should be exempt from hard scrutiny, fact checking, and validation;
6) the hype, misdirection, and human cognitive failures discussing AI technology lead to overlooking, underestimating, or ignoring the actual harms occurring right now, namely the economic risks of the AI capital investment bubble, and the adverse impacts on education due to cognitive surrender.
We do not have to, and CANNOT, rely exclusively on software engineers and their employers, with an enormous financial stake in the outcome, to have exclusive authority to mold our perceptions and decisions about AI policy. We all need to participate in the debate about restrictive AI regulations, and we need to do it now.
Leave a Reply