Robert Wack

The prior article on AI I posted provides a very high-level look at the mathematics that drives the software in an attempt to demystify the workings with some basic technical detail. The present essay comes at the subject from a different direction, examining the concepts around math and language, and our (human) blind spots wrestling with these topics.

AI is at the intersection of mathematics, language, and cognition (thinking and consciousness).  The software uses math to make explicit, to formalize, the statistical relationships between words, phrases, and sentences to represent their meanings and how we use them. This is the source of both the wondrous results AI can produce, and the key to how we misunderstand what it is and what it does.  First, some background definitions.

Mathematics is the foundation of all scientific rigor, using equations, variables, and theorems to understand data and propose testable hypotheses about how the world works. Linguistics is the effort to formally understand the structure of languages, their origins, and relationships. AI uses mathematics to elucidate the relationships inherent in grammar, semantics, and syntax and manipulate them using mathematic tools. The outputs look like language produced by cognition, but it’s not.

Cognition is the processes of a nervous system that result in behaviors, thought, and consciousness. Intelligence is a measure of how sophisticated and effective cognitive processes are and the capacity to learn and solve problems.

Mathematics is a kind of language, using numbers and symbols to communicate formal, rigid relationships in the form of equations and theorems. Computer programming languages are another form of communication, for the purpose of organizing and controlling the interactions of electronic components that comprise a computer.

Together, math, cognition and language form a triangle:

AI uses the vast computational resources of massive computers to apply mathematics to language, turning the fuzzy, slippery, malleable meanings of words into formal mathematic expressions. As such, it lies along the line between math and language in the triangle of cognition, math, and language.

The outputs of AI are statistically generated estimates of the meaning of the inputs (the query, or question). The software does this by manipulating the mathematical relationships between the words, phrases and sentences.  That’s all the software does: making explicit what those relationships are and gives them back to us in response to a prompt. We, the humans, know what the meanings are, and if we’re not careful, we can project other things, intentions, intelligence, other layers of meaning, onto a pattern of words that the software provides that aren’t there or intended.  Notice, in this formulation, there isn’t any cognition involved with AI.

A major challenge discussing or explaining AI is that the language of computer science has not developed sufficient vocabulary to describe accurately and concisely what the mathematics applied to language is actually doing. Computer science has a long history of borrowing words from neuroscience and psychology to concisely label processes that are more accurately described using long, obscure technical explanations. Using words from psychology and neurophysiology can help explain what the computer is doing, but the shorthand terminology also opens the door for other assumptions to sneak in based on how we use those words to describe living things. How does this happen?

For example, we call the digital manipulation of data in a RAM chip, “memory” (the “m” in RAM), even though the machine isn’t remembering anything. It merely reads and writes 1’s and 0’s into a physical substrate when instructed by software. A more accurate analogy would be a bookshelf with books and blank pages you can write and erase on.  The biological process of memory is far more complex than this.

A basic example is the smell of freshly baked chocolate chip cookies. That single sensory impression, complex and layered in itself, also brings with it other memories, of special events, places, people, and most importantly, emotions, all layered and intertwined with a single sensory impression. These details are encoded in the human nervous system through the interconnections of neurons throughout the brain as well as the patterns of activity within and between groups of neurons. Human memories may be both physically stored as well as embodied in transient electrical activity. Our current understanding is incomplete and still emerging. This is clearly far more complex and mysterious than the mere logging of 1’s and 0’s.

 The use of the word memory regarding computers is now deeply embedded in the language of electronics and computer science and is not going anywhere. It inadvertently carries along unstated assumptions into our understanding of how computers work.   

Another example is the word, “perception”.  This is another widely misused word when it comes to AI.

The most generic definition of perception is “processing sensory information to produce meaning.” The casual application of perception to AI functions leaked into usage from psychology. In the psychology literature, a distinction is made between “sensing” and “perceiving”, where sensing is the physical process of receiving information from the environment, and perceiving is the processing of that information to give it meaning.  Seems pretty straightforward, especially when you look at what AI software can do these days with raw data, producing results that sure look like “rendering meaning” from sensations.

However, when you look deeper into the use of the word in the psychology literature, you start seeing references to conscious versus unconscious perception, and what role attention plays in perception. There are also large areas of research around ideas like priming (how one sensory experience modifies the next), motivations (what you want affects what you perceive), expectations (what you expect affects what you perceive), and illusions.

An example is the influence language can have on the interpretation of other sensory data and the perceptions they generate. Describing otherwise identical hamburgers made from the same ground beef as “75% lean” or “25% fat” dramatically influences the results of taste tests.

“Perception” in AI only superficially resembles the neurophysiologic and cognitive process in humans. Yes, the underlying math in the ‘perceptron’ inside the more complex AI models does extract meaning from data, but the meaning resides in statistical relationships between tokens, not the far more complex web of sensory data, memory, feedback loops, and higher-level influences inherent in human perception.

All of the AI terms borrowed from psychology presume the presence of consciousness and intelligence. When the computer science people use them, they unintentionally smuggle in assumptions about consciousness and cognition.  You can also see this with the use of the term “hallucination” for AI outputs that are inaccurate, false, or just gibberish. Hallucination implies that there is a consciousness experiencing an altered state.  Software doesn’t “hallucinate”, it just produces incorrect outputs.

Cognition is what the nervous systems of living organism achieve through the interactions with their environments, and importantly, other living things. How much cognition is present in other non-human species is debated, but it is not controversial to say that cognition is primary, and language is a tool, arising from cognition, that humans use to interact with each other.  The role of language in human development is a research topic of intense interest. It’s clear from watching infants acquire language that they are thinking long before they are speaking, or even understanding words. Once language begins developing, it acts as an accelerant to further development of cognition and intelligence, and they grow together in parallel.

Because the output of AI software is rendered as language, and we know (or are learning) that language is secondary to cognition, the easy (lazy?) assumption is that the language output of AI may be the result of cognition. This is a profound error in reasoning.

Another important point is that the meanings of words, phrases, and sentences in usage by humans are not written in stone. For AI programs, the meanings they abstract are constrained by the contents of the training data.  In the wild, meanings of words change with usage, context, over time, and between cultures. Some simple examples are the many shared words between British and American English that have vastly different usages and meanings (boot, chips, pissed, biscuit).  There are also extremely important non-verbal sources of meaning that impact the interpretation of speech, like facial expressions, gestures, and voice characteristics, all of which can shift over time, are defined by usage, and are not captured in the mathematical analysis of AI (though that is likely changing). Most importantly, vocabulary and usage are always changing.  

The relationships between language, cognition, and consciousness have been a topic of exploration going back thousands of years. Philosophers from Socrates to Roger Bacon to Alfred Whitehead all opine on the relationship between language and cognition. We use words to classify and understand the world around us, which then enables the creation of models to make predictions and dig even deeper into how things work. The conflicts between the messiness of language and the need for rigor in the sciences are legion.

One shortcoming of how humans use language is a phenomenon called reification. The philosopher and mathematician Alfred North Whitehead described it as: “There is an error; but it is merely the accidental error of mistaking the abstract for the concrete. It is an example of what might be called the ‘Fallacy of Misplaced Concreteness.”

We do it with words, which at root are abstractions we use to represent the reality around us. A fundamental weakness of the mathematical approach to language that AI uses is that the software treats words as concrete entities, rendered as tokens, and that the meanings lie in the relationships between these tokens. That’s not wrong, but it’s certainly incomplete. There is meaning in the relationships between words, but there is far more, and the words themselves are not concrete but abstractions themselves.

Another challenge around the rapid advances of AI models is that they are of sufficient complexity that the inner workings of the software are now opaque, and the outputs are less predictable due to the chaotic nature of their complex systems.

This is an important area of AI research to clarify how AI systems arrive at their outputs, providing essential transparency. Systems that have feedback loops and recursion are inherently complex and cannot always be explained in detail because of the infinite possibilities created by the complexity.  Though a difficult problem at the engineering level, it is not insurmountable at the policy level: either create workarounds, don’t use those tools, or establish walls and guardrails.  These are all choices.

Further clouding people’s judgement, even for those experts closest to the software, is the human tendency to project agency onto things we don’t understand. This deep-seated tendency is as old as the gods of mythology.  The language problems make it worse.  Projecting agency, intentions, consciousness onto inanimate things or processes happens all the time. It starts as simple as seeing patterns in random things (apophenia), seeing shapes or faces in natural objects (pareidolia), all the way up to projecting agency onto natural phenomena (weather, earthquakes, astrology). Anthropomorphizing technology is rampant today, just ask Siri.

It’s very difficult for policy makers to make decisions about engineering regulations and constraints when the experts involved in the field begin using language that is more appropriately reserved for discussions of religion or philosophy. AI models are software, wondrously complex, and in some cases supremely useful as tools for solving problems. The programs are not conscious, they are not thinking, and they don’t have intentions. They do things in the world that they are designed to do. Any harm that comes from them is the result of human design decisions and failures to control the technology they created.

At the risk of reintroducing the mystical language I just finished banishing, the proper analogies for the potential harms from AI are the Wizard’s Apprentice, and the Monkey’s Paw. In both instances, humans cause the harm with their poorly informed, short sighted choices.

I’ll explore those potential harms further in the next post.