Robert Wack

This essay is the third in a series exploring issues around AI. The first is a high-level explainer of how AI works; the second about the challenges of accurately discussing AI given the linguistic challenges of describing software without anthropomorphizing.  This essay explores the risks of AI, which ones are real, and which ones are self-serving hype. What are the specific risks? What can we do about them?

In general, the challenges of AI come from unintended consequences and the irresponsible behavior of humans setting loose things we can’t control.  In the Disney classic Fantasia, the The Wizard’s Apprentice scene is a powerful metaphor. Another is the classic horror short story The Monkey’s Paw, published in 1902, by W. W. Jacobs. The age-old lessons still apply: beware what you wish for. The possible evils inflicted by AI will be unleashed by wreckless, greedy, lazy humans and the unintended (and in some cases, intended) consequences of their bad decisions.  We know how to fix those kinds of problems, and it involves laws, regulations, jail time, and ruinous financial penalties.

To properly understand the risks of AI, we need to establish some basic premises –

• AI is software;
• software resides on a machine;
• machines require electricity for power;
• machines need separate means to interact with other systems or users beyond their immediate environment, either through physical connections to electrical or optical signals, or radio signals through wireless chips and antennas;
• the outputs of AI are in the form of text, pixels, or sounds, none of which ON THEIR OWN can actually DO anything in the physical world. For an AI program to change anything in the real world, they must interface with other software, usually a control system of some kind.

Control systems are what allow software to do things in the physical world. They connect software to what the engineers call ‘effectors’.  Effectors cause effects, they are (metaphorically) the hands, arms, feet of a software system. They are valves, switches, motors, generators, things that are on/off, up/down, fast/slow, more/less.

The apocalyptic speculations about AI harms are impressive:

• Taking control of and launching nuclear weapons;
• Creating and releasing biologic weapons (viruses or toxins);
• Taking control of and unleashing autonomous weapons systems;
• Crashing the stock market;
• Hacking and crippling the banking industry;
• Interfering with or subverting elections
• Crashing the power grid;
• Crashing the telecom system;

Because of the fundamental constraints listed in the premises, these awful apocalyptic scenarios CANNOT happen without the physical or wireless connection of AI software to control systems. That can ONLY be the result of the stupid, reckless, greedy, or malevolent behavior of human beings, either allowing these things to happen, or actively enabling them.  Even then, at present, humans still HAVE TO BE involved to assist with the implementation of the effector systems.

Let’s look more closely at the specific case of manufacturing a lethal virus. Right now, AI systems are being used to great effect to probe the genomes of living things and provide valuable insights that were previously extremely difficult or impossible to extract. It’s not a stretch at all to predict that AI could be used to create a novel genetic sequence of a lethal virus to be used as a weapon. There are already reported instances of AI companies shutting down attempts along these lines.

But lines of computer code articulating a sequence of nucleotides can’t infect or kill anything. Those nucleotides must be synthesized into a stable piece of genetic material which must be able to replicate, and then must be tested in living things to assess the effects. All of this requires what microbiologists call ‘wet labs’, physical spaces filled with glassware, specialty fluids, imaging equipment, centrifuges, specialty ventilation, incubators, refrigeration, and most importantly, the human experts to maintain and operate all that equipment. It’s an enormous undertaking that takes money, time, and expertise, and most importantly leaves obvious fingerprints and footprints.  Could it be done in secret in underground complexes, hidden away? Sure, but only by humans making the decisions to undertake such an effort.

If these horrific scenarios could be easily prevented, why are AI companies talking so much about them, and getting the media and the rest of us talking about them? This is the beginning of the real harms caused by the behaviors of humans.

Much of the current discourse by and between AI companies, their leaders, and supporters in the financial sector is driven by self-serving agendas: 1) maintaining the flow of venture capital subsidies for the preposterous expenses of training models, running them, and the capital expenditures for the infrastructure to do all that; 2) propping up the preposterous valuations of their companies;  3) restraining the possibility of competition from startups; 4) jockeying for preferential treatment and market dominance, and 5) limiting or eliminating the liability from the harms that will likely come from their reckless behavior. 

Any discussion of the apocalyptic harms should be looked at through the lens of these agendas and the deceptive language used to further them.

That’s not to say there aren’t real harms we need to pay attention to and mitigate. I refer these as the “actual” harms as opposed to the apocalyptic harms. I divide the actual harms into three general buckets:  1) the consequences of design choices of the software;  2) the economic consequences of the decisions made by AI companies and their leaders;  3) the real harms incurred by users, particularly in education.

Some of the worst consequences of the misuse of AI software can be seen with mental health, where users are turning to AI for advice about relationships, their health, or important life decisions, sometimes with catastrophic consequences.

These things happen because of specific software design choices made by the engineers around what they call “engagement”, resulting in something called “sycophancy”.

Engagement is very simple: just like the manipulative algorithms of social media, the software is engineered to encourage and incentivize more user time, longer sessions, and eventually more revenue to the owners of the software. This is done using behavioral techniques built into the user interfaces of the programs which exploit universal human cognitive shortcomings and biases. 

One of the tools they use, designed into the software, is sycophancy, the responses of the software that encourage further engagement by reinforcing the beliefs of the user and providing consistent, even skewed or false positive feedback. Basically, the software confirms preconceptions, tells the user they are smart, insightful, on to something important, all based on the prompts fed into the software. 

These operations are further baked into the software with the use of Reinforcement Learning from Human Feedback (RLHF). While the models are in development, software engineers road test them and provide the models feedback about how they respond to a prompt with the specific agenda of grooming or shaping the responses to elicit specific behavioral responses from users by creating the desired “tone” of the responses: soothing, aggressive, challenging, confirmatory; whatever the engineers are aiming for.

One manifestation of this process is a very easy tell: when the software responds using personal pronouns referring to itself. There is no “I” in software. Any output that contains the words “I”, “me”, “we”, or “us” was designed specifically to exploit the human weakness of anthropomorphizing, and then it’s all downhill from there.

Once the user is encouraged to respond to the outputs as if from a thinking being, it’s only a few short steps before trust is built and invested using sycophancy, telling the user what they want to hear, building a sense of intimacy, credibility, which then can lead to catastrophic decision making based on further outputs.  The suicides and bizarre behaviors by users of AI software are a major source of future liability for those companies, which, just like social media, are the result of specific design choices.

The second bucket of actual harms are related to the economic impacts of the rapid growth of the AI industry the distortions it creates in the U.S. economy and financial markets. Together, they introduce a significant amount of volatility and fragility into the economy which will have broader impacts when it inevitably corrects.

First, the massive debt financed investments in data center infrastructure are unsustainable. The debt service for that infrastructure, lasting for decades, cannot be realistically paid for without wildly implausible assumptions about revenue. Sooner or later that reality will trigger a correction.

Second, the development of “frontier” models and their astonishing achievements are the result of massive infusions of venture capital to subsidize the staggering operating costs of the equipment used. Though certainly impressive, it remains unproven how those costs can be supported in the long run by realistic revenue models.  Combined, the capital and operating costs of current technology are a major barrier to sustainability.

Third, the debt used to finance these activities is devolving into a circular structure, where a small group of companies are loaning each other money, or making debt and revenue commitments to each other that is increasingly fragile and unstable. One missed expectation of any individual company in the circle could trigger a cascading collapse for all of them.

Last, the massive infusions of capital and the wildly inflated expectations have caused an asset bubble for those companies that are publicly traded, skewing markets. Depending on how it is measured and when, estimates are that at least 40% of the valuation of the S&P 500 is AI companies. Add in the valuations of the tech companies that sell to, or buy from them, and it is much higher. When the correction comes, and it will, the wider financial community will feel the impact, especially those passive investors holding significant investments in ETFs related to the S&P 500.

Education harms are the most difficult to quantify, but will have the widest and most durable impact.  In addition to the mental health harms outlined previously, the intrusion of AI use into all aspects of education is already having measurable negative impacts. Tests that preclude the use of AI tools are now called “secure” tests, and scores are declining, sometimes markedly across all age groups. Teachers now also can and do use AI to grade tests and review student work. In the meantime, academic institutions are struggling to formulate and propagate coherent, consistent, and enforceable regulations governing the use of AI.  This failure is compounded by the utter lack of leadership and responsible stewardship at the federal level. Academia is at the mercy of the AI industry, which, so far, is behaving in the worst predatory capitalist ways.

One manifestation of the educational harms of AI is a phenomenon termed “cognitive surrender” a term coined by researchers at University of Pennsylvania Wharton School (https://osf.io/preprints/psyarxiv/yk25n_v1) to describe the effect repeated AI use has on students. Frequent users lose critical thinking skills, information retention, and even the ability to write well. These students have off-loaded these essential skills to the software, and like any skill, abilities atrophy quickly from disuse.

An even worse harm is the smoldering despair growing amongst young people as they see their hard work and aspirations being eclipsed by software hawked by predatory capitalists. Why bother developing elite skills if you don’t see the value of investing time and energy in careers that may be replaced with software? The AI hype frequently includes talk of eliminating some percentage of white collar jobs, a dream for oligarchs bent on extracting even more wealth from global economies, but a disaster for workers. Combined with the devastating cuts to research funding from the Trump administration and the hijacking of grant making for the financial benefit of administration insiders, and the damage to quality education and research excellence may become a generational debacle.

None of this is unavoidable. We cannot resign ourselves to the inevitability of these bad outcomes.  We must reject the premise that these companies can’t be restrained and that our doom is certain. All of it comes from choices, choices that can be constrained or reversed with concerted effort and the exertion of political power.

There is hope. Institutions like MIT are actively developing the kinds of AI policies that foster innovation without stunting academic excellence and innovation (https://aiandeducation.mit.edu/report/#f9a8ce9f-b36e-42f5-bf33-6c9f152d2029). Like all powerful technologies that preceded it, the productive, beneficial use of AI requires regulations, laws, and enforceable penalties to ensure the best outcomes and prevent the worst.

Next up: an analysis of a specific example of a AI communication to illustrate the misuse of language that encourages anthropomorphizing and doomsaying.