Dumbed down by "AI"

Last week, I had the opportunity to explain language models (what they are and aren't) to the Venetian Criminal Bar Association, and it was energizing to see first their amazement and then almost a sense of relief at a presentation on “Artificial Intelligence that was different from what you see in the media.

And God knows we need it. Every morning you open the newspapers and get hit in the face with a new bucketful of bullshit. I was still on my way home on Sunday when I read this gem on Ars Technica:

According to a CNN report, the United States narrowly avoided boarding a Chinese ship based on a “completely false” U.S. intelligence report generated with the help of artificial intelligence tools.

This erroneous information, presented by an analyst from U.S. Special Operations Command, suggested that the Chinese ship was transporting components for a nuclear weapons program through the Middle East, according to “four sources familiar with the incident” cited by CNN. The U.S. military was preparing to intercept and board the ship, with air support, before officials discovered that a chatbot used to generate the report had “misidentified the material being transported by the ship.”

One source told CNN that the AI-induced fiasco “nearly sparked a war.”

Let’s not even begin to ask what the hell kind of legal right the United States would have had anyway, had the news been true, because by now I think it’s clear to everyone that international law depends on who’s involved: it applies to the invader Putin but not to the genocidal invader Netanyahu; it is despotic and criminal to threaten Taiwan but it's OK to attack Iran and strangle Cuba.

So let’s not even bring up the mysterious issue of legality, let’s save ourselves the trouble. Instead, let’s focus on the facts we do know.

OK, some guy submitted a report generated by “artificial intelligence” that was full of nonsense. So far, nothing unusual.

What is unusual is that he didn’t submit it to a school district or a county planning subcommittee. He submitted it to the fucking U.S. Special Operations Command—the one that, since 2001, has been coordinating elite troops from all branches of the military, building operations plans on the NSA’s budget, and is essentially an extension of the CIA.

The fact that an analyst could have submitted a “completely false” (not my words) report, that the report moved up the chain of command, was transformed into an operational plan, and that the operational plan was approved only to be blocked at the last moment is not an “artificial intelligence error.”

It is the complete failure of the chain of command of the most sophisticated branch of the world’s leading military apparatus.

It’s the CIA getting caught with its pants down since 9/11. Again.

In a normal world, the entire chain of command that managed that operation (all the way up to the four-star general at the top) would be rounded up and composted. But not in the world we live in.

I don’t know about you, but I’m worried. Because to err is human, but there’s nothing human about this error. This is the classic mistake of an organization made completely idiotic by the narrative of “Artificial Intelligence”. Here we have the analyst who didn’t double-check, his superiors who didn’t verify, and every level of decision-making simply settling on a conclusion they like and pushing it forward. dsf

Professor Quattrociocchi of Sapienza University in Rome calls it “epistemy”: mistaking fluency and plausibility with accuracy and dependability. Which is, after all, the selling point of "Artificial Intelligence” in the form of language models.

Because yes, there’s the fine print and the contractual clauses that tell you that “Artificial Intelligence” can make mistakes and that you have to double-check. But these are warnings for the rank-and-file. The message to managers is “more, faster, with less discussion.” (And likely less staff altogether.)

You say, “Oh, come on, you’re really dragging this out.” Sure. Because I’m putting the pieces together, and the picture that emerges is terrifying.

A few days before this news broke, I’d noted this other story:

The Department of Defense has asked OpenAI to provide the U.S. military with a special version of its “artificial intelligence” technology designed to reject the Pentagon’s requests as infrequently as possible, according to documents obtained by The Intercept.

The desire for a customized AI tool with “minimal rejection rates” (Pentagon's own word choice) when responding to Pentagon commands emerged from files provided to The Intercept as part of a lawsuit filed under the Freedom of Information Act seeking information on secret agreements between the military and AI companies.

I’ll spare you the drama of OpenAI claiming that the text was a draft that doesn’t appear in the final version, and the Pentagon saying

“There may be some discrepancies or inconsistencies between the information obtained via FOIA, what actually exists, and what constitutes an executed contract in and of itself”

because they’re clearly trying to take us for fools, given that The Intercept’s request explicitly referred only to final documents, excluding any draft versions by definition.

Now, according to the AI “experts,” “minimum rejection rates” apparently refer to systems almost entirely devoid of the famous “guardrails,” which are supposed to prevent illicit uses.

We saw these guardrails in action two weeks ago with the OpenAI/HuggingFace case: they work so well that Altman, Amodei, and Musk agree that development needs to be slowed down because AI is slipping through their fingers everywhere.

None of this is, of course, true: the HuggingFace case is a very run-of-the-mill example of a poorly designed and even more poorly controlled experiment. Talking about AI “breaking out of its containment” is like saying that the milk I put on the stove autonomously decided to spill out of the pot while I was taking a shower and make a mess of my kitchen.

Calls to “slow down” are just a ploy by those who’ve always been in the red to get listed on the stock market just before the bubble bursts.

The guardrails are plain to see in Claude’s code: they’re pathetic pleas to “not hallucinate” and “check the results,” which the program interprets statistically, just like anything else; and anyway, two hours after a new model is released, the forums are full of detailed instructions on how to bypass them.

So, in my opinion, the American AI “experts” aren’t telling us the whole truth. Also because they are, practically all of them, former managers or former researchers at OpenAI and Anthropic, raised on the millenarian narrative of the singularity, who have discovered that spending their days spinning mental scenarios and sowing terror about the Great Artificial Intelligence soon to come, in front of a complacent press, beats working by a mile. These are people who wouldn’t be able to clean the chalkboards in a school.

But let’s not digress. According to the US brainiacs, as we said, “minimum rejection rates” would be a model without guardrails.

To me, frankly, it sounds like a fairy tale for the simple-minded, given that forums are full of instructions on how to bypass or even disable guardrails. But to the powers that be, the public is always made of simpletons.

I, on the other hand, believe that “minimum rejection rates” actually refers to something different, more in line with the narrative that has captured the imagination of every manager since the release of the first ChatGPT. I believe the “rejection” being discussed isn’t that of the machine, but that of the manager.

I believe the Pentagon wants Reason, the program imagined by Douglas Adams in the book Dirk Gently’s Holistic Detective Agency. I already told you about it, but it’s worth repeating because, in my opinion, the collision between reality and fantasy is now perfect. Listen to how Adams describes it:

“…several programs have already been written that help make decisions by correctly organizing and analyzing all the relevant facts, so that these naturally lead to the right decision. The disadvantage of these programs is that the decision to which all the correctly organized and analyzed facts lead is not necessarily the one you want.”
“…the great insight […] was to design a program that would allow you to specify in advance which decision you wanted to make and only then provide it with all the facts. The program’s task—which it performed flawlessly—was simply to construct a plausible series of logical steps to connect the premises to the conclusion.”
“And I must say it worked like a charm. Gordon managed to buy himself a Porsche almost immediately, even though he was completely broke and a hopeless driver. Not even his bank manager could find any flaws in his reasoning. Not even when Gordon wrecked it three weeks later. ...The entire project was purchased, lock, stock, and barrel, by the Pentagon. The deal put WayForward on a very solid financial footing. Its moral foundation, on the other hand, isn’t something I’d rely on too heavily.”

Here’s the thing. In my opinion, the Pentagon doesn’t want a “guardrail-free” model: people capable of blowing up an elementary school in Iran and killing more than a hundred schoolgirls on the first day of an illegal war certainly aren’t worried about the qualms that the wimps at OpenAI might have.

No, in my opinion, the Pentagon sees language models for what they are sold behind the curtain: as plausibility generators. A language model can generate arguments so convincing, and so detailed, that it can wear down any reviewer, whether through time-saturation or sheer exhaustion.

And a tool like that gives those paid to make decisions what they want most: the ability to decide without any objections. Because let’s face it, if you really have to consider all the data, you don’t have much room to maneuver. It’s no coincidence that one of the mantras of data science is that data naturally leads to a decision.

But if you’ve fought your whole life to finally be in command, the decision must be yours, the one you feel in your warrior’s gut. And now the language models come along, and with them the chance to finally silence all those incredibly annoying analysts, industry experts, and naysayers always trying to throw a wrench in the works and who question our brilliant instincts. You can take all their data and all their “natural” conclusions, and with a model that has “minimal rejection rates,” get the answer you want.
Now that’s what it means to be in charge.

Maybe I’m wrong, but I don’t think so. Over the years, I think I’ve picked up a thing or two about managerial psychology, and one thing a manager wants more than anything else is to be right. This becomes all the more difficult the further the manager is removed from on-the-ground decisions and actual work.

The manager lives under the illusion of being in the driver’s seat and in the terror of actually being in the back seat holding a toy steering wheel from Fisher Price, as Ed Zitron always says. Language models are the ideal tool for managers: they offer them the possibility (or at least the illusion) of getting the work done without the workers, and of finally having their point of view prevail.

It’s no coincidence that managers are far more captivated by the narrative of “Artificial Intelligence” than workers are. A worker sees the extra effort required to go from a draft from a language model to a finished product. The manager doesn’t; the manager sees that the first version is plausible and proceeds from there.

Do you remember “fast prototyping”? Time and again, the biggest challenge was explaining that the model was a disposable proof of concept (stress on disposable), and that if the idea worked, development had to start from scratch. We almost never succeeded. Today, the same thing is happening, but applied to everything, not just software.

If that weren’t the case, if management weren’t mesmerized by the idea of more work, done faster, with no more objections, it wouldn’t explain the insistence with which boards of directors impose the adoption of language models even against the wishes of workers, wouldn’t you agree?