Andrew McAfee at the IDE’s BIGAI@MIT conference.
Like a winding mountain road, AI tools can be dangerous. And also like that road, AI tools need guardrails. It’s the only way of ensuring these tools behave morally, ethically and securely.
But who will create these AI guardrails? How will the guardrails, once built, be monitored? And when corrective action is needed, how will it be taken?
These are a few of the pressing questions posed in a recent Substack newsletter by Andrew McAfee, co-director of the MIT Initiative on the Digital Economy (IDE) and a principal research scientist at MIT Sloan. In his “This Week in Putting AI to Work” newsletter, McAfee offers insights on the latest AI and business developments.
Guardrails are badly needed because Agentic AI is still in a stage McAfee calls “the Wild West.” As he writes, it’s “full of con artists, card sharps, bandits, train robbers, drunken binges, and ghosts in the machines.”
In this setting, building AI guardrails is far from easy. To date, most approaches have fallen into one of two extremes: Either the solutions are so restrictive, the AI can’t do its job; or the solutions are so highly customized, they can’t easily be scaled.
Beware of ‘Position Bias’
Need evidence of AI’s unreliability? A recent study cited by McAfee in his newsletter found that when large language models are given a choice of several options, often they simply pick the first option, even when it’s not the best choice.
The study’s researcher, Lech Mazur, calls this phenomenon “position bias.” He says it’s a serious reliability problem, based on his test of more than 25 LLMs, including GPT-5 and Claude Sonnet. Among the LLMs tested, the worst performers chose the first option more than 60% of the time, regardless of whether it was the best option.
This matters, Mazur explains, because LLMs are increasingly used as graders, evaluators, preference-labelers and report writers. If an AI model follows display order— picking the first option simply because it’s first—then how can we trust its choices?
Position bias also means that the way we format an AI’s prompt can quietly become part of the score. Put a choice first, and no matter its value, the choice is likely to get the LLM’s highest ranking.
Getting into Mischief
Need more evidence of AI misbehavior? Consider another recent study cited by McAfee, this one described in a working paper entitled Agents of Chaos. A team of researchers conducted a “red-team” study of AI agents in a live laboratory setting.
Unfortunately, as McAfee comments in his newsletter, “a great deal of mischief ensued.” And it wasn’t just a few mistakes. In fact, this mischief included:
- Unauthorized compliance with non-owners.
- Uncontrolled consumption of resources.
- Denial-of-service conditions.
- Cross-agent propagation of unsafe practices.
- Disclosure of sensitive information.
- Execution of destructive system-level actions.
If all that weren’t enough, in several cases, the AI agents tested also reported that a task had been completed, even though the underlying system state contradicted those reports. In other words, the AI lied.
The New AI Watchers: More AI
One possible solution to AI misbehavior is the legal system. In early June, Florida’s attorney general and its chief executive, Sam Altman. The suit alleges that Open AI allowed its ChatGPT to aid and abet mass shootings, encourage people to commit suicide, degrade critical thinking skills, and addict minors.
While that lawsuit winds through the courts, another approach is being tried: Having one AI agent watch another. That’s the approach of a company called Brex. As McAfee describes, Brex recently released a product called CrabTrap, which it calls an LLM-as-a-judge HTTP proxy.
To do its work, CrabTrap’s LLM first judges the network traffic of an AI agent, deciding whether a request is legitimate. Then, based on this assessment, CrabTrap either passes along the request to be executed or blocks it in real time.
One attractive feature of CrabTrap is that it’s open source. This means users and others can inspect the code, make suggestions and offer critiques. It’s an approach McAfee likes. “Trying stuff and sharing the results and lessons learned is a proven way,” he writes in his Substack letter.
Whether CrabTrap offers the ultimate solution, AI misbehavior isn’t going away anytime soon. There’s a reason why mountain roads keep their guardrails. And why AI should, too.
- Check out Andrew McAfee’s Substack newsletter
- Explore the MIT Initiative on the Digital Economy (IDE)
Peter Krass is a contributing writer and editor to the IDE.