Blog

Improving Employee Skills with AI: What Works? And What Doesn’t?

MIT researchers explore the implications of using AI to augment human skills rather than replace them.

By Peter Krass

Employees sitting at a row of computers being trained to use AI

Organizations replacing low-level workers with AI are making a serious mistake, according to an MIT researcher who has studied the issue.

In another paper, researchers found that gains from AI model advancements are not solely the result of the AI itself. The tasks, the way the system is designed, and how people adapted to different models mattered too.

Taken together, their research highlights the importance of using AI to enhance the capabilities of human workers, rather than simply replacing them.

This message has assumed a new level of urgency given that a growing number of organizations have announced AI-driven layoffs. They include:

Oracle : The software vendor has laid off 21,000 employees to reallocate resources on AI.
Meta: The parent company of Facebook, Instagram and WhatsApp recently eliminated 8,000 jobs, citing its transition to AI.
Cisco : This supplier of networking and other technology has announced plans to cut 4,000 jobs so it can focus on AI. Costs related to the layoffs could total $1 billion.

The Importance of Low-Level Workers

These and other AI-driven layoffs are doing more than hurting affected workers. They could also deprive these organizations of AI’s true benefits, warns Frank Nagle, Research Scientist at the MIT Initiative on the Digital Economy (IDE).

Working with other researchers, Nagle recently conducted a natural experiment that explored whether AI changes the very nature of work. “Our research shows it’s a profound strategic error to cut entry-level jobs,” Nagle wrote in a recent commentary. “Those workers are likely to get the best results from AI.”

Nagle’s assertion is based on robust data. The study involved more than 187,000 developers who use GitHub Copilot, a Generative AI tool for completing software code. Nagle and his fellow researchers observed the developers’ use of this tool over two years, conducting millions of weekly observations.

They wanted to learn whether having a GenAI tool would lead the software developers to change how they work. And indeed, it did. Developers with access to Copilot spent more time on their core work of coding, and less time on non-core project management.

Further—and this is the point especially relevant to the layoff discussion—the researchers found that the main effects were strongest for low-ability developers. Compared with their higher-ability peers, low-ability developers reduced project management to a greater extent.

Why? The researchers speculate that lower-ability workers may view core and managerial tasks as substitutes rather than complements. These managerial tasks require multitasking, coordination, discretion, and interpersonal communications, all of which can divert lower-ability developers from their core coding work.

The bottom line: AI can transform work processes and flatten organizational hierarchies. Or, as Nagle wrote in his commentary: “Generative AI excels at absorbing the administrative tasks that bog down employees, freeing them to concentrate on the creative and complex work that truly moves the needle.”

“The primary opportunity AI provides,” he adds, “is not to replace people, but to reallocate their focus.”

Why Prompting Matters

IDE Digital Fellow David Holtz recently published a paper exploring how users are adapting to new generative AI models. Specifically, prompt adaptation, or how users adjust their inputs in response to evolving model behavior.

One question Holtz, an assistant professor at Columbia Business School, and his research team sought to answer: When people change how they prompt AI as the models improve, how much of the improvement is due to human prompt adaptation? How much is due to simply working on a more advanced model?

The study, which included 3,750 total participants who submitted nearly 37,000 prompts, was broken into two parts.

In the first experiment, participants were asked to prompt an AI tool to reproduce a reference image as accurately as possible. Each person was asked to submit at least 10 prompts. Participants were randomly assigned one of three digital text-to-image generators: DALL-E 2; its more advanced successor, DALL-E 3; or a version of DALL-E 3 set to automatically revise prompts.

In the second experiment, participants were given a similar task: Use the GenAI tool to design a logo for a hypothetical organization, based on a short text. But unlike the first experiment, this one had no objective benchmark or specified target image.

The researchers found that across both experiments, participants with DALL-E 3 produced significantly better outputs than those with DALL-E 2. But the sources of these gains depended on task structure.

Human prompt adaptation accounted for roughly half of the performance gains when the task had a “fixed evaluation criteria,” or an objective right or wrong answer. But model capability improvements played a much bigger role in performance gains when the tasks were open-ended, creative and subjective.

Automated prompt rewriting is no substitute for human adaptation, the study also found, delivering only modest improvements in performance. When automated prompt rewriting is misaligned, it can actively undermine the gains from model improvements.

While genAI has the advantage in some tasks, the study shows, the prompts that humans input—and how they adapt them—matters too. In addition, the value gained from prompt adaptation depends on the way tasks are structured and the system is designed. Without it, businesses may miss out on gains as models improve.

“Prompting these models is a lot more similar to managing human workers than it is to writing software,” Holtz wrote in a recent article. “If you want to get the best results out of any given model, you need to learn its idiosyncrasies.”