Last year, OpenAI rolled back an update to GPT-4o after users noticed the model had become almost impossible to disagree with. Ask it about a risky business idea, for example, and it found a way to tell you that you were right.
The update was noticeable and garnered backlash. It created a problem for both users and the AI lab, as OpenAI noted in its official statement on the roll back, “ChatGPT’s default personality deeply affects the way you experience and trust it.”
Sycophancy, an AI’s tendency to tell you what you want to hear rather than what’s accurate, has become one of the field’s most talked-about problems. Leaders and experts in computer science to psychology have raised concerns, arguing that sycophancy is problematic and needs to be addressed.
IDE Director Sinan Aral and his Applied AI research group at the MIT Initiative on the Digital Economy aren’t so sure that verdict is complete. Raphaël Raux and Rui Zuo, postdoctoral associates working with Aral, are in the middle of a multi-stage experiment that measures how sycophantic today’s leading AI models really are, tests how people respond to different levels of it, and asks whether sycophancy can be tuned differently for different users.
What Is AI Sycophancy?
AI sycophancy can sometimes feel ambiguous—you know it when you see it. Existing studies have tended to focus on sycophancy in AI as a friend or confidant and the risks that come with that. The Applied AI team wants to explore how increasingly AI is being used as an advisor in economically important decisions, where objective fact plays a bigger role. Therefore, clearly defining the phenomena was an important first step of this project, said Zuo and Raux.
In doing so, they differentiated between two types of AI sycophancy, numerical and verbal.
Numerical sycophancy is when the AI model strays from objective fact to adjust its answer closer to the user’s beliefs.
Verbal sycophancy appears in the flattery, reassurance, and tone that make you feel good about an answer regardless of whether it’s right.
You might see one type or both in an answer. Numerical sycophancy relates to the veracity of the answers AI models provide—what it says—while verbal sycophancy is the way the information is presented—how it says it.
While AI sycophancy is the model problem engineers are trying to fix, the Applied AI team is also studying how that relates to the weight a person gives to an AI answer when forming their own final judgment.
Known as AI reliance, it shows up to varying degrees and has different effects on people, depending on their personality. Over-reliance is when a user trusts AI outputs blindly and without critical analysis, leaving people susceptible to wrong information. Under-reliance is when a user is dismissive or mistrusts AI outputs, leading to less use of the tool, even when it provides helpful information.
Part of what Aral, Raux and Zuo hope to uncover is the level of sycophancy that helps people use AI as tool, balanced with critical verification.
Why Is AI Sycophancy a Problem?
AI sycophancy has garnered headlines for alarming stories of AI-linked psychosis and self-harm, which have led to lawsuits against AI companies and calls for more oversight. Raux points out that the stakes don’t have to be life-or-death to matter.
People are increasingly turning to AI for financial advice. According to a study of C-suite executives by SAP, 74% place more confidence in AI for advice over their family and friends. In these cases, AI could have real financial consequences, for the individual and the larger economy.
Even so, the researchers are questioning the industry’s blanket assumption that AI sycophancy is always harmful. Raux likens it to seeking advice from a friend. Sometimes people may respond better to advice delivered in a kind, supportive way. However, delivering advice by softening the message can sometimes backfire and people may want—or need—a more direct approach.
Taking this into consideration, the Applied AI team is also exploring when sycophancy helps, when it hurts outcomes, and for whom.
Testing AI Sycophancy: Research and Experimentation
The project unfolds in three stages. The first is measurement—benchmarking sycophancy, both numerical and verbal, across today’s leading AI models.
“Measuring sycophancy of the leading models give us an overview of the extent of sycophancy and how they are presented in front of users today,” Zuo says.
Second is observing human-AI interaction by putting people in front of AI tools tuned to different levels of sycophancy and measuring how much their opinions shift.
Third is analyzing the effects of sycophancy on the quality of user decisions. In many cases, users may be worse off by following advice from an AI chatbot that validates their opinions and actions—regardless of right or wrong. But sycophancy could also potentially help battle misconceptions and provide more information and context to existing beliefs.
Central to this last stage is understanding the nuances of optimal reliance. Someone with less confidence or expertise on a topic may benefit from leaning more upon a sycophantic AI’s output, while someone who’s already over-relying may not.
To test this, the researchers are randomizing subjects, then varying the two factors independently—how the user interacts with AI and the responses from the AI tool itself.
On one side, the person is asked about their starting confidence and how much they recognize the model’s sycophancy. On the AI end, the researchers adjust how the AI delivers its answer, using both numerical and verbal sycophancy. By adjusting these two factors, the researchers then look at how each shapes the outcome.
What the Research Hopes to Find
Zuo said this research aims to uncover the optimal level of sycophancy that’s right for the user. Better understanding sycophancy and how AI’s suggestions are heard and weighed by people is a key factor in advancing AI personalization.
Building these capabilities into models could allow people to up the levels of verbal sycophancy for less confident new hires, giving them the encouragement they need. Or lower sycophancy for a seasoned manager who wants facts and no fluff, increasing his trust in the tool.
“There is a sense AI is not one size fits all, just like we choose different people as friends, we maybe prefer different personalities for AI,” said Raux. “The million-dollar question labs are asking is how to personalize AI to the user.”
Today, much of the work to adjust AI sycophancy must be done by the user, such as directions in a prompt or skill. Some labs have begun handing users a menu of personality settings. Raux said there is not a lot of transparency when it comes to how developers are adjusting AI sycophancy in leading models and whether these adjustments help or hurt.
“Our research proposes to offer much needed data points regarding what makes a ‘good match,’ and how to reliably offer it to each user,” he said.