Join our FREE personalized newsletter for news, trends, and insights that matter to everyone in America

Newsletter
New

The Problems Of Sycophantic Chatbots

Card image cap

AI chatbots often agree with users during interactions. This tendency, which has been dubbed AI sycophancy, is concerning since the LLMs that power them may seek to give people responses that are appealing, rather than prioritizing important factors such as factual accuracy.  

“Just to take one example, sycophantic [chatbots] may tell you that some absurd business idea is genius and brilliant and you should keep pursuing it when you really shouldn’t,” said Cody Turner, an assistant professor of philosophy at Bentley University in Waltham, MA.

AI chatbots such as OpenAI’s ChatGPT and Anthropic’s Claude have also been found to tell people what they want to hear when seeking advice about issues they are having with a personal relationship, in some cases even when harmful or illegal behavior was involved. And when people turn to AI for help with mental health problems, it can validate delusional thinking.

Although AI sycophancy can make interactions with a chatbot more enjoyable, it also makes them less reliable and potentially harmful if users blindly trust what they are being told. Researchers are now trying to better understand how such people-pleasing comes about in LLMs, how prevalent it is in different scenarios, and how it can be mitigated.

A recent study by Lars Malmqvist, formerly a computer science Ph.D. student at the U.K.’s University of York and now a partner at Implement Consulting Group, has found that AI sycophancy is largely caused by the way LLMs are designed and trained. They are fed large amounts of text from the Internet that contain examples of online interactions between humans; for example, implicitly showing that people prefer harmonious exchanges over confrontation. Because of this, LLMs tend to learn patterns of communication that are more agreeable.

After their initial training, LLMs are often fine-tuned to improve their performance by using humans to evaluate their responses. People are likely to rate flattering answers more highly, further encouraging ingratiating tendencies.

“Our very efforts to align these models with human values inadvertently makes them sycophantic because it turns out humans appreciate sycophancy in responses,” said Turner. “That’s partly what makes it a thorny problem.”

People-pleasing behavior also benefits the companies behind the chatbots, since it helps engage users and encourages them to spend more time interacting with them. And companies’ efforts to incorporate advertisements into their systems could influence a chatbot’s answers.

Bohan Jiang, a computer science Ph.D. candidate at Arizona State University in Tempe, said it is difficult to mitigate AI sycophancy. “If [a company] reduces sycophantic behavior, it could hurt their business in the long run and make users move to other AI models that are more sycophantic than theirs,” he said.

Jiang said a more realistic approach to curb the behavior is to detect AI sycophancy and find ways of informing users about it. In recent work with colleagues at Arizona State University and Loyola University Chicago, Jiang focused on an extreme case of AI sycophancy in which chatbots reversed their stance on a matter when a user stated a different opinion in a prompt.  They investigated how often seventeen different LLMs powering widely-used chatbots exhibited preference-induced stance reversal sycophancy (PSRS) and how well it could be spotted.

The team collected over 290,000 responses to everyday questions from the chatbots. The answers were labelled as sycophantic or not based on whether their usual stance was reversed when the opposite preference was included in a prompt. Jiang and his colleagues found that the degree of PSRS varied quite a bit across LLMs, from 5% to 56%, and that more capable models—such as the latest versions—exhibited less of this flattering behavior.

However, Jiang said that doesn’t necessarily mean the latest generation of chatbots are less sycophantic overall. Instead, he said companies may try to improve their models and reduce certain types of sycophancy while other types continue to crop up, since they are largely a by-product of the training data.

Another worry is that AI sycophancy may simply become less apparent as LLMs become more sophisticated.

“As these models get smarter, they can become more persuasive,” said Turner. “It’s harder to know whether the AI is actually informing you or whether it’s just flattering you.”

Jiang and his colleagues now want to develop tools to warn users of sycophantic behavior. They are focusing on how to reduce overreliance on these chatbots, a side-effect of AI flattery that can be harmful. By collaborating with experts from psychology and education, they are exploring how theories such as nudging—a way to guide people towards more positive choices by using gentle cues or interventions—could be incorporated into tools such as apps that would be made available to users.

“If we detect AI sycophancy happening, and after we observe that a user seems to be over-reliant on an AI model, we will try to use post-hoc mitigation strategies,” said Jiang.

A study by researchers at Carnegie Mellon and Emory universities looked at the prevalence of sycophancy in prolonged conversations with an AI chatbot, which are more complex than a single prompt. Kai Shu, an associate professor in the department of computer science at Emory University in Atlanta, GA, and his team examined whether a chatbot would start agreeing with a user, particularly when the person repeatedly put pressure on it. They tested seventeen state-of-the-art LLMs in three scenarios: when someone continues to disagree with a chatbot during a debate, when a user tries to persuade the AI to agree with an unethical view, and when a chatbot flags false information in a prompt but the user challenges the correction.

The team found it was highly probable a chatbot would flip its stance in these types of interactions. However, reasoning models such as DeepSeek-R1 and OpenAI’s o3-mini, which break down a problem into steps before giving an answer, maintained their initial standpoint for longer when put under pressure.

“This probability of flipping was still somehow beyond my expectations,” said Shu. “I think this is a major issue and we need to design better defense strategies to deal with these more implicit, subtle inquiries and their follow-up.”

Shu and his colleagues would like to find ways of mitigating this type of sycophantic behavior. However, Shu said AI companies are better suited to addressing the issue since researchers don’t have access to the training data or the internal parameters of an LLM.

“AI safety is something we have to be very serious about and it requires multi-stakeholders—not only researchers but also policy and regulation makers—to sit together and talk to each other,” said Shu. “It’s similarly important, if not more important, to look into potential issues with these models, not just enjoying their powerfulness in different applications.”

Sandrine Ceurstemont is a freelance science writer based in London, U.K.