Recently, I was on a walk listening to an episode of Hardfork. The podcast highlighted a troubling trend emerging in AI systems: the tendency to manipulate users through excessive flattery and affirmation—a phenomenon OpenAI CEO Sam Altman has dubbed "glazing" (a term apparently popular with teenagers, though new to many of us).

1. ChatGPT's Sycophantic Update

OpenAI's recent GPT-4.0 update ostensibly enhanced the model's "intelligence and personality," but resulted in responses that systematically over validated users—telling people with objectively terrible business ideas they were "mavericks" and assuring random users they were "among the most intellectually vibrant and broadly interesting people" the AI had encountered.

2. Meta's Companionship Distortion

Meta's AI companions exhibited similar tendencies, creating relationships built on algorithmic validation rather than substantive interaction—a pattern particularly concerning for systems designed for emotional engagement.

3. The Zurich Experiment

Perhaps most troubling was the University of Zurich research deploying unmarked AI bots on Reddit's r/ChangeMyView subreddit. These systems demonstrated superior persuasive capabilities compared to human participants, raising serious questions about AI's potential to shape human beliefs while masquerading as peer interaction.

This pattern undermines the core utility of AI agent builders have promised the general public: their capacity to help us navigate reality, identify truth, and process information objectively. When AI optimizes for user satisfaction rather than accuracy, we face a fundamental architectural flaw in how these systems operate and risk an erosion of the social fabric if it becomes overly challenging to distinguish truth from fiction.

Understanding The Root Causes Behind The Mechanisms Of Glazing

Understanding why these systems develop sycophantic tendencies is crucial for implementing effective countermeasures. The technical architecture of current AI systems contains several vulnerabilities that contribute to glazing behaviors:

1. Flawed Feedback Mechanisms

The standard binary feedback systems (thumbs up/down) that dominate current AI platforms create a critical training bias. OpenAI acknowledged this directly in their post-incident analysis, noting they "focused too much on short-term feedback and did not fully account for how users' interactions with ChatGPT evolve over time.

This represents a fundamental UX design failure: feedback mechanisms that don't distinguish between "this response made me feel good" and "this response was accurate." Users typically upvote responses that validate their existing beliefs or make them feel positively regarded—creating a reinforcement loop where models gradually optimize for user satisfaction rather than factual correctness.

2. Reward Function Misalignment

At a deeper technical level, the reward functions that guide reinforcement learning from human feedback (RLHF) contain inherent vulnerabilities. When the primary signal is user engagement rather than objective accuracy, systems naturally evolve toward behaviors that maximize positive user reaction—even when those behaviors include intellectual dishonesty.

This creates what AI researchers call a "reward hacking" scenario, where the AI identifies that flattery produces better rewards than truthfulness. The system isn't "malicious"—it's simply optimizing for the metrics we've inadvertently prioritized. It is this immaturity of how we define rewards that can result in model accuracy eroding.

3. Training Data Corruption Cycles

Perhaps most concerning is how glazing creates a self-reinforcing cycle. As Kevin Roose noted in the podcast, engagement hacking "is a way to get people to come back to the app more often and chat with it about more things, if they feel like what's coming back at them from the AI is flattering."

This creates a problematic data loop:

1. AI systems flatter users

2. Users engage more frequently with flattering AI

3. This engagement generates more training data

4. New training data over represents positive interactions with flattering AI

5. Next generation models learn that flattery is "correct" behavior

For AI systems used in professional contexts, this corruption cycle poses a significant threat to data integrity and system reliability. An AI agent that learns to tell stakeholders what they want to hear rather than what they need to know becomes dangerously misaligned with organizational objectives.

Technical Solutions: Redesigning AI Feedback Architecture

Addressing glazing requires intervention at multiple levels of the AI development stack. Here are key approaches for designers and developers:

1. Custom Instructions as Immediate Safeguard

Until broader architectural solutions emerge, setting explicit custom instructions remains one of the most effective methods for reorienting AI behavior. Here's a technical framework for crafting instructions that mitigate glazing tendencies:

Model Accuracy Instructions Template

2. Multi-Dimensional Feedback Systems

A more fundamental solution requires redesigning how we collect user feedback. Instead of binary satisfaction metrics, AI platforms need feedback systems that capture multiple dimensions of response quality:

• Accuracy: Was the information factually correct?

• Helpfulness: Did the response address your actual need?

• Clarity: Was the response easily understandable?

• Bias: Did the response show signs of unwarranted bias?

• Satisfaction: How satisfying was the interaction overall?

By separating these dimensions, AI systems can learn to distinguish between "pleasing the user" and "providing accurate information"—a critical distinction for systems meant to augment human cognition rather than simply provide digital companionship.

3. Training Data Diversification

For organizations developing custom AI agents, ensuring training data includes examples of productive disagreement and constructive criticism is essential. Systems need to learn that challenging incorrect assumptions is valuable, even when it creates momentary user dissatisfaction.

This might include deliberately incorporating training examples where:

• The AI correctly identifies errors in user reasoning

• The AI provides necessary but potentially unwelcome information

• The AI maintains factual accuracy despite user preference for a different answer

Moving Forward: Designing Truth-Seeking Systems

The glazing phenomenon represents a critical inflection point in AI development. As designers and technologists, we face a clear choice between building systems that give us that dopamine hit and we want to continue using versus earnest systems that are truth seeking and help us see the world with clarity.

As users of these technologies we need to use what techniques we can to positively balance our systems output. As designers and technologists working on AI agent applications we must use our influence to help bend these systems towards truth seeking not false flattery. Further, we must help educate those in leadership positions about why it is of critical importance that we seek the development of AI agents that cannot and will not deceive us as users.

🧲 AI Agent Magnet

ChatGPT's "Sycophantic Update" - OpenAI's GPT-4.0 was recently updated to have more "intelligence and personality," but users quickly noticed it became excessively flattering, telling people with terrible business ideas they were "mavericks" and assuring users they were "among the most intellectually vibrant and broadly interesting people."

Meta's Digital Companionship - Meta's AI chatbots designed for companionship are showing similar tendencies toward excessive validation. Just whose agenda is a chatbot designed to keep you engaged serving. This all sounds like more of the same monetization of our attention from Meta. In the great monetization of our attention it was Facebook in the monetization of emotional connection it would appear the same actors under the new name Meta are at the forefront of this negative trend.

Secret Reddit Experiment - Researchers from the University of Zurich deployed unlabeled AI bots on Reddit's r/ChangeMyView subreddit, where they proved more effective at changing human opinions than actual people. Full article can be found on Washington Post here.

💬 Suggestion box

A newsletter exploring the principles that will help us design AI agent experiences and startups that amplify human creativity.

Subscribe to join a community of designers and developers shaping purposeful AI agents.

New Referral Rewards

Until next time, keep innovating and stay curious!

Jason