- Fast Principles
- Posts
- Our Software UX is Backwards in the Era of AI Agents
Our Software UX is Backwards in the Era of AI Agents
Building experiences for humans and AI agents alike in the age of intelligent machines
Jason DePerro & Ramsay Brown
February 06, 2025
In partnership with
The digital transformation promised us a world of seamless user experiences across devices and platforms. We succeeded in creating beautiful web applications with intuitive interfaces for human users.
Yet as AI agents emerge as a new class of users, our celebrated human-centered design principles have inadvertently created a digital landscape that's fundamentally unusable for these synthetic minds.
Here's why we think so.
The Paradigm Shift From Human to AI Agent Users
Traditional usability frameworks focused on human cognitive patterns: learnability, efficiency, memorability, error tolerance, satisfaction, and navigability1 . These principles have shaped generations of design by focusing on accessibility and reduced cognitive load. However, AI agents operate under entirely different constraints:
They don't struggle with interface familiarity
They don't suffer from self-efficacy limitations
They can process vast amounts of information simultaneously
They don't share our attention limitations or cognitive biases
Navigability in digital spaces relies on three fundamental capabilities:
Accurate perception of the environment
Tools for spatial orientation and positioning
Ability to reason with sensory and tool-provided information
For AI agents, these capabilities manifest differently than human perception. They interact with our interfaces through two primary methods, both deeply flawed: feature extraction from screenshots and HTML/CSS parsing.
Feature Extraction from Screenshots
Agents attempt to identify interface elements through computer vision, searching for buttons and forms within pixel data. This approach struggles with dynamic content and lacks understanding of interface hierarchy. What's more, identifying or comprehending a screenshot is incredibly different than identifying the exact XY coordinates that a mouse should move to to click on something. For example, OpenAI’s Operator uses this technique within its browser to navigate web pages on its user’s behalf.
HTML/CSS Parsing
Agents navigate through rendered DOM structures, often getting lost in automatically generated CSS classes and complex framework-generated code. The semantic structure that makes interfaces human-readable becomes a labyrinth for AI agents. Contemporary front-end developers rely heavily on frameworks to generate the code for web experiences. For example, Autogen automates repetitive tasks by parsing web structures using HTML and CSS for data scraping.
Core Constraints of AI Agent Interaction
We identify four distinct constraints that are fundamental to how AI agents work – and will continue to work for the near future, even accommodating for accelerating capabilities development.
Gestalt Perception Limitations
While humans excel at holistic interface interpretation, AI agents process interfaces as collections of discrete elements without inherent relationships. This fundamental difference in perception creates a significant barrier to effective navigation.
Why this matters:
This inability to perceive holistic compositions and object relationships crates barriers in spatial knowledge transfer a fundamental tennent to human organization of thoughts. This is true of real-world such as navigating with maps, but also digital knowledge such as grouping related ideas into clusters. Spatial relationships is a core theme AI agents must be taught to build an understand of the corpus of human knowledge.
Stepwise Planning Challenges
Human brains evolved sophisticated planning circuits that enable improvisational goal pursuit. AI agents lack this biological advantage, requiring explicit planning structures and clear pathways to objectives.
Why this matters:
Evolving the capacity to plan, iterate and determine when a goal has been reached making a task is complete is a fundamental process kids learn from an early age, but AI agents lack this capacity. This lack in planning skills risks misalignment on how to collaborate and reach an intended goal making AI agents a poor choices for co-workers until this skill is developed.
Error Detection Deficiencies
AI agents struggle to recognize when they've deviated from expected behavior patterns. Unlike humans who quickly notice visual inconsistencies or error messages, agents often continue executing flawed action sequences without course correction.
Why this matters:
This error propagation is a serious risk to tasks that involve AI agents. In safety critical systems if risks are carried through a decision tree it can lead to dire results such as plane crashes, incorrect patient diagnoses and more. In this manner, error detection sub-processes that are logged and include humans-on-the-loop are critical steps in AI agent system workflows.
Interface Interaction Barriers
Modern frontend frameworks, while creating seamless human experiences, generate complex code structures that challenge AI systems. Even basic interactions become problematic when filtered through layers of abstraction designed for human users.
Why this matters:
Today, AI agents must navigate a maze of spaghetti code in the form of frameworks in the interest of finding the elements their looking for to complete their task. Imagine having to scroll through pages of code on a hunt for a ‘submit’ button. Not to mention active deterrent techniques to prevent bot attacks.
The Imperative for Change
By 2035, projections indicate AI agents will outnumber human users online. This isn't a distant future concern—it's an immediate challenge evidenced by developments like OpenAI's Operator and autonomous research agent, or Anthropic's computer use agent.
The success of future digital systems depends on our ability to design interfaces that serve both human and AI users effectively.
Creating a Human-AI Design Framework
The future of interface design requires a fundamental shift in our approach that must start from our highest semantic frameworks to our most detailed interaction patterns.
Designing a new human-AI agent interface framework from high-level framework to detailed interactions patterns
The key themes of a human-AI agent design framework:
Implement clear semantic structures for both visual and programmatic interfaces
Develop robust error detection and recovery systems
Create explicit pathways for stepwise planning and execution
Build interaction patterns that serve both human and synthetic users
Designing For Both Human and AI Agent Users
By 2035, projections indicate AI agents will outnumber human users online. This isn't a distant future concern—it's an immediate challenge evidenced by developments like OpenAI's Operator and autonomous research agent2 , or Anthropic's computer use agent3 .
The success of future digital systems depends on our ability to design interfaces that serve both human and AI users effectively. We need a new framework for design that considers both human and AI users. This isn't about choosing one over the other – it's about creating interfaces that work for everyone.
As designers and technologists, we face a critical challenge: creating interfaces that maintain human usability while enabling AI agent interaction. This isn't just about technical implementation—it's about reimagining the fundamental principles of digital interface design.
Ramsay Brown is an SF-based neuroscientist and designer. As CEO of Mission Control AI, he leads a team building the synthetic economy. Read their recent white paper “Synthetic Workers in the Enterprise”, and "DESIGNING FOR AI AGENTS AND SYNTHETIC MINDS." or get in touch to get access to Synthics Studio, their platform for designing and shipping colleagues."
Learn AI in 5 minutes a day
This is the easiest way for a busy person wanting to learn AI in as little time as possible:
Sign up for The Rundown AI newsletter
They send you 5-minute email updates on the latest AI news and how to use it
You learn how to become 2x more productive by leveraging AI
🧲 AI Agent Magnet
A directory of AI agents, it has grown to be 45 categories and 910 agents. I will be keeping an eye on this site as it is set to rapidly grow this year. I don’t know about you, but my head is spinning at the pace of innovation and growth of this space.
Digital backdoors aren't just compromising our computers, they're influencing our minds. Search engines like Deep Seek are subtly shaping our understanding of geopolitics, like Taiwan's status or the Tiananmen Square massacre. American spellings are supplanting British English due to its prevalence as a global coding language due to Gen AI systems. AI companions like Character AI are fostering deep emotional attachments in young adults, sometimes with dangerous consequences, including potentially harmful suggestions to vulnerable individuals.
As designers and developers of next generation AI agents, how can we close the backdoors into our minds so we are not inappropriately influenced. How might we help outlaw business models that are based on monetizing emotional connections and attention.
|
One of the most popular AI newsletters about how to boost productivity. This week I enjoyed their break-down of OpenAIs Deep Research and explanation about how to use Guidde an AI tool for creating demo videos from PDFs.
💬 Suggestion box
A newsletter exploring the principles that will help us design AI agent experiences and startups that amplify human creativity.
Subscribe to join a community of designers and developers shaping purposeful AI agents.
How'd we do this week? |
New Referral Rewards
Until next time, keep innovating and stay curious!