Agent design through the lens of human–machine co-driving
As AI moves from answering questions to executing tasks, design must address control, collaboration and responsibility alongside capability.
Agent design extends beyond capability to control, collaboration and responsibility.
When machines can act
Public understanding of AI has largely developed through generative models: writing, answering questions, generating images and assisting with code. Their value lies mainly in information processing, so discussion often centers on whether models are smarter, more accurate or more human-like.
The problem changes when AI begins executing tasks. An agent that books tickets, sends email, operates software, manages a calendar or controls a device does more than offer advice: it directly affects outcomes.
Autonomous driving provides a practical reference: a continuous perception–decision–action–feedback loop operating under human oversight, safety constraints, legal responsibility and changing conditions. This study treats autonomous driving as a comparatively developed, real-world reference for high-risk agents. That is a research proposition, not an industry-wide verified conclusion.
Research boundary: This is a design study and structural analogy, not a quantitative evaluation of autonomous-driving safety. The concept interfaces have not been implemented or validated with users.
Why autonomous driving resembles an agent
In classical AI and robotics descriptions, an agent continuously perceives its environment, makes decisions, acts and adjusts its behavior through feedback. The basic loop is:
Autonomous driving closely matches this loop. It handles a changing physical environment rather than static text or a single API call. Road conditions, pedestrians, weather, other drivers and temporary traffic rules affect the next action. The system must reach a judgment and turn it into action within limited time.
Autonomous driving as a developed reference for high-risk agents
Developed does not mean solved. Autonomous driving has exposed the constraints of autonomous systems earlier and more extensively than many digital agents.
Text, structured data and software interfaces
A changing physical world and continuous world model
Many generative systems can take seconds to respond
Real-time, stable decisions and actions
Errors often reduce efficiency or produce incorrect content
Errors may affect property and personal safety
Often understood as isolated automation or passive execution
Switching between assistance, supervision, takeover and autonomy
Many legal and accountability questions remain under discussion
Regulation, logs, misuse and liability already matter in practice
Short-term accuracy can conceal inconsistent behavior
Stability, predictability and manageable exceptions
High stakes make redundancy, fail-safe behavior, fallback, human takeover and protection against extreme conditions unavoidable. Autonomous driving is not the answer for every future agent; it is a design setting where these questions already have real consequences.
How control shifts with automation
For design, L1–L5 can also be read as a changing relationship between the controlling party and the user's role—not only as levels of technical capability.
As automation increases, the central question becomes who has the final say, and under which conditions. L3 is particularly instructive: the system can operate independently for a period, but a person may still need to take over. Early alerts can interrupt; late alerts can miss a useful takeover window. Consistently good performance can also lead users to overestimate capability.
Human-led
Driver
Passive AI execution; full user control
Human-led, AI-assisted
Continuous supervisor
Clear, stable assistance without overpromising
Conditionally AI-led
Ready to take over
Attention, timing, risk understanding and responsibility
AI-led within defined conditions
Passenger
Clear operating boundaries and fallback
Fully AI-led
Person being served
Governance, responsibility and trust under full autonomy
Many current LLM agents share this structure: they can execute tasks but still need human review. System capability, user expectations and actual responsibility do not always move together.
The action loop, risk constraints and transfer of control show why task completion alone is not enough. Design must also constrain the goals an agent pursues, the behavior it may perform, how it can be understood and stopped, the powers it holds and how it learns from feedback. These form five connected dimensions.
Five design dimensions for high-risk agents
Together, these dimensions shape safety, usability and trust. They are concurrent constraints on action, not sequential steps. The original Chinese concept images show complete interfaces; the English HTML examples alongside them make key behaviors interactive.
Goal & Collaboration
Goal & Collaboration
Autonomous driving is not free to act without limits: traffic rules, operating conditions and the journey from A to B constrain its goals. More complex agents must clarify ambiguous goals, resolve conflicts between short- and long-term objectives, confirm preferences during execution and let users revise goals with the agent. Multiple agents must also coordinate plans and resolve conflicts.
Behavior Design
Behavior Design
A model produces candidate judgments; behavior design determines whether the system may act, how and how far. It covers constraints, triggers, failure paths and rollback. For complex tasks, reversible automation can be as important as automation itself.
Trust & Experience
Trust & Experience
Trust depends on predictability, understanding and reversibility—not only average accuracy. Users need to know what the system observed, what it is doing, why and what may happen next. Unexpected automated behavior can cause automation surprise even when overall performance is good. Explanation should support expectations before an action as well as understanding afterward.
Power & Ethical Boundaries
Power & Ethical Boundaries
Autonomous driving has powers to act, sense and use data. Future agents may also have economic and social powers, and influence perception through information ranking. Design must define final decision authority, data-collection boundaries, overdependence risks and high-risk actions requiring confirmation, while preserving an auditable chain of responsibility.
Adaptation & Learning
Adaptation & Learning
Agents may adjust risk preferences, initiative and collaboration rhythm through environmental feedback. Learning can also reinforce mistakes, accumulate bias or reward aggressive strategies. Adaptation must be visible, controllable and correctable, with ways to understand and reset learned preferences.
Design principles that transfer to agents
Control and authorization
Keep users in control of important decisions; support pause, cancel, correction and rollback; introduce automation gradually.
Transparency and explanation
Show state, reasons, evidence and next steps; make sources, reliability and uncertainty clear.
Collaboration and aligned expectations
Define goals and success conditions before execution; give staged feedback and allow ongoing correction.
Permissions, ethics and responsibility
Define boundaries the system cannot cross; preserve an audit trail; require human confirmation for high-risk behavior.
Learning and controllable adaptation
Make initiative and strategy adjustable, explainable, correctable and resettable.
Agent design is about governable action
The strongest lesson from autonomous driving is not to turn every agent into a vehicle interface. It is to see autonomy as a relationship that continually allocates control: when the system suggests, acts, waits or yields, and how every action can be understood, supervised and traced.
As agents move from answering questions to executing tasks, capability still matters. But it must operate within behavior that is predictable, understandable, pausable, reversible and auditable.
A trustworthy agent is more than a smarter model: it is a system whose power to act can be understood and governed by people.
Still to be validated
- This study uses structural similarities between autonomous driving and agents as a design analogy, not a quantitative safety assessment.
- It supplies no accident-rate, response-time, regulatory-coverage or user-study data. It does not establish universal safety or complete maturity.
- L1–L5 are used here to examine shifts in control. Engineering and legal definitions need verification against authoritative standards in further research.
- Automation surprise, responsibility and long-term trust need additional primary research, standards and case evidence.
- The principles still require validation through specific agent scenarios, prototype testing and real user feedback.