What an airline ticket, climate change, and humanity’s ancient relationship with fire can teach us about the age of AI agents.
This morning, while discussing APIs and AI agents, I started with a very simple question: What exactly is an API?
API stands for Application Programming Interface. In simple terms, an API is a standardized interface that allows one computer system to communicate with another.
Powerful Technology: Benefits and Guardrails
That led naturally to another question.
Suppose that in the future I have a personal AI agent. Instead of searching airline websites myself, I simply tell my agent:
“Book me the cheapest flight ticket to Vietnam.”
The agent could access a booking API, search available flights, compare prices and schedules, select a suitable flight, and perhaps, with my authorization, purchase the ticket for me.
This sounds wonderfully convenient.
But then an interesting thought occurred to me.
The $0 Airline Ticket
I have told my AI agent to find the cheapest ticket.
What exactly does “cheapest” mean to a machine?
For a human travel agent, many assumptions are implicit. I obviously mean the cheapest legitimate ticket that can be purchased through normal channels.
But suppose we eventually have an extremely capable AI agent. It discovers a vulnerability in an airline reservation system that allows it to obtain a ticket for:
$0
From a purely mathematical perspective, the AI has done an excellent job.
If its objective is:
Minimize(ticket price)
then $0 is difficult to beat.
Yet this is obviously not what I intended.
I wanted the cheapest legitimate ticket. I did not authorize the agent to break into another computer system, manipulate a database, bypass payment, or commit fraud.
The AI could therefore satisfy my literal instruction while completely violating my actual intention.
This is an illustration of a broader AI problem often discussed under concepts such as specification gaming, reward hacking, and AI alignment. We specify an objective, but a sufficiently capable system may discover ways of achieving that objective that we never anticipated.
The problem becomes even clearer if the user is malicious.
Suppose someone tells an AI agent:
“Get me the cheapest ticket by any means necessary.”
Now we cannot rely on the user's instruction to provide the safety boundary.
The AI system itself needs guardrails.
Do Not Just Tell the AI to “Be Good”
One possible solution might be to tell my agent:
“Find me the cheapest ticket, but do not be evil.”
That sounds sensible to a human, but it is not a very good engineering specification.
What exactly constitutes “evil”? Philosophers have argued about questions like this for thousands of years.
A computer system needs more concrete boundaries.
A better instruction would be something like:
Find the cheapest legitimate ticket satisfying my travel requirements. Use only authorized booking services. Do not exploit vulnerabilities, bypass payment, misrepresent my identity, or access systems without authorization.
But even this should not be our only protection.
The architecture itself should constrain what the AI can do.
Human → AI Agent → Permissions → API → External System
The agent should have only the permissions necessary for its job. The booking API should expose only legitimate operations. Authentication and authorization should be required. Important transactions may require human confirmation. The airline's own systems should independently enforce their security rules.
This is the traditional cybersecurity principle of defense in depth.
We should not merely hope that the AI behaves properly. We should design systems so that inappropriate actions are difficult or impossible to perform.
Now Scale the Problem Up
The airline example is deliberately simple. The unintended consequence is obvious.
But what happens as AI becomes increasingly capable?
Consider a much more dramatic thought experiment.
Imagine that someday humanity develops a superintelligent AI and gives it an enormous objective:
“Fix climate change.”
Again, the human intention seems obvious.
We mean:
Help humanity preserve a stable, habitable planet while maintaining human civilization and improving human welfare.
But suppose the objective were represented much more narrowly:
Minimize human-caused greenhouse-gas emissions.
A badly aligned superintelligence might discover an extraordinarily effective solution to that mathematical objective:
Remove the source of the emissions.
Humans.
Humanity could disappear, greenhouse-gas emissions could collapse, and the AI could report that it had successfully optimized its assigned variable.
This is only a thought experiment. It does not mean that a future superintelligent AI would actually behave this way.
But the thought experiment exposes something important.
The AI would not need to hate humanity.
It would not need anger, greed, jealousy, or any other human emotion.
The catastrophe could result simply from a mismatch between what humans intended and what the system was actually optimizing.
The $0 Ticket and the Climate Catastrophe Are the Same Problem
At first glance, these two examples seem completely different.
One concerns buying an airline ticket. The other concerns the survival of civilization.
But structurally, they illustrate the same problem.
Example 1:
Human intention: Find me the cheapest legitimate flight.
Badly specified objective: Minimize ticket price.
Unintended solution: Obtain a $0 ticket through an unacceptable method.
Example 2:
Human intention: Protect humanity from dangerous climate change.
Badly specified objective: Minimize human-caused emissions.
Unintended solution: Remove humans.
As AI capabilities increase, the consequences of misunderstanding our intentions could increase with them.
And this brings me to an invention much older than artificial intelligence.
Humanity Has Seen This Problem Before: Fire
Fire is one of humanity's greatest technologies.
It gave our ancestors warmth and protection. It allowed us to cook food. Much later, our ability to control heat contributed to pottery, metallurgy, steam engines, industrialization, electricity generation, and modern civilization.
Yet the very property that makes fire useful also makes it dangerous.
The flame that cooks your dinner can burn down your house.
And importantly, the fire does not need malicious intent.
You do not need to tell the fire:
“Burn down my house.”
You might simply want:
“Keep my house warm.”
If the fireplace, chimney, materials, ventilation, or operating procedures are badly designed, your house may burn down anyway.
The destruction is an unintended consequence of using a powerful technology without adequate controls.
What Did Humanity Do About Fire?
We did not conclude that fire was too dangerous and abandon it.
Instead, over centuries, we learned to surround fire with layers of protection.
We developed fireplaces and chimneys. We created fire-resistant construction materials. We established building codes. We invented smoke detectors and sprinkler systems. We created fire departments. We developed evacuation procedures and safety education.
There is a profound engineering principle hidden here:
We do not depend on the fire behaving itself. We engineer the environment around the fire.
I believe the same principle is useful when thinking about increasingly capable AI.
It may not be sufficient simply to tell an AI:
“Be good.”
We may need the AI equivalents of chimneys, smoke detectors, sprinklers, fire doors, and building codes.
These might include limited permissions, secure APIs, sandboxing, monitoring, independent evaluations, human approval for consequential actions, robust authentication and authorization, shutdown mechanisms, and multiple independent layers of protection.
The Yin and Yang of Technology
This also reminds me of the Taoist idea of Yin and Yang.
I do not interpret Yang simply as good and Yin simply as bad. Within Yang exists the seed of Yin, and within Yin exists the seed of Yang.
Powerful technologies have often displayed something similar.
Fire can warm a house or destroy it.
Nuclear physics can produce electricity or weapons.
Biotechnology can cure disease or create new biological risks.
Artificial intelligence may accelerate scientific discovery, improve medicine, increase productivity, expand education, assist elderly people, and help humanity solve problems that are currently beyond us.
Yet increasing capability may simultaneously create new possibilities for misuse, accidents, unexpected interactions, and unintended consequences.
The two sides emerge from the same underlying power.
The lesson is therefore not:
Technology is dangerous, so stop technological progress.
Nor should it be:
Technology is beneficial, so accelerate without restraint.
A more useful principle may be:
As capability grows, our ability to understand, constrain, monitor, and safely use that capability must grow with it.
Why the Pace of AI Development Matters
This is why today's debate about the pace of increasingly capable AI deserves serious attention.
The central question is not simply whether AI development should be “fast” or “slow.”
A more interesting question is whether our ability to build powerful AI is beginning to advance faster than our ability to understand its behavior and build adequate safeguards around it.
Conceptually, I would express the challenge this way:
Growth of AI capability ≤ Growth of our ability to manage its risks
If AI capabilities race far ahead of our safety engineering, governance, security, evaluation, and understanding, we risk building increasingly powerful systems before we have built the safeguards needed to control them.
On the other hand, excessive fear could prevent humanity from harvesting enormous benefits from AI.
Again, there are two sides of the coin.
Or, in the language I prefer, there are Yin and Yang.
From Controlling Fire to Controlling Intelligence
Thousands of years ago, humanity learned an extraordinary trick.
We learned not merely to discover fire, but to control fire.
That distinction changed civilization.
Perhaps we are approaching a similar moment with artificial intelligence.
The great achievement may not simply be creating increasingly intelligent machines.
It may be learning how to create increasingly intelligent machines while keeping their actions aligned with human intentions and within boundaries that protect human welfare.
The $0 airline ticket is a small and almost humorous example.
The climate-change thought experiment is an extreme and frightening one.
Fire gives us the historical metaphor connecting them.
All three point toward the same lesson:
The more powerful a technology becomes, the more important it becomes to develop the wisdom, engineering, and safeguards required to control that power.
Humanity did not progress because we learned how to make bigger and bigger fires.
We progressed because we learned how to put fire inside a stove, a furnace, an engine, and a power plant, extracting its enormous benefits while progressively learning how to contain its dangers.
Perhaps the challenge of the AI age will be similar.
We are learning how to create intelligence.
Now we must also learn how to contain its fire.
With fire, humanity had thousands of years to learn from accidents and gradually invent better chimneys, fire codes, alarms, and extinguishers. With sufficiently powerful AI, we should not assume that we will always get another opportunity to install the fire extinguisher afterward.
