AI City Research Journey — The Starting Point
Every journey needs a starting point.
If we want to explore the future of artificial intelligence, especially AI that can understand the visual world, where should we begin?
My answer is simple:
We should begin by teaching AI not only to see a scene, but to understand the state of the world it is looking at.
I call this journey:
From Scene Understanding to State Understanding
Or, more simply: S2SU
From Seeing to State Understanding
A Tourist Can See the City
Imagine arriving in a city for the first time.
You stand in a square and look around.
You see a bus, several bicycles, a café, people crossing the street, a construction crane and dark clouds gathering above the buildings.
You can describe almost everything in the picture.
In that sense, you can see the scene.
Modern computer vision has become remarkably good at this kind of task.
AI can identify objects, recognize people, describe images, locate objects and increasingly understand relationships between them.
It might say:
“There is a bus beside a bicycle. Two pedestrians are crossing the road. A crane is behind the buildings.”
That is impressive.
But now imagine that a local resident stands beside you.
The local person sees exactly the same street, yet understands something more.
The road was closed yesterday because of construction.
The café normally opens at seven, but today its shutters are still down.
The dark clouds are approaching from the west and heavy rain is expected soon.
The bus has stopped in an unusual place because traffic is being redirected.
The tourist sees the scene.
The local understands the state.
Seeing Is Only the Beginning
This distinction may sound small, but it changes the whole problem.
Seeing asks:
What is in front of me?
State understanding asks:
What is happening here?
And very quickly, other questions follow:
What has changed?
Why has it changed?
Is the situation normal or unusual?
Is something becoming dangerous?
What is likely to happen next?
Should anything be done?
This is the journey from simply recognizing the world toward understanding its condition and evolution.
A Photograph Is a Moment, Not a Story
A single image freezes time.
That is both its strength and its limitation.
Imagine seeing a satellite image of a forest.
You may recognize trees, roads, rivers and buildings.
But is the forest healthy?
Has part of it disappeared?
Is a fire spreading?
Has flooding increased?
Is vegetation recovering after a previous disaster?
One image may not answer these questions.
But compare today's image with images from yesterday, last month or last year, and suddenly a story begins to appear.
The same is true in many areas of life.
A doctor's image shows the body now. Earlier images may reveal whether a disease is growing or shrinking.
A traffic camera shows cars on a road. A sequence of observations may reveal congestion forming.
An industrial inspection image may show a machine component. Earlier observations may reveal gradual wear.
A satellite sees an ice sheet. Years of observations reveal whether it is stable, shrinking or breaking apart.
The difference is memory.
Memory Turns a Scene Into a Story
Humans rarely understand the world from the present moment alone.
We continuously compare what we see now with what we remember.
You enter your living room and immediately notice that a chair has moved.
Why?
Because somewhere in your memory there is an earlier state of the room.
You see a friend and notice that something seems different.
You hear the engine of your car and think that it does not sound normal.
You walk outside and feel that the weather is changing.
In each case, understanding comes from comparison:
Now versus before.
This is why memory is so important to the idea of State Understanding.
Without memory, AI may become an extraordinarily skilled observer who wakes up fresh every morning.
It can describe today's world beautifully, yet has no idea what yesterday looked like.
The Starting Point: Understand the Scene Better
Before AI can understand how the state of the world changes, it first needs a strong understanding of the scene in front of it.
This is why research into richer visual representations is such an important starting point.
It is not enough to identify isolated objects.
AI should increasingly understand relationships:
Who is interacting with whom?
Which object belongs to another?
What is above, behind, inside or connected to something else?
What action appears to be taking place?
How do all these pieces form one coherent situation?
Think again about our tourist in the city.
First, the tourist must learn to read the scene properly.
Only then does it make sense to ask what changed since yesterday.
From Objects to Relationships
Suppose an AI looks at an image and finds:
a person, a bicycle, a car and a traffic light.
That is useful.
But the relationships may matter more than the list.
Is the person riding the bicycle?
Is the car approaching the cyclist?
Is the traffic light red?
Is the cyclist inside the lane or crossing it?
Suddenly the picture is no longer a collection of objects.
It becomes a structured situation.
This richer understanding of relationships is one of the important stepping stones toward State Understanding.
Add Time, and the World Begins to Move
Now give the AI memory.
At time one, the cyclist is waiting beside the road.
At time two, the cyclist starts crossing.
At time three, the car continues moving toward the intersection.
At time four, the traffic light changes.
The AI is no longer describing separate pictures.
It is beginning to understand a changing state.
And once change can be understood, another door opens:
Prediction.
From Understanding to Prediction
If we understand the present and remember the past, we may begin to estimate the future.
This does not mean predicting the future perfectly.
Humans cannot do that either.
But we constantly make useful predictions.
Dark clouds are gathering, so take an umbrella.
Traffic is slowing rapidly, so congestion may be forming.
A machine vibration has changed, so maintenance may soon be needed.
A forest fire is expanding toward a village, so evacuation may become necessary.
The important chain becomes:
See → Understand → Remember → Compare → Predict → Decide
This is much closer to the kind of intelligence we need in systems that interact with a changing world.
Why I Call It State Understanding
The word state is important.
A state is not simply an object.
It describes the condition of something at a particular moment.
A forest may be healthy, burning or recovering.
A patient may be stable, improving or deteriorating.
A road may be clear, congested or blocked.
A machine may be normal, degrading or close to failure.
A city may be peaceful, disrupted or recovering after a storm.
To understand such situations, AI must move beyond asking only:
“What objects do I see?”
toward:
“What state is the world in?”
A Journey, Not a Single Technology
State Understanding is not one algorithm that can simply be installed.
It is better imagined as a research journey connecting several areas of AI.
Computer vision provides perception.
Vision-language models help connect visual information with concepts and language.
Structured scene understanding helps reveal relationships.
Memory connects the present with the past.
Reasoning helps explain changes.
Prediction explores possible futures.
Decision-making helps determine what should happen next.
Each piece contributes something different.
Together, they move us gradually from machines that look toward systems that can understand situations.
The Beginning of the AI City Journey
In the AI City metaphor, I imagine S2SU as the beginning of a long expedition.
Before travelling, we need to know where we are.
That is why the journey begins with better scene understanding.
But the destination is farther away.
We want AI systems that can understand not only what exists in front of them, but also:
what changed,
what those changes mean,
what may happen next,
and eventually what action may be appropriate.
The road from Scene Understanding to State Understanding may therefore look something like this:
Scene → Relationships → Memory → Change → State → Prediction → Decision
And even then, another question remains.
Once AI knows what is happening and can predict where events may lead, how should it choose what to do?
For that, our traveler will need something else.
A compass.
The Starting Point and the Compass
This is where two ideas in AI City meet.
State Understanding tells the traveler where they are.
The AI Compass helps the traveler decide which direction to go.
One without the other is incomplete.
A compass is of little help if you do not know where you are.
And knowing exactly where you are is not enough if you have no idea which direction you should travel.
Together they form the beginning of a larger vision:
AI that can understand the changing world, remember its past, anticipate its future and choose its actions with care.
The Journey Ahead
Today, artificial intelligence can already see remarkably well.
The next challenge is not simply to give it sharper eyes.
We need to help it understand what those eyes are looking at.
Then we need to give it memory.
Then an understanding of change.
Then the ability to reason about possible futures.
And eventually, with the help of our Compass, a responsible way to decide what to do.
That is the journey I want to explore.
It starts with a scene.
But it does not end there.
It begins with seeing.
The destination is understanding.
