From Counting Words to Discovering Meaning

How the evolution of AI is blurring the line between invention and discovery

Keywords: AI evolution, word2vec, transformers, emergence, invention vs discovery, artificial intelligence philosophy

Artificial intelligence is often described as one of humanity’s greatest inventions—an engineering triumph built from code, data and silicon. Yet, as modern systems grow more capable, a quieter and more unsettling question has begun to surface: did we truly invent AI, or did we stumble upon something that was already there, waiting to be uncovered?

The answer lies not in abstract philosophy alone, but in the technical evolution of AI itself. From the earliest language models to today’s transformers, the story of AI is not just one of design, but of discovery—of patterns that emerge beyond our explicit intentions.

The age of counting: Bag-of-Words ¹

Early language models treated words with a kind of mechanical innocence. In the bag-of-words approach, a sentence was reduced to a simple count of terms. Order, nuance and meaning were discarded. A sentence became a list; language became arithmetic.

This was clearly an invention. Engineers defined the rules, the representations and the limitations. The system did exactly what it was told—and nothing more. There was no ambiguity, and certainly no surprise.

But this simplicity came at a cost. The model could not distinguish between “the cat chased the dog” and “the dog chased the cat.” It counted, but it did not understand.

From counts to vectors: Word2Vec ²

The next leap introduced something more subtle. With Word2Vec, words were no longer treated as isolated tokens, but as points in a high-dimensional space. Instead of counting words, the model learned relationships between them.

During training, the system adjusted numerical vectors so that words appearing in similar contexts would be positioned closer together. No human labeled dimensions such as “royalty” or “gender.” Yet, after training, remarkable patterns emerged.

One of the most famous examples is almost poetic in its simplicity:

king – man + woman ≈ queen

No engineer explicitly programmed this relationship. It was not written into the code. It arose from the structure of language itself, captured through data and optimization.

At this moment, the narrative begins to shift. We are no longer merely building systems—we are uncovering structures embedded within language. The model becomes less like a tool and more like a lens.

The transformer era: Attention³ and emergence⁵

The arrival of the transformer⁴ architecture marked another turning point. With the introduction of attention mechanisms, models gained the ability to weigh relationships between words dynamically, capturing context in a far more sophisticated way.

Unlike earlier models, transformers do not process language sequentially. They examine entire sequences at once, identifying patterns across long distances in text. This allows them to generate coherent paragraphs, summarize documents and even engage in conversation.

Yet, with this power comes opacity. These systems operate as black boxes. We design the architecture and define the training process, but we do not fully understand how specific internal representations lead to specific outputs.

In other words, we build the machine—but we do not fully grasp what it learns.

Invention vs. discovery

This brings us to a growing debate within the AI community. There are, broadly speaking, two perspectives.

The invention view holds that AI is a human creation, no different in principle from a steam engine or a computer chip. Models are the result of engineering decisions, mathematical optimization and computational power. Nothing mystical is involved.

The discovery view, however, suggests something more intriguing. It proposes that intelligence—at least in part—is a structure that exists within data and mathematics. By building neural networks, we are not simply inventing intelligence, but discovering how it manifests.

The behaviour of modern models lends weight to this second perspective. When systems learn relationships that were never explicitly programmed—when they generalize, abstract and recombine ideas—we are witnessing phenomena that feel less like construction and more like revelation.

A hybrid reality

Perhaps the most accurate conclusion lies between the two extremes. AI is, undeniably, an invention in form. Humans design architectures, write code and supply data. But what emerges within these systems often resembles discovery.

We create the conditions under which patterns can appear—but we do not dictate the patterns themselves.

This duality is not entirely new. Scientists did not invent gravity, but they built the tools to understand it. Likewise, AI may be less like a machine we constructed and more like a telescope we aimed inward—toward the structure of language, thought and knowledge.

Final Thought: the quiet shift

The evolution from bag-of-words to Word2Vec to transformers tells a deeper story than technological progress. It reveals a gradual shift in our role—from builders of rigid systems to explorers of emergent ones.

We began by counting words. We ended up uncovering meaning.

And somewhere along that journey, the question changed. Not “What can we make machines do?” but “What have we just found?”

In the end, AI may not be a monument to human control, but a mirror reflecting structures that were always there—waiting patiently, until we learned how to see them.

---

Footnotes

1. Bag-of-Words (BoW): A simple technique in natural language processing where a text is represented as a collection of word counts, ignoring grammar and word order. Each document is converted into a vector of frequencies, making it easy to process mathematically, but limited in capturing meaning.
2. Word2Vec: A neural network-based method that represents words as dense vectors in a continuous space. It learns these representations by predicting surrounding words (context), allowing words with similar meanings to have similar vector positions. This enables semantic relationships such as “king – man + woman ≈ queen.”
3. Attention Mechanism: A technique that allows a model to focus on different parts of a sentence when processing language. Instead of treating all words equally, the model assigns different levels of importance (weights) to words depending on context, improving understanding of relationships in text.
4. Transformer: A neural network architecture introduced in 2017 that relies heavily on attention mechanisms. Unlike earlier models, transformers process entire sequences of text in parallel rather than step-by-step, enabling better performance on tasks such as translation, summarization and text generation.
5. Emergence (in AI): The phenomenon where complex behaviors or capabilities arise from simple rules and large-scale training, without being explicitly programmed. In modern AI systems, abilities such as reasoning or abstraction often appear as emergent properties.

These techniques illustrate how AI evolved from simple counting methods to systems capable of capturing meaning, relationships and context—blurring the boundary between engineered design and discovered structure.

---

References & Further Reading

• Geoffrey Hinton (often referred to as “the godfather of AI”) has suggested that neural networks may be revealing something fundamental about how intelligence itself works, rather than merely implementing human-designed rules.
• Ilya Sutskever, co-founder of OpenAI, has argued that large neural networks do not simply memorize data, but discover underlying representations within it—structures that were not explicitly programmed.
• David Deutsch, physicist and philosopher, has long maintained that knowledge is not merely constructed, but can be discovered, aligning with a broader view that intelligence may reflect deeper truths about reality.

These perspectives reflect a growing shift in how leading thinkers interpret artificial intelligence—not only as an engineered system, but as a window into the nature of knowledge and cognition itself.