The History of Neural Networks

Hitting Walls

A history of AI told through the walls we hit—and the ladders we built to climb them. A chronicle of the outcasts who built the modern mind.

[ PROLOGUE | COMING JAN 17 | ON SUBSTACK ]

The Premise

"A history of artificial intelligence told through its failures—where confident theories broke, and what those breaks reveal about intelligence, learning, and generality."

What this is not
  • Not a startup book
  • Not a trend piece
  • Not an AI optimism or doom manifesto
What this is
  • A historical reconstruction
  • Focused on moments of overconfidence
  • Written for readers who care about how ideas fail

W

e like to tell the history of science as a story of inevitable brilliance. We imagine it as a clean ascent. A straight line. A sequence of geniuses handing each other the same torch. But the story of AI allows for no such comfort. It is, in its purest form, a biography of wreckage.

We did not build the modern mind by slowly adding smarter algorithms to a stable foundation. We built it by speeding into dead ends at a hundred miles an hour, totaling the car, and then scavenging the scrap metal to build a ladder over the wall that stopped us.

From the logic gates of the 1940s to the scale wars of today, every breakthrough was born from a breakdown. This book is a chronicle of those crashes. It is not about the machines we built, but the walls we hit—and the specific human price we paid to climb them.

The Map

The Biography of Wreckage

Trace the lineage of machine logic. Each BOXED NODE is a Ladder that scaled the red Wall beneath it—only to hit its own limits at the Wall ahead. The blue labels mark the Dimensions where the tribes diverged before finally converging into Tokens. Trace the history of the climb.

Foundation (1943–1959)

There was a single dream: to copy the brain. But the early models hit wall after wall. The biological neuron was too Rigid, the Perceptron too Linear, and the first deep networks too hard to train. It took a 30-year "Winter" of isolation to build the mathematical ladder out of this hole.

Divergence (1960s–2010s)

To survive the walls of Space, Time, and Complexity, AI had to fracture into specialized tribes. Each architecture was a specific ladder built to scale a specific wall.

Convergence (2017–Present)

The walls crumbled when we realized that images, words, and actions were all just Tokens. The Transformer unified the tribes, but it did not solve the puzzle—it merely industrialised it.

TOKENS ↑
FRAGMENTATION ↑
SPACE ↑ TIME ↑ COMPLEXITY ↑
The Deep History +
Abstraction
Rigidity
Linearity
Scale
Amnesia
Sequence
Dimensions
Fragmentation
Space
Time
Complexity
Tokens
Biological Inspiration
TLU
Perceptron
Deep Belief Nets
PILLAR 1
Seeing
PILLAR 2
Reading
PILLAR 3
Doing
CNNs
LONs
RNNs
LSTMs
Transformers
Q-Learn
Deep RL
Architectural Unification
LLMs & Foundation Models

Hover over the nodes to trace the lineage. Pan and tap the nodes to trace the lineage.

01

The Myth of Progress

Intelligence wasn't built in a straight line. It was built by hitting walls at full speed.

02

The Exile

The founders of deep learning were ignored for decades. They survived on conviction alone.

03

The Unification

Language, Vision, and Action have finally merged. The result is the modern mind.

04

The Next Walls

We have not reached the summit. We are just at the foot of the Energy, Plasticity, and Epistemic walls.

The Timeline

Walls & Ladders

A structural post-mortem in five acts. Tracing the moments where the physics of the architecture forced the field to reroute—from the first rigid gates to the unified fever dreams of today.

Act I: The Foundation
1943

The Rigid Mind

Walter Pitts & Warren McCulloch attempt to map the soul to logic.

1958

The Linear Trap

Frank Rosenblatt's Perceptron hits the first mathematical wall.

Act II: The Wilderness
1986

The Spark in the Dark

The "Conspiracy" of Hinton, LeCun, and Bengio keeps the flame alive.

1998

The Sliding Glass

Yann LeCun teaches a machine to see (CNNs).

1997

The Gate

Schmidhuber & Hochreiter solve the memory problem (LSTMs).

Act III: The Explosion
2012

The Explosion

Scale meets Data: AlexNet conquers ImageNet.

2016

The Strategist

AlphaGo plays Move 37 in Seoul.

Act IV: The Unification
2017

The Attention Span

The Transformer shatters the sequential wall.

Act V: The Reckoning
Present

The Fever Dream

We built a machine that can dream, but forgot to teach it to wake up.

THE CAST

Who Builds the Ladders?

History was rarely made by the comfortable, but by the outcasts who stayed near the wreckage long enough to understand it. These are the architects who refused to negotiate with the wall.

The Outcast Founders

Walter Pitts & Warren McCulloch

A homeless teenager and a philosopher mapping the brain on a kitchen table.

The Showman & The Sceptic

Frank Rosenblatt & Marvin Minsky

Promising the world a machine that could see—right before the math hit back.

The Exiles

Hinton, LeCun, Bengio

The back-injured professor and the "conspiracy" that stayed near the wreckage for decades.

The Strategists

Demis Hassabis & Ilya Sutskever

The video game designer and the scale-engineer who realized that to build a mind, you first have to build an engine.

The Author
The Author

Kanean S A

A tech leader and AI practitioner at the intersection of laboratory research and industrial scale. Having pioneered AI agents for global leaders in BFSI and Healthcare, Kanean is a GTM leader at Google Cloud, where he collaborates with Google’s AI researchers and the world’s most ambitious AI unicorns to shape the future of machine logic.