Twenty Years of Modern Large-Scale Computing Infrastructure, Seen Through Jeff Dean

Jeff Dean

Twenty Years of Modern Large-Scale Computing Infrastructure, Seen Through Jeff Dean

In August 2026, Jeff Dean left Google. Sanjay Ghemawat, Oriol Vinyals and Quoc Le left with him, and the four of them started Discovery Loop, an AI company aimed at automating scientific and engineering research with AI. I read the story the day it broke, and the framing was the same everywhere: Google lost another batch of AI stars.

Jeff Dean Just Left Google. The Market Priced Five People at $200 Billion.

Artificial Intelligence is moving from the cathedral to the bazaar

Pull the camera back, and it looks less like an exit and more like the end of an era.

Jeff Dean and Sanjay Ghemawat, as a pair of names, run through the entire golden age of Google’s large-scale computing infrastructure. Many data-system concepts that feel obvious today — distributed file systems, MapReduce, columnar analytics, NoSQL, globally consistent databases, parameter servers, distributed training frameworks — started as Google engineering problems.

Google was not the first company to build a database, and not the first to build a distributed system.

But Google’s situation was unusual.

Very early on, it hit several problems at once most enterprises never hit:

The web is too large 
There are too many machines 
Hardware fails too often 
The index changes too fast 
Query latency budgets are too tight 
The service spans too much of the planet 
Machine learning models keep getting bigger

Those constraints meant Google could not just buy bigger machines, and could not lean on a traditional commercial database either.

It had to build the whole stack itself.

So writing about Jeff Dean is not writing about one engineer’s résumé.

The more interesting thing is to follow that line back through twenty years and watch something else take shape:

how modern large-scale computing infrastructure actually grew, one layer at a time.

1. Search forced distributed infrastructure into existence

Google’s biggest early product was search.

From the outside, search is a product where a user types keywords and gets pages back. From an infrastructure angle, there were at least three enormous engineering problems sitting behind it.

First, pages have to be crawled and stored continuously. The web kept growing, and page content kept changing. The system could not hold a small pile of structured records. It had to hold huge volumes of semi-structured content, inverted indexes, the link graph, page features, and intermediate results.

Second, indexes have to be recomputed and refreshed. Search quality depends on PageRank, text processing, spam fighting, language models, and ranking features. Many of those jobs scan enormous datasets over and over and emit new indexes and new statistics.

Third, the service has to stay up and stay fast. A user query cannot wait for an offline job to finish. Search has to answer reliably across the globe, and no single dead machine should ever be visible to the person typing.

Together, those three points at one fact:

What Google needed was not a single database. It was a data infrastructure that could keep running on a cluster of cheap machines.

That is where Google’s large-scale computing systems begin.

The industry later talked about the Hadoop ecosystem, big data platforms, data lakes, distributed compute. Most of the basic questions were already on the table at this stage:

where does the data live? 
how do jobs run? 
how do we recover from failure? 
who manages state? 
how do queries get fast? 
how does a global service stay consistent?

What Jeff Dean’s generation of engineers contributed was moving those questions out of “every product team solves it again” and into “the infrastructure solves it once.”

2. GFS: start by admitting the machines will break

MapReduce is the name people reach for first when they talk about Google and big data, but the real foundation is GFS.

GFS, the Google File System.

The problem it solves is blunt:

if we have tens of thousands of ordinary machines, 
and every one of them can fail, 
can we still organize them into one reliable large file system?

Traditional enterprise storage leaned on expensive hardware and single-machine reliability.

Google went the other way.

It assumed hardware fails.

Disks die.

Machines drop off.

Networks jitter.

Data centers have bad days.

The design does not try to eliminate failure. It treats failure as the normal operating condition, then uses replication, chunking, master-side management and automatic recovery to assemble a pile of ordinary machines into a usable distributed file system.

That thinking went straight into Hadoop HDFS.

For a lot of enterprises, the first big data platform they ever touched was really this idea:

do not stake reliability on any single machine. 
split the data, replicate it, spread it across the cluster. 
let the software layer carry fault tolerance.

Looked at from there, GFS is not an ordinary file system.

It is Google’s first basic judgment about infrastructure at scale:

once you get big enough, failure stops being an exception and becomes part of the runtime environment.

3. MapReduce: batch processing becomes a programming model

GFS answered where the data lives.

The next problem was computing over all of it.

If every team wrote its own distributed program, every team would keep re-solving the same details:

• data splitting
• task scheduling
• data locality
• spilling intermediate results
• shuffle
• retries on failure
• stragglers
• merging final results

What made MapReduce powerful is that it collapsed all of that into a minimal model:

map:    turn an input record into intermediate key-value pairs 
reduce: aggregate the intermediate results for a given key

Say you want to count how many times each word appears across the whole web.

On one machine the logic is trivial:

read pages -> split words -> count -> total

But Google was not looking at tens of thousands of pages. It was looking at web-scale documents and logs.

At that point, the hard part is not counting. It is this:

the data is too large for one machine to read; 
there are many machines, so the work must be split automatically; 
intermediate results must be regrouped by word; 
when a machine dies, its task must re-run on its own; 
computation should happen near the data, to avoid hauling bytes over the network.

The execution path looks roughly like this:

raw page / log shards 
  -> map:     each machine emits (word, 1) in parallel 
  -> shuffle: the system pulls all intermediates for the same word together 
  -> reduce:  sum per word 
  -> output:  word -> count

So on the surface MapReduce is two functions.

Where it actually earns its keep is the system layer:

• Data locality: schedule compute onto the machine holding the data, and move less of it
• Automatic sharding: cut a huge input into many small tasks and raise parallelism
• Shuffle grouping: regroup intermediates by key, so nobody hand-writes network transfer
• Retries: a dead task re-runs without corrupting the result
• Straggler handling: slow tasks get rescheduled instead of holding up the whole job

The user writes map and reduce.

The system handles scheduling, fault tolerance, data movement, and retries.

Looked at today, MapReduce is clumsy. It is bad at interactive analysis and bad at complex iterative computation. Multi-stage work means chaining several jobs, and the developer experience is coarse.

Its historical weight is not that you should still hand-write MapReduce. It is that it completed one crucial abstraction:

ordinary engineers could now write data-processing programs on a large cluster.

Hadoop MapReduce inherited the model almost intact.

Later, Spark raised expressiveness and performance with DAGs and in-memory computation, Flink made stream processing and stateful computation a first-class capability, and Trino and Presto put interactive SQL on top of the data lake.

All of them inherit the direction MapReduce opened:

the user describes the computation; 
the system handles distributed execution.

4. Bigtable: Google also had to manage state at scale

MapReduce handles batch.

Google was never only offline computation.

Plenty of online systems need to read and write state continuously:

• index metadata for web pages
• personalization data
• Google Earth data
• logs and monitoring data
• ad system features
• sparse attributes for all kinds of online services

That kind of data does not necessarily fit a traditional relational database.

It is enormous, sparsely structured, demanding on read and write throughput, and its access patterns tend to revolve around keys, column families and timestamps.

Bigtable’s abstraction is:

a distributed, persistent, multidimensional sorted sparse map

It does not emphasize full SQL, complex joins, and strong transactions the way a relational database does.

It cares about other things:

• horizontal scale
• high-throughput reads and writes
• automatic sharding
• ordering by row key
• column-family organization
• multi-version data
• working alongside infrastructure like GFS and Chubby

The industry impact was large.

HBase clearly took after it, and you can see similar thinking in Cassandra, in the LevelDB and RocksDB lineage, and in a lot of wide-column, time-series, and log storage systems.

Bigtable filled in another piece of the puzzle:

GFS stores huge files reliably; 
MapReduce runs large-scale batch; 
Bigtable holds massive online state.

Together, they form the basic shape of Google’s early data infrastructure.

They are also the prototype for the early Hadoop ecosystem:

GFS       -> HDFS 
MapReduce -> Hadoop MapReduce 
Bigtable  -> HBase

5. Dremel: big data moves toward interactive SQL

GFS, MapReduce, and Bigtable covered storage, batch, and online state.

One problem was still open: how does a human query enormous data quickly?

MapReduce can process big data, but it is not a comfortable analysis tool.

Write a job, submit, wait, look at the output, change the code, submit again. That loop is too slow.

For an analyst or an engineer, the natural interface is SQL:

SELECT country, COUNT(*) 
FROM logs 
WHERE date = '2026-08-06' 
GROUP BY country;

Dremel showed up against that background.

It targets interactive analysis at large scale.

Its key ideas include:

• columnar storage
• tree-structured query execution
• a nested data model
• aggregation over very large data in seconds or near-interactive time

Google BigQuery later drew heavily on Dremel.

This step matters a lot.

It means big data infrastructure moved from “engineers write programs to process data” toward “users ask questions in SQL directly.”

From then on, a big data platform was no longer only a background batch system.

It became the substrate for enterprise analytics, BI, reporting, experiment analysis and product insight.

That shift also explains why Spark SQL, Presto and Trino, ClickHouse, Doris, StarRocks, Snowflake and BigQuery became the main characters later.

What most enterprises actually want is not a platform that can run distributed programs.

It is this:

can people ask questions of the data in SQL, 
quickly, reliably, and cheaply?

6. Spanner: once you go global, consistency becomes infrastructure

If Bigtable is highly scalable storage, Spanner represents a different kind of ambition on the database side.

As the business went global, the system had to handle not only big data but globally distributed transactions.

Cross-data-center replication brings a lot of grief:

• Which region does the write land in?
• How do replicas sync?
• Is the user reading the latest result?
• How does a cross-region transaction commit?
• What happens when a data center fails?
• How do you trade consistency against latency?

The traditional answer was that the application absorbs some of that complexity itself.

Spanner tried to put those capabilities into the database.

Through infrastructure like TrueTime, it makes clock uncertainty an explicit part of the design, which lets it offer external consistency and transactional semantics in a globally distributed setting.

The significance for the industry is not only that Google built a very strong database.

It is closer to a change of paradigm:

before: the application handles cross-region replication and consistency. 
after:  the database provides global consistency as a base capability.

Spanner is not the answer every company needs, of course.

If your system is single-region OLTP, or mostly offline analytics, global strong consistency can easily be overdesign.

What Spanner suggests is something else:

At sufficient scale, the consistency model itself becomes an infrastructure capability.

That is the part people underestimate when choosing a modern data system.

Performance is not the only question.

What you are really buying may be a particular complexity, wrapped up and handed over:

batch complexity 
state management complexity 
interactive query complexity 
global consistency complexity

7. DistBelief: machine learning turns into a systems problem

In the deep learning era, Google’s infrastructure problem shifted again.

Early big data systems mostly handled pages, logs, indexes, ads, and online service data.

Deep learning brought new pressure:

• more and more model parameters
• larger and larger training data
• not enough compute on one machine
• inter-machine communication as the bottleneck
• checkpointing and fault tolerance getting complicated
• the input pipeline affecting training throughput

DistBelief was Google’s early large-scale distributed deep learning system.

It introduced mechanisms like the parameter server, which let a model train across a large number of machines.

From today’s vantage point, DistBelief is not a framework most people know.

But it marks an important turn:

Machine learning stopped being purely algorithm research and became part of large-scale computing infrastructure.**

Training a big model is not just writing a loss function.

It also means data loading, parameter sharding, gradient communication, distributed scheduling, failure recovery, checkpointing, and utilization.

The shape is a lot like the MapReduce era.

MapReduce wrapped up the complexity of batch data processing. DistBelief started wrapping up the complexity of distributed training.

Google going from a search company to an AI company was not a jump.

It walked there along the line of large-scale computing infrastructure.

8. TensorFlow: AI computation gets abstracted into a graph

TensorFlow was the more successful and more open generation after DistBelief.

It expresses a machine learning program as a computation graph:

nodes are operators 
edges are tensors 
the system executes across CPU / GPU / TPU / multiple machines

The abstraction has a lot in common with a SQL engine.

A SQL user describes the result they want, and the optimizer decides how to execute. A TensorFlow user describes the computational relationships, and the runtime decides placement, execution, differentiation, and device scheduling.

From an infrastructure angle, TensorFlow is more than a machine learning framework.

It pushed AI computation into a more engineered stage:

• automatic differentiation
• distributed execution
• heterogeneous device support
• graph optimization
• model deployment
• a training and inference ecosystem

It also follows Google’s usual path for system design:

hit an extreme-scale problem internally; 
abstract it into a unified system; 
then reshape the industry.

GFS, MapReduce and Bigtable went that way.

So did TensorFlow.

9. Pathways: in the large-model era, the substrate moves up again

Pathways is Google’s direction for the next generation of AI infrastructure.

If TensorFlow solved computation graphs and heterogeneous device execution, Pathways faces a larger problem:

• one model spanning a large number of TPUs and GPUs
• different tasks sharing underlying capability
• sparse activation and conditional computation
• combinations of parallelism strategies
• compiler, runtime, scheduler, and hardware topology working together

Training and serving a large model is not the old framework made bigger.

It has become a more complicated piece of systems engineering.

The model itself, the data pipeline, the compiler, the communication library, accelerator topology, the scheduling system, and the fault-tolerance mechanism all have to be designed together.

What Pathways wants to solve is how one computation can span a large fleet of accelerators while the person on top still describes the model and the task at a fairly high level.

The spirit is the same as MapReduce:

MapReduce: do not make users manage batch failures on a cluster of ordinary machines. 
Pathways:  do not make users manage execution complexity on a huge cluster of accelerators.

Twenty years on, the objects of computation went from pages and logs to tensors and models.

The core infrastructure problem did not change:

The bigger the scale, the more you need a new layer of abstraction.

10. What this line suggests about choosing a data stack

Looking back over Google’s infrastructure line is not an argument that every company should copy Google.

Quite the opposite. Most companies should not.

A lot of those systems were forced out of Google by extreme scale.

Without comparable scale, comparable organizational capability, and comparable engineering investment, copying a complex architecture usually adds cost.

What it does give you is a decent framework for choosing engines today:

Start from which complexity you want wrapped up.

If the complexity you need wrapped is large-scale offline batch, Spark is still one of the realistic choices. It is not the fastest system for every operator, but it has a mature ecosystem, scheduling, SQL and DataFrame APIs, data lake integration, and a lot of production mileage.

If the complexity is real-time streaming and stateful computation, Flink fits better. It pulls state, checkpointing, event time, windows, and exactly-once semantics inside the framework.

If the complexity is interactive SQL across data sources, engines like Trino and Presto are the natural fit. They are good at putting several sources behind one SQL layer.

If the complexity is high-concurrency OLAP and real-time analytics, columnar engines like ClickHouse, Doris, and StarRocks suit it better. Their strengths are scans, aggregation, compression, columnar storage, vectorized execution, and query latency.

If the complexity is single-machine analysis, systems like DuckDB and Polars keep getting more attractive. Plenty of jobs that used to default to Spark never had data large enough to require distribution.

So the selection question is not:

which engine is the most advanced?

It is:

where does my complexity actually sit? 
What do I want the system to solve on my behalf?

That is the most valuable part of Google’s system history.

It is not a list of product names.

It is a sequence of migrating abstraction layers:

GFS:        machine failure and distributed file storage 
MapReduce:  large-scale batch processing 
Bigtable:   massive sparse state 
Dremel:     interactive analysis at scale 
Spanner:    globally consistent transactions 
DistBelief: distributed training 
TensorFlow: computation graphs and heterogeneous execution 
Pathways:   cross-device execution in the large-model era

11. Why this exit is the moment worth writing about

Jeff Dean leaving Google for Discovery Loop is not a swerve from infrastructure into something unrelated.

From what has been made public, Discovery Loop wants to automate scientific and engineering research with AI.

That sounds more like AI for Science.

From a systems angle, it still runs along the same line:

How do you abstract a complex research process into a loop 
a system can execute, evaluate, and iterate automatically?

Google’s infrastructure wrapped up a lot of engineering complexity: machine failure, batch processing, state storage, global consistency, distributed training, heterogeneous accelerator execution.

The name Discovery Loop hints at a different kind of complexity:

propose a hypothesis 
design an experiment 
run the experiment 
evaluate the result 
update the model 
start the next round

This is no longer traditional big data processing.

The underlying capabilities it needs are still large-scale computing infrastructure:

• data management
• experiment scheduling
• model training
• automatic evaluation
• resource orchestration
• result tracking
• long-horizon state management

This moment is worth writing about, not because a famous person changed jobs.

It happens to join two eras:

the era before: Google used infrastructure to automate large-scale internet computation. 
the era after:  AI systems try to automate scientific and engineering exploration.

From the search index to large models, from MapReduce to Pathways to Discovery Loop, the theme behind the line has not moved:

Take the complicated processes humans should not be hand-running over and over, and settle them into system capability.

Closing: Google’s big data history is infrastructure taking over complexity

Jeff Dean’s career is worth rereading because it is close to a miniature of modern large-scale computing infrastructure.

GFS let Google store huge amounts of data reliably on ordinary machines.

MapReduce let engineers write batch programs on big clusters.

Bigtable let online systems manage massive sparse state.

Dremel moved large-scale analysis toward interactive SQL.

Spanner made globally consistent transactions a database capability.

DistBelief and TensorFlow turned machine learning training into systems engineering.

Pathways pushed the problem into the era of large models and heterogeneous accelerators.

Together, these systems say one thing:

large-scale computing did not advance by chasing speed alone. It advanced by pushing complexity into lower, more general, more reusable infrastructure.

That way of thinking still holds when you pick a data platform today.

Do not shop for new nouns.

Do not shop for benchmarks alone.

Do not shop for who wrote it in C++, who wrote it in Rust, who runs on the JVM.

The sharper question is:

Which complexity does this system actually take off my hands? 
Does its abstraction match my scale and my team's engineering capability?

That may be the most useful thing twenty-odd years of Jeff Dean and Google infrastructure work left the industry.

References

• Jeff Dean and other top AI researchers are leaving Google to launch their own startup: https://techcrunch.com/2026/08/05/jeff-dean-and-other-top-ai-researchers-are-leaving-google-to-launch-their-own-startup/
• Google’s Top AI Brains Are Leaving to Launch Discovery Loop: https://www.wired.com/story/jeff-dean-google-discovery-loop-startup/
• Google just announced a major shakeup of its top AI leadership: https://www.theverge.com/tech/975677/google-deepmind-ai-demis-hassabis-shakeup
• The Google File System: https://www.cs.princeton.edu/courses/archive/fall09/cos518/papers/gfs.pdf
• MapReduce: Simplified Data Processing on Large Clusters: https://www.usenix.org/conference/osdi-04/mapreduce-simplified-data-processing-large-clusters
• Bigtable: A Distributed Storage System for Structured Data: https://www.usenix.org/conference/osdi-06/bigtable-distributed-storage-system-structured-data
• Dremel / BigQuery: A Decade of Interactive SQL Analysis at Web Scale: https://www.vldb.org/pvldb/vol13/p3461-melnik.pdf
• Spanner: Google’s Globally-Distributed Database: https://www.usenix.org/conference/osdi12/technical-sessions/presentation/corbett
• Large Scale Distributed Deep Networks: https://papers.nips.cc/paper_files/paper/2012/hash/6aca97005c68f1206823815f66102863-Abstract.html
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems: https://arxiv.org/abs/1603.04467
• Pathways: Asynchronous Distributed Dataflow for ML: https://arxiv.org/abs/2203.12533
• Image sources: Google Cloud Icons: https://cloud.google.com/icons ; GCP Icons: https://gcpicons.com/ ; TensorFlow Logo: https://www.vectorlogo.zone/logos/tensorflow/