In 2019, the world was beginning to discover GPT-2.
The story everyone wanted to tell was about language.
Could a machine write convincing paragraphs?
Could it finish a sentence?
Could it generate an article?
Could a neural network learn enough from the internet to produce text that looked increasingly human?
But at almost exactly the same moment, a French programmer was approaching the same technology from a completely different direction.
His question wasn't:
“Can a machine write?”
It was:
“Can a machine predict the next symbol so accurately that I can use those predictions to make the file smaller?”
The programmer was Fabrice Bellard, the engineer behind FFmpeg, QEMU, QuickJS and a remarkable catalogue of low-level software projects.
And the problem he was attacking was one computer scientists had been studying for decades:
compression.
Bellard's 2019 paper, Lossless Data Compression with Neural Networks, described neural-network compressors using LSTM and Transformer models. The basic mechanism was brutally simple: predict the next symbol, assign it a probability, and encode the symbol using roughly the number of bits implied by that probability.
That sounds like a compression trick.
It is.
But it is also one of the deepest ways to understand what a language model actually does.
A language model is trying to predict what comes next
Consider the sentence:
“The capital of France is…”
A language model assigns probabilities to possible continuations.
Paris gets a high probability.
London gets a much lower one.
Banana gets an extremely low one.
That prediction can be used to generate text.
But there is another thing it can do.
It can compress the text.
If the model is extremely confident that the next symbol is “Paris,” you don't need to spend many bits communicating that outcome.
If the model is uncertain among hundreds of possibilities, you need more bits.
This is the connection between prediction and compression.
The better you predict the next symbol, the fewer bits you need to describe the actual sequence.
Bellard's 2019 paper explicitly describes this process: the neural network produces a probability distribution for the next symbol, and an arithmetic encoder converts those probabilities into a losslessly compressed representation.
The decoder performs the same predictions and reconstructs the original data exactly.
Nothing is thrown away.
That last point matters.
This is not JPEG.
It is not MP3.
It is not “summarising” the file.
Every original byte must come back.
And suddenly GPT looks different
This gives us a strange way of looking at a large language model.
When GPT predicts the next token, it is doing something mathematically related to what a compressor does.
The model estimates:
P(next token | previous tokens)
A compressor can then use those probabilities to encode the actual next token.
That means the training objective behind language models has a direct information-theoretic interpretation.
The model is learning a statistical representation of regularities in its training data.
The better it learns those regularities, the better it can predict.
And better prediction means lower expected coding cost.
This is why cross-entropy matters so much.
Cross-entropy isn't merely an arbitrary score invented for AI researchers.
It can be interpreted as the expected number of bits needed to encode outcomes using the model's probability distribution.
So when an AI researcher watches the loss curve fall during training, there is another way of describing what is happening:
the model is getting better at predicting the data — and therefore potentially better at compressing it.
That doesn't mean an LLM literally stores a compressed copy of the internet.
It doesn't.
A model's weights are not a ZIP file containing the training corpus.
But the mathematical relationship between predictive modeling and compression is real.
And researchers have been exploiting that relationship for decades.
Marcus Hutter saw the connection long before ChatGPT
In 2006, artificial-intelligence researcher Marcus Hutter created what became known as the Hutter Prize.
The idea was deliberately provocative:
compress human knowledge.
The competition originally used the first 100 million bytes of a particular snapshot of English Wikipedia. In 2020, the benchmark expanded to the 1-billion-byte enwik9 dataset and the nominal prize pool increased to €500,000.
Hutter's reasoning was that compression and intelligence are closely connected.
If you understand a body of information extremely well, you should be able to predict its structure.
And if you can predict its structure, you should be able to encode it efficiently.
Hutter's own description of the prize puts the idea bluntly: the intention was to encourage progress in AI through compression of human knowledge because better compression implies better prediction.
The prize therefore turned something philosophical into something measurable.
Instead of asking:
“Is this machine intelligent?”
you could ask:
“How many bits does it need to describe this body of human knowledge?”
That is a much harder number to argue about.
Bellard attacked the problem from the opposite direction
This is what makes Bellard's story so interesting.
Modern AI researchers were increasingly interested in prediction.
Bellard was interested in compression.
They were approaching the same mathematical territory from opposite ends.
His 2019 NNCP paper tested neural networks directly as lossless compressors on the Hutter benchmark.
He experimented with both LSTMs and Transformers.
And he wasn't merely proposing the idea theoretically.
He built working software.
The compressor would process the data sequentially, predict the next symbol and use those predictions to encode it. Because the decoder could reproduce the same sequence of predictions, the original data could be reconstructed exactly.
This is an important distinction from today's enormous pretrained models.
Bellard's early NNCP approach was designed around online learning: the system could learn from the data as it processed it.
That made the experiment especially interesting because the neural network wasn't simply being treated as a gigantic pretrained oracle.
It was being tested as part of the compression mechanism itself.
Then Bellard put a Transformer inside the compressor
In January 2021, Bellard released NNCP v2.
This version used a Transformer architecture based on Transformer-XL.
Bellard reported state-of-the-art results on the enwik9 benchmark and emphasized something that sounds almost absurd by today's AI standards:
the experiment could be run on a personal computer with a single GPU.
The larger model had 187 million parameters.
That's microscopic compared with today's frontier models.
But the important thing wasn't the parameter count.
It was the demonstration.
A Transformer — the same broad architectural family that would dominate modern language modeling — could be used as a serious component of a lossless compression system.
And compression gave Bellard an extremely unforgiving scoreboard.
The original file had to come back exactly.
There was nowhere to hide behind a subjective evaluation like:
“this paragraph feels pretty good.”
Either the decompressed file matched the original or it didn't.
This is where the Hutter benchmark becomes fascinating
The Large Text Compression Benchmark has existed for years as a laboratory for increasingly sophisticated compression techniques.
Traditional compressors such as gzip and xz exploit repeated patterns and statistical structure.
More advanced compressors build increasingly elaborate models of their input.
Neural compressors add learned predictive models.
Bellard's NNCP became an important milestone because it showed that Transformer-style neural prediction could compete seriously in this environment. Bellard's later published results show NNCP outperforming conventional gzip and xz by substantial margins on enwik9.
His current NNCP documentation reports an enwik9 result of about 106.6 MB for NNCP's compressed data, compared with about 197.3 MB for xz and 322.6 MB for gzip under the listed configurations.
But it is important not to confuse that with the current Hutter Prize record.
The benchmark has continued evolving. The listed Hutter Prize winners include later CMIX-family systems, with fx2-cmix taking the 2024 record at about 110.8 MB including the relevant executable.
The significance of Bellard's work is therefore not that he permanently “won AI compression.”
It is that he demonstrated, unusually clearly, how powerful neural prediction could become when measured as compression.
Then came the experiment that made the connection impossible to ignore
In 2023, a DeepMind-led research team published a paper with an almost provocative title:
“Language Modeling Is Compression.”
The paper made the relationship explicit.
Predictive models can be transformed into lossless compressors.
And compressors can be transformed into predictive models.
The researchers then asked a remarkable question:
What happens if a large language model trained primarily on text is asked to compress other kinds of data?
The answer was surprising.
Their Chinchilla 70B model compressed ImageNet image patches to 43.4% of their raw size, compared with 58.5% for PNG in their experiment.
For LibriSpeech audio, Chinchilla compressed the samples to 16.4% of raw size, compared with 30.3% for FLAC.
Think about what that means.
A model trained primarily on text was able to make useful predictions about representations of images and audio.
It wasn't because Chinchilla suddenly became a dedicated image compressor.
The researchers were demonstrating the generality of predictive modeling.
The model had learned structure.
And structure is exactly what compression exploits.
This is much deeper than “AI compresses the internet”
There is an important caveat here.
It would be misleading to say:
“GPT is just a compressed copy of the internet.”
That slogan is catchy.
It is also too simplistic.
A language model does not function like a conventional archive.
If you put a 100-million-word document into ZIP, you can recover the exact document.
If you train a language model on that document, you cannot simply ask the model to reproduce the entire training corpus perfectly.
The model learns statistical patterns.
It generalizes.
It loses information.
It transforms information.
And, depending on the model and training data, it can sometimes memorize particular sequences.
So the relationship is not:
LLM = ZIP file.
The better statement is:
LLM training and lossless compression share a mathematical foundation in probabilistic prediction.
That distinction matters.
Why does compression require something that looks like understanding?
Now we reach the philosophical heart of the story.
Suppose I give you a completely random sequence:
X7Q9L2M4Z8...
There may be little structure to exploit.
But give you a Wikipedia article.
Suddenly there are patterns everywhere.
English grammar.
Common words.
Names.
Dates.
HTML structures.
Programming syntax.
Repeated concepts.
Causal relationships.
Facts.
Formatting.
Human conventions.
The more structure your model understands, the more accurately it can predict what comes next.
And the more accurately it predicts, the fewer bits it needs.
That is why Hutter's argument is so interesting.
He wasn't claiming that compression is identical to intelligence.
He was proposing that compression provides a concrete mathematical proxy for some of the capabilities associated with understanding and prediction.
It turns an abstract question into an information problem.
Bellard's later work makes the story even stranger
Bellard didn't stop at NNCP.
His ts_zip utility uses a large language model for practical text compression.
The current implementation uses an RWKV 169M model, quantized to eight bits, and an arithmetic coder driven by the model's next-token probabilities.
Bellard reports, for example, that enwik9 compresses to about 135.4 MB, or roughly 1.084 bits per byte, using ts_zip.
For comparison, the same page reports about 213.4 MB for xz on that file.
Again, there are important differences between benchmarks, model-weight accounting and competition rules.
But the basic demonstration is spectacular:
a language model can function as a compressor.
Not metaphorically.
Actually.
It predicts.
An arithmetic coder turns those predictions into bits.
And the original text comes back.
The arithmetic coder is the missing piece most AI explanations leave out
This is worth understanding.
A neural network alone doesn't magically shrink a file.
It produces probabilities.
Suppose the model says:
- A: 50%
- B: 25%
- C: 12.5%
- everything else: 12.5%
A coding algorithm can exploit those probabilities.
The highly probable symbols receive shorter effective descriptions.
The unlikely symbols receive longer ones.
Arithmetic coding is one of the classic mechanisms for doing this.
So the complete system is:
data → neural prediction → probabilities → arithmetic coding → compressed data
And decompression reverses the process:
compressed data → same probabilities → recovered symbols → original data
Bellard's paper describes precisely this predictive-coding mechanism.
This is why the quality of the prediction model matters so much.
A terrible predictor gives you little compression.
A superb predictor gives you much more.
Suddenly, “scaling” has another interpretation
Modern AI has spent enormous amounts of money making models larger.
More parameters.
More data.
More compute.
Longer contexts.
Better training.
Why does this work?
Because larger and better-trained models can often capture more structure in the data.
From the compression perspective, that means something very concrete.
They become better predictors.
And better predictors can potentially encode information more efficiently.
This gives another interpretation of the famous scaling curves.
A falling language-model loss isn't just a line going down on an AI research dashboard.
It represents improved probabilistic prediction.
And probabilistic prediction has an information-theoretic price measured in bits.
The connection is not a metaphor.
It is mathematical.
The really strange part: generation and compression are connected too
The DeepMind paper went further.
If you have a probabilistic model that predicts the next symbol, you can use those probabilities not only to compress data but to generate it.
Compression and generation therefore become two sides of the same model.
During compression:
Tell me what probably happened.
During generation:
Tell me what will probably happen next.
The probability distribution is the bridge.
This is one reason the connection between language modeling and compression is so fundamental.
The same predictive machinery can be pointed in two directions.
Bellard saw something the AI industry would eventually rediscover
This is perhaps the most compelling part of the story.
The mainstream AI industry spent the late 2010s and early 2020s building increasingly enormous predictive models.
Meanwhile, a software engineer with a long history of writing extraordinarily efficient systems was asking:
What if the predictor itself is the compressor?
He wasn't trying to build the next chatbot.
He wasn't trying to generate viral images.
He wasn't trying to create an AI assistant.
He was trying to save bytes.
And that forced him to confront something fundamental.
Prediction is compression.
If you know enough about the data, you don't have to describe every part of it independently.
You describe the unexpected parts.
The predictable parts become cheap.
That is what intelligence, in one information-theoretic interpretation, begins to look like.
From FFmpeg to neural compression
There is also something fitting about Bellard being the person to make this connection practical.
FFmpeg became foundational because Bellard understood something software engineers often forget:
efficiency matters.
QEMU virtualizes entire computer architectures.
QuickJS brings a JavaScript engine into tiny environments.
His projects repeatedly push complicated computing tasks into surprisingly compact implementations.
Neural compression fits that philosophy.
Instead of throwing enormous amounts of compute at a problem without worrying about efficiency, Bellard asked whether learned prediction could produce a better representation of information.
The result was not a chatbot.
It was a compressor.
But the underlying mathematics was moving toward the same territory.
The billions spent on AI did not invent this idea
This is perhaps the most important historical correction.
The relationship between prediction, probability and compression is much older than GPT.
The theory of information predates modern deep learning.
Arithmetic coding predates Transformers.
Neural language modeling predates GPT.
Hutter's compression-based AI argument dates to the 2000s.
Bellard's neural compression experiments date to 2019.
What changed in the GPT era was scale.
The industry discovered what happens when predictive modeling is given enormous datasets, huge neural networks and enormous amounts of computation.
The resulting systems became capable enough that the old compression perspective suddenly became relevant to a much broader audience.
DeepMind's 2023 paper effectively put a modern mathematical sign above the door:
Language Modeling Is Compression.
And this may explain why “understanding” remains such a difficult word
People often ask whether AI models understand what they read.
Compression offers an interesting way of reframing the question.
Instead of asking:
“Does the machine understand this sentence?”
ask:
“What regularities about the world must the machine have learned to predict this sentence accurately?”
The answer can be surprisingly deep.
To predict a sentence about politics, you may need knowledge of institutions.
To predict a medical sentence, you need knowledge of biology.
To predict computer code, you need knowledge of programming syntax and conventions.
To predict a physics explanation, you need patterns associated with physical relationships.
As models improve, they become capable of exploiting more of those regularities.
That does not settle the philosophical question of consciousness or genuine understanding.
But it gives us a measurable engineering question:
How much structure can the model extract from the data?
Compression gives us one way to measure that.
The future may belong to models that understand structure, not merely text
The DeepMind experiments are particularly interesting because Chinchilla's compression ability wasn't restricted to ordinary language.
The model trained mainly on text nevertheless produced useful predictions over representations of images and audio.
That suggests something important about foundation models.
They may be learning increasingly general statistical machinery.
Language is one manifestation.
Images are another.
Audio is another.
Code is another.
The common denominator is structure.
And wherever there is predictable structure, there is potentially something to compress.
The quiet revolution hiding underneath AI
The popular history of modern AI is usually told as a sequence of increasingly powerful models:
GPT-2.
GPT-3.
ChatGPT.
Transformers.
Multimodal models.
Reasoning models.
Agents.
But there is another history running underneath all of it.
Information theory.
How much information is actually present?
How predictable is it?
How efficiently can it be represented?
How much structure can a model discover?
How many bits are required to describe what happened?
Those questions were being asked long before anyone was posting AI-generated pictures on social media.
And they remain fundamental to what today's models are doing.
The one-GPU lesson
Perhaps the most humbling part of Bellard's story is the scale.
His 2021 Transformer compressor was designed to run on a personal computer with a single GPU.
Meanwhile, the modern AI industry is spending billions on data centers containing enormous numbers of accelerators.
That doesn't mean Bellard “solved AGI on one GPU.”
He didn't.
Compression and frontier language modeling are different engineering problems.
But it does demonstrate something important:
The fundamental idea does not require a trillion-dollar industry to exist.
The industry scaled the idea enormously.
Bellard isolated one of its mathematical cores and pushed it in another direction.
The bigger lesson for AI
Perhaps the most useful way to think about an LLM isn't:
“It's a machine that writes text.”
That's the interface.
Underneath it is a predictive model.
It has learned statistical relationships.
It estimates probabilities.
It predicts what comes next.
Those predictions can generate language.
They can also be turned into compressed representations.
And, as the DeepMind research demonstrated, the same predictive machinery can sometimes discover useful structure in data outside the domain on which it was primarily trained.
That doesn't mean compression proves intelligence.
It doesn't prove consciousness.
It doesn't prove that an LLM understands the world in the same way a human does.
But it reveals something much more concrete.
The mathematics underneath modern AI has a surprisingly old soul.
Before the chatbots.
Before the billion-dollar AI startups.
Before the GPU arms race.
There was a simple question:
Can you predict what comes next?
If you can, you may be able to generate it.
If you can predict it extremely well, you may be able to compress it.
And if you can compress enormous amounts of structured information because you have learned the patterns underlying it, you have begun to do something that computer scientists have associated with intelligence for decades.
Fabrice Bellard approached that idea from the world of compression.
The modern AI industry approached it from language.
They met in the same place.
Prediction.
And perhaps that is the deeper story of the Transformer revolution:
We thought we were teaching machines to write.
In another sense, we were teaching them to predict the structure of information itself.
Comments
Post a Comment