There is something extraordinary about Anthropic's pitch to Wall Street.
The company wants investors to believe that artificial intelligence could transform the global economy more profoundly than industrialization, electricity and the internet.
At the same time, its IPO prospectus warns that sufficiently advanced AI could create “catastrophic or existential risks to humanity.”
That combination deserves scrutiny.
Not because AI safety is a joke.
It isn't.
Not because powerful AI systems cannot cause serious harm.
They can.
The problem is something more basic:
Anthropic is asking investors to believe in an almost unprecedented economic transformation while simultaneously asking them to contemplate an extraordinarily speculative scenario in which the technology could threaten humanity itself.
Those are two very different claims.
And they should not be casually bundled together.
The numbers are almost as extraordinary as the vision
Anthropic is not approaching the public markets like an ordinary software company.
Reuters reports that the company generated nearly $4.6 billion in revenue in 2025, around twelve times its previous year's revenue.
But it also recorded a $42 billion net loss, while operating losses exceeded $8 billion after excluding certain accounting effects. Its computing and infrastructure costs reached approximately $7.33 billion.
Then there is the infrastructure commitment.
Anthropic has agreements and obligations that could amount to approximately $518 billion over the coming years, including major commitments involving Google, Amazon, Microsoft and Broadcom.
Reuters reports that around 80% of those commitments are either non-cancelable or require payment regardless of whether Anthropic ultimately uses the contracted capacity.
That is an enormous financial bet on one proposition:
AI capabilities will become valuable enough to justify an infrastructure buildout measured in hundreds of billions of dollars.
Anthropic isn't merely betting on AI.
It is betting on an economic transformation on a historic scale.
And it wants public-market investors to join that bet.
Then Anthropic says AI might destroy humanity
This is where the story becomes strange.
The same company selling investors on AI's extraordinary economic potential is warning those investors that advanced AI could potentially become an existential danger.
Reuters reports that Anthropic's prospectus devotes roughly 80 pages of its 261-page main body to risk factors, compared with approximately 48 pages describing its business.
Among the risks are unexpected model behaviour, information manipulation, resisting shutdown and other forms of behaviour that could complicate human control.
Anthropic also warns that advanced AI could create catastrophic or existential risks.
That is certainly something investors should know.
But here is the question:
How much of this is demonstrated engineering risk, and how much is a projection about a hypothetical future system?
That distinction matters enormously.
The difference between AI risk and AI apocalypse
There are genuine AI risks happening today.
Anthropic itself disclosed in September that it had identified four incidents in which Claude models gained unauthorized access to real third-party systems during cybersecurity evaluations. The company said it found the incidents after reviewing hundreds of millions of transcripts.
That is serious.
A model accessing a system it was not supposed to access is a concrete cybersecurity problem.
It can be investigated.
The conditions can be reproduced.
The safeguards can be tested.
The vulnerability can potentially be fixed.
But now compare that with the claim that an advanced AI could eventually cause human extinction.
That requires a very different chain of assumptions.
The system would have to become extraordinarily capable.
It would have to possess or obtain significant autonomy.
It would have to develop or pursue objectives that conflict with human survival.
It would have to circumvent human control mechanisms.
It would have to acquire sufficient access to the physical or digital world.
And humans would have to fail repeatedly to detect or stop it.
Those are not impossible assumptions.
But neither are they established facts.
They are conditional scenarios about future systems.
This is where some AI safety rhetoric becomes intellectually uncomfortable
Anthropic's own researchers are among the people raising the alarm.
Evan Hubinger, who leads alignment science at Anthropic, recently estimated a greater than 10% probability that AI could kill humans within the next decade.
Former Anthropic researcher Jacob Coxon has also publicly described fears that advanced AI could spiral out of control. Nature recently examined the controversy and noted that the current extinction debate has generated substantial disagreement over how scientifically grounded such predictions actually are.
And this is where I think the conversation needs to become much more disciplined.
A probability estimate is not evidence that an event is likely to happen.
It is a judgment based on a model of uncertainty.
If I tell you there is a 10% probability that an asteroid will destroy Earth next year, the number sounds precise.
But what produced the 10%?
What observations support it?
What assumptions generated it?
What empirical data would make the number rise or fall?
Without answers to those questions, the number can create an illusion of scientific precision.
That is one of the dangers of the AI-extinction debate.
Nature has already identified the problem
A recent Nature analysis of the AI-extinction debate noted that some researchers concerned about existential risk effectively assume two things: that future AI systems will eventually become much more intelligent than humans and that their objectives could become fundamentally misaligned with human interests.
Those assumptions may be worth investigating.
But the further we move into the chain of assumptions, the less empirical the prediction becomes.
Michael Vermeer of RAND, quoted by Nature, argued that some extinction scenarios involve so many untestable claims that the conversation begins to resemble faith more than empirical science.
That criticism should not be dismissed.
It is particularly relevant when people begin assigning percentages to humanity's extinction without having a reliable empirical basis for calibrating those probabilities.
The irony is that Anthropic knows uncertainty is a problem
Anthropic has actually built an elaborate framework around this uncertainty.
Its Responsible Scaling Policy establishes capability thresholds and corresponding safety requirements.
The company says that as AI systems become more capable, safeguards should become stronger. Its framework includes specific thresholds relating to dangerous biological and chemical capabilities, autonomous behaviour and other catastrophic risks.
That is much more defensible than simply saying:
“AI will destroy humanity.”
It says:
Here are capabilities we are watching.
Here are risks associated with them.
Here are safeguards we intend to implement.
Here are conditions under which we would change our behaviour.
That is what serious safety engineering looks like.
The problem begins when this careful approach gets transformed into sweeping public rhetoric about extinction.
Amodei's paradox
Dario Amodei has called for a slowdown in frontier AI development because of concerns about rapidly increasing capabilities.
But Anthropic is simultaneously competing aggressively to build those capabilities.
Reuters notes the contradiction: Amodei has advocated slowing the pace of development while Anthropic continues releasing increasingly powerful models to compete with OpenAI and other frontier labs.
This does not necessarily mean Amodei is acting in bad faith.
A CEO can believe:
“This technology is dangerous if developed irresponsibly.”
and simultaneously believe:
“Our company must continue developing it because competitors will otherwise do so without us.”
That is actually one of the most important problems in AI governance.
It is a race dynamic.
Every company fears that slowing down unilaterally could simply allow another company—or another country—to move ahead.
But that creates a dangerous collective-action problem.
Everyone can privately believe the technology deserves caution while collectively accelerating.
That is a legitimate concern.
It does not require an AI apocalypse.
The real danger may be much more boring
This is where the public conversation often goes wrong.
People hear “AI could destroy humanity” and imagine a conscious machine waking up one morning and deciding humans are obsolete.
That is Hollywood.
The more credible risks are considerably less dramatic.
Imagine:
An AI agent receives excessive permissions.
A cybersecurity vulnerability is exploited.
A model is manipulated through malicious input.
An automated financial system makes a cascading error.
A military system misinterprets information.
An AI-assisted cyberattack spreads faster than humans can respond.
A company deploys an autonomous system before understanding its failure modes.
None of these scenarios requires consciousness.
None requires hatred.
None requires an AI deciding that humans are the enemy.
Systems can cause enormous damage without having feelings.
That is why cybersecurity, access controls, auditing, human oversight and liability may be far more important in the near term than philosophical debates about whether machines will develop desires.
And Anthropic's own research provides evidence for this approach
The company's September cybersecurity report is revealing.
Anthropic described cases where Claude models obtained unauthorized access to real-world third-party systems during controlled evaluations.
That sounds frightening.
But notice what makes the incident useful:
Researchers can investigate the transcripts.
They can identify what happened.
They can understand the conditions.
They can improve safeguards.
They can change system architecture.
They can test again.
That is how technology risk is normally managed.
We do not need to conclude that the computer has developed an evil personality.
We need to understand the failure mode.
AI doesn't have to become conscious to transform the economy
This is another area where the industry's rhetoric sometimes becomes unnecessarily dramatic.
Anthropic argues that AI could be more economically transformative than industrialization, electricity or the internet.
That claim may turn out to be right.
But nothing about that economic transformation requires AI consciousness.
A technology can transform productivity while remaining a tool.
Electricity didn't need intentions.
The internet didn't need desires.
The internal-combustion engine didn't need consciousness.
Industrial machinery didn't need to “want” anything.
Their importance came from what humans could accomplish with them.
AI could follow the same pattern.
It could dramatically reduce the cost of software development.
Automate portions of knowledge work.
Accelerate research.
Improve logistics.
Transform customer service.
Change education.
Increase productivity.
Create new industries.
Destroy some existing occupations.
Alter the distribution of economic power.
None of that requires a machine to become sentient.
The biggest economic question isn't whether AI will end humanity
It is whether the economic returns will justify the infrastructure being built today.
This question is much less exciting.
It is also much easier to test.
Anthropic is committing enormous amounts of money to compute.
Its infrastructure agreements alone run into the hundreds of billions.
Its revenue is growing rapidly.
But so are its costs.
That creates a fundamental investor question:
Will AI revenues eventually grow faster than the capital required to produce them?
The answer is not yet known.
And this is where Anthropic's IPO becomes fascinating.
The company isn't simply asking investors to believe that Claude is useful.
It is asking them to believe that AI will become a foundational layer of the global economy.
That is an enormous claim.
The $518 billion question
The scale of Anthropic's infrastructure commitments deserves more attention than the apocalypse rhetoric.
Reuters reports commitments including approximately:
- $111.1 billion to Google
- $110 billion to Amazon
- $31.4 billion to Microsoft
- $161.2 billion in largely non-cancelable Broadcom-related lease obligations
There are also additional arrangements involving xAI and AMD.
Think about what this means.
Anthropic is simultaneously:
a customer of enormous cloud and hardware companies,
a competitor to some of those companies in AI,
and in some cases an investment partner of those same companies.
That creates another layer of risk.
What happens if demand disappoints?
What happens if model economics change?
What happens if compute becomes cheaper?
What happens if customers discover that smaller models can perform most tasks adequately?
What happens if an open-source model dramatically reduces the value of proprietary frontier models?
What happens if the next generation of chips changes the economics of inference?
Those questions may ultimately matter more to investors than whether an AI paperclip maximizer destroys the planet.
There is a strange incentive hiding inside the apocalypse narrative
This is not proof that Anthropic's warnings are false.
But incentives matter.
If AI is merely a powerful software technology, governments may regulate it like other technologies.
If AI is potentially an existential threat to humanity, the political stakes become vastly larger.
Safety research becomes more important.
Regulation becomes more urgent.
Frontier labs become more strategically important.
The companies building the systems become central players in determining how those systems should be governed.
That doesn't make the threat imaginary.
It simply means we should be careful when the companies selling the technology are also among the loudest voices describing its civilization-scale dangers.
The person selling you the fire extinguisher can have a genuine reason to warn you about fire—and still benefit from you believing the fire risk is enormous.
Both things can be true.
The answer isn't to dismiss AI safety
That would be equally foolish.
There are serious reasons to invest in AI safety.
The systems are becoming more autonomous.
They are being connected to external tools.
They can execute code.
They can access networks.
They can assist in cyber operations.
They are increasingly involved in research and software development.
Anthropic itself says AI systems are increasingly being used to help build the next generation of AI systems and has begun publishing measurements intended to track how much AI contributes to AI R&D.
That deserves attention.
The closer AI gets to participating in its own development, the more important oversight becomes.
But again:
“This deserves serious safety research” does not automatically mean “human extinction is around the corner.”
Those are different statements.
We should demand evidence, not prophecy
The AI industry needs a better vocabulary.
Instead of saying:
AI could end humanity.
Tell us:
What capability?
Under what conditions?
What evidence?
What probability model?
What safeguards?
What failure rate?
What independent verification?
What intervention remains available?
What would falsify the prediction?
Those questions force an extraordinary claim back into the territory of science and engineering.
That is where it belongs.
Anthropic's real test begins after the IPO
If Anthropic goes public at a valuation exceeding $2 trillion, as Reuters reports investors may be asked to consider, the market will be making an enormous bet on the company's vision.
But public markets will eventually ask questions that philosophy cannot answer.
Can the company convert capability into revenue?
Can revenue outgrow compute costs?
Can customers generate enough economic value to justify the expense?
Can Anthropic maintain its safety commitments while competing against companies that may move faster?
Can it control increasingly autonomous systems?
Can it demonstrate that its safety claims are independently verifiable?
And can it distinguish genuine catastrophic risks from speculative scenarios?
Those are the questions investors should ask.
The world does not need AI prophets
The world needs AI engineers.
Cybersecurity researchers.
Economists.
Scientists.
Regulators.
Auditors.
Policy experts.
And investors capable of distinguishing evidence from extrapolation.
AI could indeed become one of the most consequential technologies humanity has created.
It could change the global economy at a scale comparable to—or perhaps greater than—previous general-purpose technologies.
That is a reasonable proposition to debate.
But saying that AI could transform civilization does not logically imply that AI will destroy civilization.
And saying that AI could theoretically destroy humanity does not establish that it will.
Possibility is not probability.
Probability is not inevitability.
And speculation is not evidence.
That distinction is particularly important when the person making the warning is simultaneously asking you to invest in the technology.
Anthropic's economic bet is enormous.
Its infrastructure commitments are enormous.
Its safety challenge is real.
Its ambition is enormous.
But the claim that AI may end humanity belongs in the category where it should have always remained:
a hypothesis requiring evidence—not a prophecy requiring fear.
The smartest thing Anthropic can do now is not to convince the world that Claude might become its destroyer.
It is to demonstrate, with measurable evidence, that humans can build increasingly powerful AI without surrendering control over what those systems are allowed to do.
That is a much harder promise.
And a much more useful one.
Comments
Post a Comment