There is something striking about the latest AI safety debate: the people warning that artificial intelligence could become an existential threat to humanity are increasingly proposing that they—and people from their own intellectual and institutional networks—should help police the technology.
That is the central issue raised by the controversy surrounding Anthropic CEO Dario Amodei's latest proposal.
Amodei has called for AI companies to slow the development of increasingly powerful systems and has proposed independent external evaluators with access to frontier AI laboratories. He has also advocated international coordination on AI safety.
The New York Post's framing is far more provocative, describing the people around this emerging AI-safety establishment as Amodei's "handpicked" globalists and questioning whether organisations such as the Model Evaluation and Threat Research (METR) can genuinely be considered independent when they operate within the same intellectual ecosystem as Anthropic and parts of the effective-altruism movement.
And that raises a much bigger question:
Who gets to decide what humanity should fear about AI—and who gets to decide what humanity is allowed to build?
The AI apocalypse has become an argument for governance
Amodei isn't simply warning that AI could cause problems.
His argument goes much further.
He has repeatedly described scenarios in which increasingly capable AI systems could enable biological weapons development, cyberattacks, authoritarian surveillance, mass manipulation and potentially catastrophic loss of human control.
In his recent push for slowing frontier AI development, Amodei has proposed external monitoring, common safety standards and international cooperation.
Those proposals sound reasonable on their face.
If a technology becomes extraordinarily powerful, why shouldn't someone independently inspect it?
But the difficult question begins immediately afterward:
Independent according to whom?
If the same small network of AI companies, researchers, investors, philanthropists and ideological organisations keeps producing the people qualified to judge AI safety, independence becomes considerably harder to define.
That is the uncomfortable issue behind the controversy.
Enter METR
One organisation at the centre of the debate is the nonprofit Model Evaluation and Threat Research, commonly known as METR.
METR evaluates advanced AI systems for dangerous capabilities. Amodei has pointed toward organisations of this kind as potential external monitors capable of evaluating frontier AI systems from inside or alongside AI companies.
The concept is straightforward:
AI companies should not necessarily be trusted to grade their own homework.
An external organisation could test models for dangerous capabilities and report what it finds.
But critics have questioned METR's independence because of connections between the broader AI-safety ecosystem, effective altruism, Anthropic personnel and people who have funded AI-safety organisations.
METR disputes the suggestion that it is simply an arm of the AI companies. The organisation says it does not accept funding from AI companies and emphasises transparency around its work.
That distinction matters.
There is a difference between being connected to a particular intellectual ecosystem and being controlled by it.
But the fact that the question is being raised tells us something important about where AI governance is heading.
The rise of the AI priesthood
For years, technology companies sold the public a relatively simple message:
Build.
Scale.
Innovate.
Disrupt.
Now some of the same companies are increasingly telling governments and society:
Slow down.
Regulate.
Monitor.
Coordinate.
Create international institutions.
That represents a remarkable transformation in Silicon Valley's relationship with government.
The people who built the technology increasingly want a seat at the table determining how that technology should be governed.
And there is an obvious paradox.
If a company becomes one of the world's most powerful developers of artificial intelligence, it gains enormous influence from accelerating technological progress.
But if it can simultaneously persuade governments that uncontrolled competition is dangerous, it can also gain influence from controlling the rules around that competition.
This doesn't prove that AI safety proposals are a cynical attempt to eliminate competitors.
It does mean that AI safety policy has economic and geopolitical consequences, and those consequences deserve scrutiny.
The China question makes this even bigger
Amodei's position is not simply about domestic regulation.
He has argued that democracies should maintain leadership in advanced AI while also pursuing international arrangements around safety. His recent proposal has included cooperation with China on certain AI risks.
That creates an extraordinary geopolitical contradiction.
The United States and China are engaged in a technological competition over semiconductors, computing infrastructure, AI models and military capabilities.
At the same time, AI executives are asking governments to coordinate internationally to control the risks created by the technology.
You therefore have two competing ideas operating simultaneously:
AI is too strategically important to lose the race.
And:
AI is too dangerous to allow the race to continue unchecked.
Both arguments can be made simultaneously—but the tension between them cannot simply be wished away.
Who gets excluded?
There is another question that receives considerably less attention.
Suppose governments adopt the safety framework proposed by leading AI companies.
Who benefits?
The largest laboratories already possess enormous computing resources, researchers, infrastructure and capital.
A regulatory system requiring expensive evaluations, specialised safety teams, reporting requirements and external monitoring could be manageable for companies worth hundreds of billions of dollars.
For a small startup?
It could be devastating.
For an open-source developer?
Potentially worse.
The paradox is that regulation designed to make AI safer could also make access to frontier AI development more concentrated.
The companies that can afford the compliance infrastructure become even more powerful.
That doesn't make regulation inherently wrong.
It means the structure of regulation matters.
A world in which only a handful of corporations can afford to develop advanced AI is not automatically a safer world.
It may simply be a world in which fewer people control more powerful technology.
And then there is Africa
This conversation becomes particularly important outside Silicon Valley.
Much of the global AI debate is being shaped in Washington, San Francisco, London and Brussels.
But AI will be deployed across countries that have very different levels of technological infrastructure, regulatory power and political influence.
If an international AI governance system emerges from Western technology companies and Western governments, African countries could find themselves primarily as rule-takers rather than rule-makers.
That would repeat an old pattern.
Technology is developed elsewhere.
Standards are established elsewhere.
Data is collected everywhere.
And developing countries eventually receive the finished system and the rules attached to it.
The AI era offers an opportunity to break that pattern—but only if countries outside the major technology powers participate in determining how AI is governed.
Otherwise, "global AI governance" could become another phrase for rules designed by a relatively small group of powerful institutions and exported to everyone else.
The real danger may be concentrated power
The most interesting part of this debate isn't whether an AI apocalypse happens in 2030, 2040 or never.
Nobody knows.
Even experts disagree dramatically about the probability and timeline of catastrophic AI scenarios. Recent reporting has highlighted both serious warnings from AI leaders and researchers who argue that some apocalypse scenarios remain highly speculative and face significant real-world barriers.
The more immediate question is much easier to understand:
Who controls the technology while everyone is arguing about its future?
That question doesn't require science fiction.
It already exists.
A handful of companies possess enormous computing resources.
A small number of researchers influence the direction of frontier-model development.
Private investors finance the laboratories.
Governments increasingly depend on these companies for technological expertise.
And now some of those same companies are helping define the institutions that could determine how the technology is regulated.
That concentration deserves attention regardless of whether one believes the AI-apocalypse scenario.
From AI developers to AI governors
This is perhaps the most important shift.
The AI industry began by asking:
What can we build?
It increasingly asks:
What should humanity be allowed to build?
That is an enormous change in political and economic power.
Dario Amodei's warnings should therefore not be viewed merely as predictions about machines becoming superintelligent.
They are also arguments about human institutions, international governance and who should have authority over frontier technology.
The debate over METR, effective altruism and the broader AI-safety ecosystem is ultimately a debate over legitimacy.
Who watches the watchers?
Who chooses the watchers?
Who finances them?
Who defines "dangerous"?
Who determines which risks deserve immediate intervention?
And who ensures that AI safety does not become a mechanism through which today's technology giants acquire even greater control over tomorrow's technological ecosystem?
Those questions are more concrete than an AI apocalypse.
And unlike the hypothetical superintelligence that may or may not arrive, the concentration of technological power is already here.
The real story, therefore, may not be about a machine suddenly taking control of humanity.
It may be about humans gradually deciding that the people who control the most powerful machines should also have an unusually large role in deciding how everyone else is allowed to use them.
Comments
Post a Comment