The gap between China's chip manufacturing capabilities and the USA's is only going to shrink, right? ASML obeys some export controls for their most sophisticated machines, but those machines are in China's backyard (Taiwan).
Taiwan manufactures the world's most advanced chips. CCP wants "re-unification" with Taiwan. AI may be THE key to world dominance. These are scary times.
Are guard rails meaningful if they can be removed from the weights? Can America even prevent the release and proliferation of these models?
It seems obvious to me that the whole question of regulating a file is a bit silly. Any law that pushes against these things will just make it more secretive. I'm not sure that's any better.
> In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression.
Aren't Anthropic models used in project maven: https://en.wikipedia.org/wiki/Project_Maven ?
he's asking for the most powerful models to be in the hands of the few while the majority will be at their mercy.
Statement is a whole lot of nothing, as expected, but I also don’t know what people are expecting from these guys. That Dario will have a sudden change of heart and publish weights of all his models, flushing $1T down the drain?
distillation: Pirates people’s lifetime of copyrighted work, makes billions selling access to it through APIs, then tells us we can only use it in ways they approve.
I'm curious if he's also in favor of banning biology books.
As an American I’d gladly take payments from China to feed them my Claude transcripts for distillation. I’m surprised I haven’t heard of such an initiative.
I wish we had a good LLM developer toolset that is not Anthropic or OpenAI with sensible business and ethical practices and good performance.
Can't wait for local on machine LLMs that are on par with Opus/Fable.
oh, it‘s the CEO of a well known AI company educating us all – and all he wants is us to hear his hunch on open weight models?! Amazing, please help us understand the situation a bit better, thanks
If your concern is that China will develop models that are significantly more powerful than those of the US, why would you care so much about distillation? It seems like distillation is a way to catch up on capabilities, but not so much a way to jump ahead in capabilities.
Anthropic starting to sound like Microsoft from the 1990s.
This seems hypocritical, out of touch, and hollow. "Safety testing" is just a hair away from censorship when it comes to government control.
"In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression."
I'm so sick of all this anti-China shilling. There's zero chance that whomever is in power in the U.S. won't use AI in drones and in FBI/CIA/local Police/etc., for surveillance and repression right here in the good old U.S.A too. These government use cases for AI are both sides of the same coin.
China fear-mongering by business leaders only happens from businesses that have something to gain by it. Obviously, Anthropic fits the bill in this regard.
Talk about a giant whiff.
I was hoping they would announce their first open weights model, perhaps an older model they don’t offer anymore, but no. Instead he get this bs statement that reeks of “dam it I’m so close to being a billionaire” desperation. Not even acknowledgement of how much data they stole from others yet he whines about distilling.
It’s like his goal in life is to be a Scooby-Doo villain.
Anthropic (and some others) should or will soon learn how to do less pontificating and more engineering. They are in no position to be an arbiter of things, although for sure they can and should voice an opinion. If we collectively decide AI is a dangerous tool, company making it is not the arbiter. Governments are.
It's like hearing Smith & Wesson opine on the policies.. oh, wait.
Demand #1 weakens america and everyone around them
Demand #2 Why does this matter? The answer was that it does not. (https://news.ycombinator.com/item?id=49007610)
Demand #3 This doesn't exist. You cannot have 'safe' opensource models, it's simply impossible. You can always post train sufficiently capable models to become 'unsafe'. The flip side of that is that sufficiently capable models are banned therefore it is a ban on open intelligence completely defeating the point of this entire manifesto.
The Roman empire fell for the same reason, didn't it!? They wanted to control it all ... But it was too big
Finally, some sense. This is the only argument I have seen that genuinely engages with the problem and approaches it with humility, rather than charging ahead on the basis of assumptions and without a shred of evidence. OpenAI should have been the one making it.
“Questions like this should be answered empirically through rigorous pre-release testing, not assumed in advance.”
Exactly.
Do distilled models have to inherit guardrails? Can a model distilled from a “safe” model be made unsafe?
We distilled all the proprietary material into our token-based money making machine that is more expensive on every new release, but "we should crack down on industrial-scale distillation operations".
> "We should crack down on industrial-scale distillation operations" I'd prefer if there was a crack down on industrial-scale scraping. Maybe even reimbursement for the problems it has caused (some nasty AWS bills, tons of man-hours spent on preventing new nasty AWS bills, etc).
Reads like fearmongering about oppressive and violent applications of AI that the Chinese government hypothetically could deploy, which the US government is already actively working on in the meantime.
> Anthropic has never advocated for a ban on open-weights models.
If you wonder why this is written as an opening to a list of reasons that advocate for banning the open weight models, it’s because
I'm cynical enough that I would suspect all of his statements are duplicitous anyway, but my recent experiences with Claude Fable give weight to it.
I asked a question about a series of tokens - bam, denied and downgraded. There's no cyber security or public risk here, but Fable doesn't want me to learn how things work.
I asked a question about quantization in models - bam, denied and downgraded. I edit my question to make it clear I'm talking about Google's Gemma QAT models. Oh, that's fine then, and it answered the question helpfully.
Anti-competitive bullshit. I hope they fail.
1. We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling and workarounds used to obtain access to such chips.
Regulate others, but not us, please. And f.u. Jensen for your tweet.
2. We should crack down on industrial-scale distillation operations.
Boogeyman to still not allow Chinese models but pretend to support open-weights. Also, please ignore our distillation of research, illegally. That's different!
3. All sufficiently capable models, open and closed, should go through mandatory safety testing.
...That we author. Oh, and please ignore our own easing-of-guardrails when it comes to money: https://x.com/NoahLebovic/status/2081277517709922501
“By virtue of being the good people everything we do is good, and if it happens to be in our best personal and financial interest then that is good too because it enables us to do more good. And if it’s to the detriment to others then it’s because you don’t understand the good behind it. Which is fine, because we’re good and you can trust that what we do is good because what we do is good by virtue of us being the good ones.”
"All sufficiently capable models, open and closed, should go through mandatory safety testing."
I love how they invoke fear of "terrorism" to justify their oppressive position.
Anthropic would love the US to do everything in this list under the guise of "safety testing":
This reads almost dishonestly.
> Anthropic has never advocated for a ban on open-weights models.
This is not an unqualified never. The very next sentence makes a qualified statement: "Open-weights models that don’t have dangerous capabilities are a public good". That prompts the question, what about ones which do have "dangerous capabilities"? Are they not a public good? If not, then should they be banned? Who gets to decide on the definitions of these terms?
As an European I really dislike this consistent anti-Chinese narrative.
It's not China starting a war every few years, now causing a global economic fallout in Iran, it's not China threatening to annex Greenland/Canada/Panama, it's not China attacking foreign countries and kidnapping their leaders, it's not China who has been found to spy and intercept the communications and movements of its citizens and its allies and their leaders for the longest time, it's not China bombing civilians or stopping countries from obtaining basics like food, gas or oil.
I'm not saying that China is a paradise and US is bad, nor the contrary. We could make similar lists about most of the biggest countries out there.
I'm simply stating that this never ending US exceptionalism "US has to be the first and at the frontier of military, technology and this and that, but does not need to comply with the rules of the institutions it itself created" was already sickening and annoying before, but increasingly malign in the last decade and strongly accelerating as of recently.
I miss the time US CEOs were globalists and used their influence to advocate for a simpler world.
"needs safety eval"
also anthropic
"we're upset were not being considered for military contracts"
come on, which is it? Is it all about saftey or is it that only US/Israeli ai is allowed to kill? Seems to me that the only real threat is to the techno fudalism OAi, Anthropic & co are trying to build.
Hey, no fair stealing the value thay we already stole fair and square!
> Open-weights models that don’t have dangerous capabilities are a public good
Note the hedging against 'dangerous capabilities'. Undoubtedly, all the useful ones trigger this condition in Anthropic's eyes. The rest of the post is filled with similar weasel-wording. Make no mistake, this absolutely confirms that Anthropic is against open models in the sense that any reasonable person understands them.
The way the rest of the post unabashedly appeals to the current US administration's China hysteria is hilarious, and not at all subtle.
I guess we'll see about all the doomsaying here, won't we? Kimi K3 is frontier-level, and there's no stopping it now. As far as the world is concerned, anyway. If the US wants to kneecap itself that's another matter.
This is more terrible PR for Anthropic.
I will be so happy if/when the AI bubble bursts and we don’t have to deal with Dario’s god complex anymore.
Yikes
> Open-weights models that don’t have dangerous capabilities are a public good
This statement (and the entire post) couldn't possibly be more two-faced.
Open-weights models by definition have "dangerous capabilities" (according to Anthropic's own definitions of "dangerous", not mine), you can't bake in guardrails that can't be finetuned out.
hahha
[dead]
[dead]
[flagged]
[flagged]
despite all evidence, this white guy thinks he is more responsible than both chinese entrepreneurs and chinese authorities
"'To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category. We should instead focus on keeping powerful chips out of authoritarian hands, stopping industrial-scale distillation, and requiring safety testing of all sufficiently capable models, open and closed." This is all I needed to know: Dario Amodei is a f*king lizard who wants to create the next AI oligarchy. Instead of just signing the letter, he's bitching and denying that he opposes open-source models. I wonder if any Anthropic employees actually disagree with him.
The ban on distillation seems hypocritical.
"F#$% you, I got mine!"
Knew Anthropic were the bad guys.
The mental gymnastics here are incredible.
What’s blocking Anthropic from fighting Chinese companies abusing their services? Why go nuclear against all open weight models? Testing and compliance is technically banning.
> Anthropic has never advocated for a ban on open-weights models.
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
lmao the sort of lies people come up with when their only business model is “the government picks me as the winner” are so funny
First thing I'm going to do with my uncensored Kimi K3 release is to engineer a virus that only targets gatekeepers in the AI field.
(obviously this is a joke)
Woof, talk about self aggrandizing.
Amazing how this company went from having such goodwill to hardly any. At the end of the day they want to make as much money as possible. Anthropic will have to lose a ton of their profitability if they compete with open source. I see them moving into the application layer, which they’ve already started doing. It’s clear open models are the future for the vast majority of use cases that don’t need frontier capabilities.
Generational good will loss from these guys.
“We care about safety so powerful open models are bad” - and what of OpenAI? What if Mythos got into the wrong hands? It’s a pretty weak argument
As an act of good faith and proof, they could stop spending the record-breaking lobbying money for couple years.
wanting to ban your competitors via regulatory capture via the trump admin of all things shows how little of a backbone anthropic has
ofc half of them are of the ai rationalist lesswrong crowd so i think they’ve always been a little of their rocker
"To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category." Translation: If it's so strong that it threatens my business, ban it.
"We should instead focus on keeping powerful chips out of authoritarian hands, " Translation: Let's kneecap competitors.
"stopping industrial-scale distillation" They stole the work of every book author, and now are trying to say their AI's output should be protected from competitors.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US,
Begging, ugly crying, spitting for that sweet-sweet regulatory capture. These nerds need to be bullied harder.
Seems pretty reasonable.
Boris here. This post does not give us good publicity so I will refrain from commenting. Please wait for another demo thread of a new feature of ours or AI appraisal blog post and I will gladly go into as much detail as I can to give us as much hype as possible!
> We should not sell powerful chips or chipmaking equipment to China
Yes, please. We don't know whether we'd have open weight models today, had the chip-prohibition not been in place. Nor would we see the more optimized models such as DeepSeek or qwen.
We also would not see new players entering RAM market after you and your pals in Silicon Valley hoarded the entire world's hardware.
So by all means, double, no, triple down on this.
> We should crack down on industrial-scale distillation operations
And let's apply this retroactively to Anthropic too. You industrial-scale-operation-distilled all of humanity's knowledge. Let's have some of that crack down on you too.
I was pleasantly surprised by this release and the tone at the very beginning, but quickly it is painfully obvious that it's blatantly requesting a ban on open-weight models. The absurd demand for some safety arbiter by World Police America is farcical.
Yeah, the rest of the world is going to bow out of your busted idiocracy, guy.
Further, Anthropic needs to can it with the horseshit distillation bullshit. No, you aren't really the secret sauce, and this is basically trying to con stakeholders by pretending that there really is a moat, only you just need to add more crocodiles.
A significant percentage of innovations in AI lately has come from China. China is now making their own seriously competitive hardware, and they can steal content just as effectively as Anthropic to train their models. Why wouldn't they be competitive?
The pathetic claim that if you just stop distillation and prevent hardware smuggling and Anthropic and OpenAI will have the same moat is delusional. I mean, more correctly it's simply fraudulent, and he clearly knows it's bullshit meant to convince much stupider people.
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."
This sort of stuff betrays a stunning lack of self awareness. The US are the worldwide risk. The US are the ones threatening allies and bombing 10+ countries. The US are the ones carrying out war criming and pillaging, pirating and burning? The US are the ones with the guy threatening to use nuclear weapons on a weekly basis.
If Anthropic remotely believed their bullshit, they would shut down today and burn the hard drives. But they don't, and the pathetic call out to Vance (please daddy, ban those dangerous models!) is deplorable garbage.
This ridiculous, shameless "note" has an audience of one: JD Vance.
Ironic to oppose giving AI tech to “authoritarian governments” while approvingly quoting the authoritarian-wannabe government of the US and framing that government as the good guys.
Frankly, why would I put much weight into what Anthropic say about open weight models?
> In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression.
Welcome to bizarro world!
Fist off: "the most dangerous model may be one that is trained in secret" <-- Says the guy that not only restricts commercial use for some of their models but develops them in utter secrecy. With the pretext of guardrails. Then show us the guardrails you really use by opening the weights.
Second: "use in drones [...] for surveillance and repression" <-- writes the King of FUD, as the US is an an active campaign with the help of their models. And/or OpenAI's.
I am very appreciative of the freedoms of the west but this type of hypocrisy and lack of self-awareness is bonkers and it should be called out.
"Nobody has the intention to build a wall" - Walter Ulbricht, June 1961, 2 months before building the berlin wall.
It's refreshing to see how there's almost no person in this thread who can't see the BS. All the goodwill that Anthropic could have had is basically gone. Anthropic is likely on the path of becoming the most hated company in the world.
So my question is: is this by design (they know nobody's buying this), or is Dario simply so out of touch with reality?
If it's the former, then why publish this?
[dead]
I always find it interesting when people choose to so carefully stress and spell out the words "Chinese Communist Party".
It's a bit like spelling out "Barack Hussein Obama". It's a dogwhistle.
Yes yes, it's still called the Chinese Communist Party, I know.
But since we are talking about a one-party authoritarian state with a hybrid economy that underwrites much of western prosperity (including by producing a large percentage of the components of the data centres Anthropic is dependent on), that has long-since abandoned many of the salient principles that mark it out as conceptually communist rather than totalitarian, and since we're talking about a man who runs a debt-ridden business in a country where the president is seemingly shaking down a 10% share of everything profitable for the state while running an entirely arbitrary tariff regime and suddenly calling anyone remotely left-winga Communist, the choice to spell out "Chinese Communist Party (CCP)" when he could just as easily and arguably more usefully and appropriately have written "Chinese government" is telling.
This is some ham-fisted Republican-fishing. He must really be worried Sam is Donald's favourite.
The only real surprise is he didn't illustrate it with a Silmarillion analogy.
[dead]
Their number one concern is about the the commies perpetrating deep repression of their own people! How noble! This whole time I thought it was because they wanted more money and power. I guess we should probably ban these open weight models.
[dead]
I think I lost some brain cells reading that copium. LLMs providing a “Permanent military advantage”.. dumb!!
Jesus Dario we get it man, you want clout for the IPO.
This constant whining from anthropic about distillation attacks continues to be rich given the amount of stolen data that went into any Claude variant.
[dead]
>My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
This reads like a satire. I know Dario isn't that dumb.
"China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips"
source: Trust me bro.
There are hundreds of articles showing that China have developed their own chips and have a massive manufacturing capacity. This blog post feels like is pondering to the brain dead Fox News audience.
TL;DR:
It is ok that we digest all information we can get, (il)legally and/or (a)morally because we are the good guys. Trust me bro.
It is not ok if others digest from us. They are bad guys. Ban them pl0x.
China this and China that. Playing right from the playbook..
[dead]
How about we don't let the leading model makers make the rules? How about we make AI models that were trained on public data public goods? Let's circle back on that, kthxbye.
[dead]
[dead]
This is a whole lot of words to say "we don't support open weight models".
It's so obvious they are hoping to regulate out their competition rather than compete
It's so ironic to me the way people will say "china should not have these chips" meanwhile they manufacture like 99% of all the electronics we have in the united states.
If you read between the lines, this piece is just “Oh my god we (OpenAI and Anthropic) accepted too much investment and are totally fucked if these open weight models are competitive, please rescue our equity bags by banning open weight models and ensuring we can charge the maximum possible price for tokens.”
> For example, I worry that biology will have a strong attacker-defender asymmetry, where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials,
If he had just left that bit out it wouldn't be so obvious that he's just clutching at straws at this point. In some twisted sense it's almost sad to see.
[dead]
> > All sufficiently capable models, open and closed, should go through mandatory safety testing.
The problem with this is the cycles required to abliterate a model is significantly less than the cycles required to train a model.
This is the biggest reason why I'm against locking these models down / preventing their use. It's just delaying things by ~3-6mo, while in the process preventing legitimate use and adding red tape overhead.
The distillation commentary is really crazy coming from a company that stole all it's training material.
Distillation has to be way more energy efficient and beneficial for the planet. But if they can figure out a way to ban it, go ahead, that’s not a regulation problem, it’s an Anthropic problem.
The danger of an authoritarian government having some AI is muted by everyone else having that same capable open model. The only authoritarians to fear are those that keep models private. What kind of chance did Estonia have it having their own AI model at the level of Fable without China donating Kimi to the world?
When they use ill gotten data for training their models the world's smallest violin plays when they complain about distillation attacks.
I think this letters tells us everything we need to know about the man. I've rarely seen so much flattery towards the current administration, contradiction, deflection, half-truths and hypocrisy on a single page. This man is scared of everyone: the current admin, the public, and the other companies promoting open-weights.
> Open-weights models that don’t have dangerous capabilities are a public good
Knives should only cut during the day, knives which cut at night are bad.
Dario, as your unpaid therapist I would tell you that models are a commodity and you are having a hard time coming to terms with it. You are doing everything except accepting it. It's a common defense mechanism, but as your unpaid therapist, i will tell you that it's not going to work. Your company will cease to exist or exist like how ferrari or buggati exist.
The concerns are legitimate but the proposals are nothing more than a stopgap solution.
If US wants to maintain engineering superiority, we needs to invest in it -- education, research and infrastructure. Bring in top researchers across the globe and not make it harder.
China is building infrastructure for the future generations and investing in growth sectors while the US is cutting of university grants and spending billions on a war without clear path to resolution.
It's so convenient that the US government already classified China as "authoritarian". If they hadn't, Dario would have to say “it’s a risk that other people build models more powerful than us”. I have to wonder what his response would be if for instance a lab in France or Germany came out with an open-weight model this good.
What Dario misses time and time again, is that people don't trust the US to create aligned AI anymore. His entire strategy rests on the assumption that the US (and their government) are exceptional.
This is clearly false to the rest of the world.
> Anthropic has never advocated for a ban on open-weights models.
Not even Anthropic's own Claude believes that.
> Open-weights models that don’t have dangerous capabilities are a public good.
Who decides what is dangerous and what isn’t? Lawmakers usually have the say but Anthropic can easily bribe… I mean lobby them to favor your viewpoint.
What does cracking down on distillation look like in practice? I imagine data retention would be a part of the strategy, like we saw with Fable?
It seems really hard to allow usage via API and prevent distillation. Maybe limiting usage to within a specific harness would help a bit more. But ultimately the only way to prevent it is by locking down models to trusted entities (like with Glasswing). But then the profit potential of a model is significantly reduced. It really puts the labs in a bind.
I am so surprised of Dario's inclination for centralization that obviously makes unsustainable and overleveraged governance systems. There is definitely mismatches over the principle of distribution of power...
Wow, so much cynicism and distrust in the comments, to the level of conspiracy theory, in my opinion. Yeah, this is the company that fairly recently refused to allow the government to use its models for autonomous weapons or mass surveillance, and refused to remove its guardrails.
Am not saying we should take what Dario is saying at face value, but he already has shown by his actual actions that he can be well intentioned. There might be elements of truth to what he’s saying.
Hate to break it to you Dario but the way things stand right now the world at large trusts the Chinese establishment a lot more than the American one.
> We should crack down on industrial-scale distillation operations.
IP for me, not for thee.
Not that I expected him to, but I note no acknowledgement that it took a non-locked-down open weight model (GLM) to stop the Hugging Face attack.
I'm less concerned that the attack was caused by a closed model, than I am that no closed model was willing to stop it.
The worst part is I'm confident Fable would have done a better job stopping the attack, but their 'guardrails' made it decide not to want to.
Unless of course, you pay up: "Anthropic GTM people used large comitted spend contracts as a prereq for lowering safeguards"
-Noah Lebovic, former Anthropic staff
Well their valuation is going down the drain so no wonder they don't like it. The cherry on top will be China developing their own chips and chip making tech.
Anthropic's fall from grace and mindshare seems rather accelerated.
I wonder if these rapid movements are going to be the norm now. I imagine there would be angry investors if this sort of thing happened with a public company.
I don't really see the link between use of AI and military superiority.
My current understanding is a lot of current US military problems are due to rare earths supply chains.
I don't see how AI would either help or hurt with that.
I'm tired of being strung along on these silly narratives. I can't wait for open-weight models to be deployed around the world just so people like Dario will shut up about the mystical levels of power these models have.
So your concern is safety, and you claim you are the only one that can give us safety but do so by keeping your product closed? Then how about you release the weights?
I think it's only fair to introduce this if you're willing to have a real skin in the game, otherwise that's just weakness disguised as principle.
[flagged]
Do you have some browser extension that replaces phrases with that? That quote doesn't exist in the piece.
Anthropic must be forced to open source all of the books they have scanned and burned.
For what? Would authors and publishers like that?
> where sufficiently capable models may be able to quickly weaponize pandemic-level viruses with widely available materials
if someone figures out a way to give an LLM full operational control over a virus lab, we've got a whole different set of problems than the ones Dario is describing
If someone gives LLM control of such lab my best guess is soon they have no lab... Actually giving LLMs control of virus lab might be best thing one can do for continued existence of humanity.
It's very hard to take his position seriously when he repeatedly refers to the Chinese Communist Party and specific Chinese ministries specifically, like some two bit China-watching cold warrior, instead of addressing China as a sovereign state actor. Imagine if any Chinese AI founder talked about the Democrats, the Republicans, about Trump or ICE etc. It's gauche.
As someone who isn't American, I'm amused when someone from the US talks about the terrors of China or some other nation having more power than them as being bad for the world. The way the US is currently threatening to invade and damage all its allies, well, the world is already bad.
China hasn't threatened to annex my country yet, at least.
Every single risk he identifies as a concern regarding China is exactly my concerns with the US having absolute control. Literally the exact same concerns
It's projection. These tech CEOs know what they are doing and it scares them that someone else might do the same.
Yeah. Exact same concerns here as a non-american. Their "national security" is a constant threat to the rest of us.
In the first paragraph,
> Anyone who has read my past writing should know that I don’t regard such bans as a useful measure,
Later (on banning chip sales to china)
> we should crack down on the rampant smuggling and workarounds used to obtain access to such chips.
If you truly believe that bans don't work, the same applies to hardware too.
Furthermore, Dario says later "To address these concerns, I do support the following three measures...": 1. ban chip sales to China 2. crack down on distillation 3. all capable models should go through mandatory safety testing
Just so happens that all these moves commercially benefit Anthropic. If Dario really wanted to make a point, it would land a lot better had Anthropic released a single open-weights model
TL;DR: We like money, and Anthropic should stay in a position to make loads of it despite any competition.
If this is about safety, am I being too naive & idealistic to think that a "Kamar-Taj" rule would solve some safety issues?
The "Kamar-Taj" rule is, no knowledge is forbidden, only certain practices. If a model gives you detailed instructions on how to kill all humans, the knowledge itself isn't the problem. The problem is the person who acts on it.
[dead]
It is obviously not going to be until some really bad series of cyberattacks or a chemical/bioweapon attack before anyone takes regulation of models seriously.
They are _obviously_ (please convince me otherwise) going to be capable of carrying these terrible things out almost completely autonomously at some point in the near future, in potentially clever ways. Therefore we must, at some point, ban or heavily regulate them. Seems we should start figuring that shit out _now_, as progress has remained very fast and regulation and enforcement take forever on these time scales.
Why would you assume that the same capability could not be used to harden systems?
Sure, I don't have an argument with this. But I'm unconvinced that "therefore all models must be proprietary" follows at all naturally from it.
Crack down on distillation, just for Chinese companies or is it ok for Chinese/US companies to distil? I find it hard to take Anthropic/OpenAI on distillation, because the way see it they started with "distillation" of another kind. They used all the content out there without consent of the creators and its still happening. Model distillation is just a different layer of abstraction, but same thing more or less.
Distillation is just the act of explaining things simply. Richard Feinman rolling in his grave.
Basically we shouldn’t ban open-weights models but we shouldn’t allow them to become as good as the frontier models because china bad. And let’s not have someone else be able to produce a frontier model.
>We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling3 and workarounds used to obtain access to such chips. China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips. This is the most efficient and direct way to block threat #1, and by hampering the training of models that are out of reach of US law, it also indirectly helps with threat #
We should crack down on industrial-scale distillation operations. Distillation is a much more compute-efficient process than training models from scratch. It allows China to build much better models than its number of chips would ordinarily enable, and thus partially evade chip bans. Distillation does not allow the CCP to obtain equivalent or superior AI capabilities to the US, but it can bring the Chinese frontier to within a few months of the US frontier
2.
> Distillation does not allow the CCP to obtain equivalent or superior AI capabilities to the US, but it can bring the Chinese frontier to within a few months of the US frontier
A message to their investors, it would seem. "They caught up just because they distilled! Obviously they couldn't actually be as good as us!" Really funny thing to say right after an OpenAI higher-up stated point-blank that the performance of K3 can't be chalked up to mere distillation of American models.
Why has "open weights" become synonymous with "Chinese" in the first place? To me, that is the problem. I also prefer US models. But I want there to be competitive open weights models too. Those aren't actually incompatible preferences...
Seems like the maximal position he could take compatible with his expressed principles. There’s no way to allow for bioweapon and cyberweapon grade models being open weight if one doesn’t want widespread human damage.
So I cannot disagree with him on the idea. It’s only a matter of degree and whether we’re already there or not. I have $50k in GPUs that incentivizes me to believe we are not.
This is where I'm at, or rather, will be. I like open-weight models and I've done my part in facilitating them, but if you believe in the increasing capability of these systems - and I do, to some measured extent - it seems plausible to me that an incident will happen at some point in the future.
I don't agree with his argument as a whole, especially not on some of the specifics (it is not great that this technology is being developed under the current US government), but I am sympathetic to the idea that some bells can't be unrung, and thus we should proceed with caution.
Anthropic will continue being a victim of their own naive positions on AI safety. They keep dancing around it but their communication is essentially pro-regulation if you read between the lines.
Or if you read the lines. "All sufficiently capable models, open and closed, should go through mandatory safety testing."
If you listen to interviews with Dario and Daniela Amodei it's pretty obvious they think they will be the ones helping define the regulations on AI instead of a group of partisan politicians who don't understand technology, lean on experts from random political think tanks/non-profits, and operating based on fear of foreign competition.
Someone who until yesterday did not seem bothered by his technology being possibly used to bomb elementary girls school in another country seems to suddenly care about the repression of citizens in yet another country.
No, we don't buy your virtue signaling. And we certainly don't need your better-than-thou opinions on this year's "nightmare scenarios".
Yeah Dario is just flat out disgusting in term of how shamelessly hypocritical he is.
Do people actually believe that he gives a shit about the well being of the Chinese people? If the U.S. starts a war with China start bombing Chinese cities Dario would absolutely jump onboard supporting it. He'd probably make Claude to add DeepSeek and Moonshot HQ to the targeting list lmao.
He is super pro-Israel as well, and never once has he brought up the risk of the Israeli government using AI to control and repress people in other countries.
He is also 100% onboard with working with Palantir, who has the explicit goal of using AI for population control and repression and building out a surveillance state.
Meanwhile the world's most repressive government is North Korea, and obviously they don't even need AI to achieve that.
If you talk to people in China they'd laugh their ass off at Dario's notion that somehow they are all getting oppressed by DeepSeek or Kimi.
Every accusation is a confession!
I can't remember the last time (if ever) a company managed to go from golden goose to.. whatever this is.. so quickly. The permanent defensiveness in his presentation is really hard to swallow, it actively puts me off wanting to believe in or rely on their product line with Dario at the helm. I don't even understand the logic leading up to this post. Who was it even hoping to convince. Is it possible Anthropic is due an oil change?
I think they are trying to please a split group of investors and public, of their major investors Google and Nvidia have signed the open source petition letter and Amazon hasn't, so their non position is trying to please every who has taken a clearer stance, although it's quite bad.
Anthropic has been surprisingly bad at comms for the last few months, which I find crazy in such a competitive market.
Current big picture reality is bad for comms unfortunately. As another commenter quoted, "You can put lipstick on a pig, it'll still be a pig". Cuban Missile Crisis wasn't very calm. Do you think a company this well-capitalized and smart is just bumbling around like idiots? Everything makes sense if one just actually entertains the idea that they are earnest and we are in a dangerous arms race
He says that using the set of questions and answers from one model to train another model (deatilation) is cheaper than training the model without those datasets.
But he didn't mention that training any model from a set of texts and books is much cheaper than writing those books in the first place.
In other words, it's ok when Anthropic learns from others, but it is not ok when others learn from Anthropic.
It's also cheaper to just pirate the author's work, which Anthropic has also done.
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."
Hmmmm.
The US is already under an authoritarian regime, the only thing keeping the wheels on the bus is a very tired and barely-effective judiciary.
This is a temporary situation because either this regime is going to be knocked out of power, or it's going to follow through on its core Seven Mountains Mandate[1] theology and go full totalitarian.
Normally totalitarianism fears are overblown, but I think that these zealots would absolutely use the latest frontier models and pervasive surveillance to make The Handmaid's Tale look like a liberal fantasy by comparison.
If distillation leads to a model that is much cheaper to run yet provides similar intelligence then why doesn’t Anthropic distill their own models?
Don't they do it already ? Aren't smaller models distilled versions of bigger models
Demand #1 is standard political nonsense
Demand #2 is hypocritical ladder pulling
Demand #3 is contrary to freedom of speech
so they can clarify however they like, their position is still a stinker
Yep.
> We should crack down on industrial-scale distillation operations.
"We consume all intellectual property for our model but you cannot do the same"
It's worth adding that distillation is not a violation of whatever valid copyright interests a model maker may have (if any). The US Copyright Office has already said AI model-generated output by itself isn't copyrightable. Unless new regulations or laws are enacted, distillation will remain, at most, a customer violating a term of a provider's commercial ToS/EULA.
World's greatest IP thieves propose crackdown on IP theft
It's not even IP. Model outputs are not protected (to the best of my understanding).
“Anthropic has never advocated for a ban on open-weights models.
Open-weights models that don’t have dangerous capabilities are a public good…”
A bit confused on this part, what model doesn’t have dangerous capabilities?
Because Dario is still thinking in the past. He's having a Ben Carsons "the pyramids were to store grain" moment and no one is stopping him.
FTA > "My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks"
If this is the sort of attack he thinks is to be worried about then I dont know what to tell him. We already opened pandoras box on this. Look at what the Ukraine has done with open source drones (hunting people autonomously)
It takes minimal funding to build enough drones to destroy enough power infrastructure to shut down a large chunk of our grid. It takes even fewer talented resources to put that together with the help of already available AI.
The question I would ask Dario is this: what would some one like Ted Kazniski come up with given the resources of AI. It sure as shit would not be hacking or bioweapons or bombs in the mail.
IF they really gave a shit about safety, the would be funding (in conjunction with other AI companies) actual anonymous red teams (Ala wall facers) with some degree of independent over sight to put in the work that they arent. We're talking about a company that could not even keep its own harness code secure.
It seems possible to deliberately not train on some offensive capabilities and still have a very useful model. For example, Opus 5 deliberately did not train on exploiting vulnerabilities, and so performed less well on exploits than Mythos, yet was equally proficient at finding such vulnerabilities, according to the Opus 5 system card in their "OSS-Fuzz" eval [1].
[1]: https://www.securityweek.com/anthropics-opus-5-nears-mythos-...
That's a good example, but I'm unsettled by Anthropic's growing refusals in the areas of chemistry and biology. If they think that scientific assistant models should be as unhelpful as Fable, because applied scientific knowledge is inherently dangerous, I don't want Anthropic or like-minded thinkers setting the standards for model safety.
It's hard to imagine a frontier model being proficient at finding vulnerabilities, but not so proficient at exploiting them.
Surely finding is the hard part, and any LLM should be able to easily exploit a vulnerability it already knows about?
No, this is not the case for human security researchers and I don't see why it should be true for LLMs.
Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)
The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic's bottom-line.
Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips so obviously they want to restrict what models are out there and more importantly who can produce new ones. Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.
Do people actually believe that he gives a shit about the well being of the Chinese people? If the U.S. starts a war with China start bombing Chinese cities Dario would absolutely jump onboard supporting it. He'd probably make Claude to add DeepSeek and Moonshot HQ to the targeting list lmao.
Meanwhile he is also 100% onboard with working with Palantir, who has the explicit goal of using AI for population control and repression and building out a surveillance state.
If you talk to people in China they'd laugh their ass off at Dario's notion that somehow they are all getting oppressed by DeepSeek or Kimi.
It's baffling that they thought the mental gymnastics in this blog post would make them look better. I'd rather they simply fall silent on the issue; I would respect them more (or at all) for it. Open models obviously threaten fierce competition, if not outright destruction of their bottom line. But no, they needed to try and argue that they have the moral high ground for attempting to singularly consolidate power over all human labor.
Exactly.
And the article specifically talks on restricting hardware for the China and restricting China's open source models for the west. All while leading us on with "we're all for competition (but...)"
I think China did great by releasing AI innovation as open source, thereby limiting or sooner-bursting the AI bubble; which is clearly in their interest.
The models are not open source. They are deeply proprietary since we have no access to the source materials and cannot reproduce the model independently. They are opaque binary blobs that the Chinese labs are just allowing other providers to run directly instead of only access through an API.
Yeah some real main character energy from Dario as usual.
I'll never get why he thinks China would just sit there and let the US dominate them in AI when all it would take is a few of their boats blockading Taiwan to put a stop to it all.
wouldn't stop AI companies from continuing to develop using their current infrastructure.
the West could retaliate by halting shipments of photoresist and other materials to China.
meanwhile, Intel second-sources Nvidia and starts pumping out GPUs.
the economic fallout would be devastating as trade wars and export bans on both sides make Trump's "Liberation Day" tariffs look like NAFTA.
The rest of the world would likely side with China. The export ban on Fable wasn’t even extended to five eyes countries.
US is going to find itself isolated and irrelevant. And not a moment too soon.
> We should not sell powerful chips or chipmaking equipment to China
This is so short-sighted given that the US needs China equipment for.. everything. They are part of the supply chain needed for building the machines that build these very chips.
Jensen was right. We should have sold chips to China so they'd be depenent on us.
Now they have their own chips and most of Nvidia product line is internally banned.
It's unlikely that China would've become completely dependent on us in either case. They're good copycats, with incredible talent in engineering and manufacturing. They're aware that we depend on them for a lot - I think they'd be similarly aware of the risk of becoming dependent on us.
> Now they have their own chips
I never understood that argument, are you saying Chinese companies are not going to build their own chips if they get access to Nvidia chips?
They were, just not as quickly. The export controls directly accelerated their development.
The general rule is: USA bans China from having thing, they make their own version of whatever that thing is. USA bans China from the ISS, they make their own space station. USA bans China from having ASML, they make a Manhattan project to clone it, the "20 years behind the west" line is history. They ban GPU exports, they just start making their own GPUs.
I gotta respect the chinese. I wish my own country had the balls to do this.
> We should crack down on industrial-scale distillation operations.
Also Anthropic:
AI firm Anthropic agrees to pay authors $1.5bn to settle piracy lawsuit https://www.bbc.com/news/articles/c5y4jpg922qo
but Dario is the good guy, distillation when done by Dario should be called innovation /s
I wish this were higher up!
They pirated my work and now they want government protection from other people doing the same.
I generally disagree with the claim that distilling a model is equivalent to training on freely available internet content – mostly due to investment required to turn it into a model – but piracy is another story. Pretty inexcusable.
If they paid for tokens say via subscriptions. Wouldn't that just be same as acquiring books and using them as "fair use" to train? I really see no difference.
"And when their eloquence escapes me Their logic ties me up and rapes me"
Police, 1980
They did not need to destroy the models they got with pirated content. Would have been more fine if they would have needed to buy the books afterwards and train based on then, again.
> All sufficiently capable models, open and closed, should go through mandatory safety testing
What happens if a model fails the test? Surely one can use Kimi K3 for evil, somehow or other. What now?
"Mandatory safety testing" implies consequences for failing, yet Dario has nothing to say about what the consequences should be. He says he doesn't advocate a ban but it's hard to imagine what his alternative would be if he won't say it.
If a model fails the test, it should be banned. He is not advocating a ban of open-weight models. He is advocating a ban of models that fail mandatory safety testing. Seems reasonable and straightforward.
Hm, I interpreted it to mean they are more worried about loss of control/misalignment risks rather than misuse?
Nah, the statement is the mechanism for a ban. The proctor will be someone anthropic trusts and "surprise" as it turns out all the open weight models fail or aren't eligible.
Imagine the Dieselgate levels of adaptation they'll do while being tested too.
Guys. Guys, you got it all wrong. We don't want to ban open-weight models!
We just want to ban the competition guys! Very different.
--
The ridiculous anthropic/openai strategy of selling shovels at a loss in a gold rush isn't going to play out, and the hilarious thing is that these AI companies are going to create tons of value and _capture none of it_.
Their only path to profitability is if they get to capture it and they're going to do everything to do so. Put it this way: *all the blog posts that Anthropic and OpenAI are putting out are DESIGNED to scare you so that you let them capture the market*.
...and "distillation attacks" (hilarious framing of "saving the output of our models")... Whatever.
The genie is out of the bottle. Anthropic and OpenAI have been trading mirrors for gold, and people started to realize what a mirror actually is.
"No guys, Anthropic actually loves open weight models!" Yeah, and Microsoft famously loves Linux. Sure.
http://www.omgubuntu.co.uk/wp-content/uploads/2018/04/micros...
> Open-weights models that don’t have dangerous capabilities are a public good: they don’t cost anything besides the compute needed to run them, and they provide value to businesses, developers, and researchers.
No "love" of open weights asserted, just acknowledgement of value.
(And their call for safety was for both open and closed models.)
A bit off-topic from the core of the post, but:
> At Anthropic we’re committed to cracking down on industrial-scale distillation through our own practices, including identifying and banning accounts that use our models in this way. This is challenging—for instance, the relevant accounts can often only be identified after substantial distillation has occurred, and distillation often involves creating large numbers of fake accounts that form a moving target. The practices of any individual company cannot entirely solve the problem, which is why we have called for policy on this issue.
One thing I've never really understood is what sort of policy could possibly deter or hamper Chinese labs' distillation efforts. The only thing I can imagine is some sort of strict KYC regulation applied to all models above a certain threshold, which seems both painful for the broader US AI ecosystem and bound to fail anyways.
KYC hinders Anthropic's growth and Anthropic wants growth for IPO
so Anthropic's ask is for US gov to ban open weight models so that its growth (and IPO) is not affected
A sober look at actual rogue nations and their use of AI shouldn’t have us fretting over that particular hemisphere
Yes, it should. It absolutely should.
The United States making questionable decisions and behaving recklessly and dangerously as a country does not suddenly make China any better.
China is as worse as the United States, if not more worse, by many measures.
Hmm.. how many illegal wars has Xi started, just in his most recent term of office?
China is nowhere near as bad as the US at this point. The rest of the world is changing lanes to not be implicated in your car crash of a country.
Dario has like three 'paranoias' / strong-motivating-concerns
1) LLMs turning into Skynet
2) China as geopolitical competitor
3) Claude being 'distilled' by competitors (this has led Anthropic to cut service to various American companies too from time to time -- OpenAI, xAI etc have been cut off from using Claude for coding in the past)
So this post just reiterates that these 3 concerns fuse together in his mind when thinking about open weight models
Is he actually concerned about China being a geopolitical competitor or is that just the most logical position for him to take as CEO of a US corporation with national security implications?
He's just pandering to the lunatics in office. They're even quoting Vance, like he was a respectable and wise politician and not an insane puppet built by oligarchs and for oligarchs.
Is China opposition unique to the current administration? Didn’t they soften the CHIPS act put in place by the previous one?
Oh, I trust he'd be pandering to the Harris administration too, if they were in office. It's true that antagonizing China has been a constant in US politics for a while now. It's just especially funny to see someone pretend to be concerned about the military threat of the CCP while glazing such a shamelessly warmongering government.
As expected they will try hard to use government to kill competitors.
> apply such testing to the most capable models regardless of their country of origin or whether they are open or closed ...
> ... (while exempting less capable models, such as those from startups and academia, entirely)
The devil is in the details, but this isn't anti-competitive as stated.
Then you did not read it clearly.
Edit: Typo
Which of those aren't related to regulation? Preventing Chinese models from entering our market? Using boogeyman 'distillation' as a means to target competitors? Point 3 is literally about ADDING regulation.
Please elucidate things clearly for everyone else.
Perhaps they saw it crystal clear.
"open weight" models are not open source. They are still deeply proprietary. It is not possible to know what they do or what they are capable of without interrogating them since we have no access to their source materials.
The only difference is the Chinese labs have allowed 3rd party inference providers run the proprietary models for them since they cannot do it themselves due to domestic GPU compute constraints.
It would be awesome to have actual open source (and open weights) models! I hope someone will make a real go of this at some point.
https://allenai.org/ is what you're looking for
Yes, I've seen it! It's exciting, but I don't have enough information to say whether they are making "a real go of it". That is, it's unclear to me whether they have the tens to hundreds of billions of dollars necessary to create a frontier model.
You can put lipstick on a pig, it'll still be a pig
"Anthropic has never advocated for a ban on open-weights models."
---
"We should crack down on industrial-scale distillation operations"
"All sufficiently capable models, open and closed, should go through mandatory safety testing"
These are in tension with advocating for open weight models. Not direct but enough that it calls into question the first statement. What is the testing criterion? How do you pass it? Is it a government body that approves a pass fail or a global body? If it is government, and boy does it seem to be, how do you disambiguate MASSIVE corporate lobbying to set up the safety testing in such a way that the boys in blue are let through and all others are barred out of safety concerns?
My concerns aside, much of the soft-points being made are non-historic
"But I don’t agree with the letter’s assertions that open-weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. It seems at least as likely to me that the opposite will be true."
It doesn't mater what his opinion is. The fact is that an advanced, closed, American AI model hacked another company. The only defense was open-source AI from China. We aren't in a vacuum, we have real world examples now and these statements are counter-factual.
I have no idea what to do about the government interference. There's probably not a lot anyone can do.
However, your last point is quite a strong one. Corpos aren't just going to stand there with their collective pants down, and there's not a lot anyone can do to stop them from protecting themselves. There are ways they can get what they want without getting caught.
Remember when the US tried to ban strong cryptography in the 1990s, and how well that went? They may have more leverage with AI because it's a bit harder to hide large scale computing usage, but I don't think it's impossible at all.
It's political doublespeak. They want to have their cake and eat it too.
I don't really understand how they can argue the security angle with a straight face. It's not like GLM 5.2 is a slouch. I've seen it do things like exploit an IDOR issue when I was experimenting with a quick-and-dirty web automation task. I simply fixed it, as one does. Open models make the world better to a far greater degree than they set it aflame.
Their position is analogous to trying to, say, ensure digital privacy for everyone not by making encryption freely available (because that would let the bad guys use it!), but by making it so you can't use general purpose communications devices that can listen to transmissions not intended for you. Do they hear how moronic that sounds?
Each passing frontier-level open model release makes Anthropic's patronizing rhetoric a little more insufferable, because it becomes clearer how unmoored from reality they've become in pursuit of profit.
What do they even really mean by "safety"? I mean, I can have an Anthropic model do something incredibly unsafe if, for example, I put it in charge of a hydroelectric dam and don't explain properly how the controls work. On some level, everything is simultaneously "safe" and "unsafe". I've never found Amodei's reasoning here to be particularly well thought-through. I think he, like a lot of folks in the area, are starting to realize that they may never be able to build a moat around their businesses.
> an advanced, closed, American AI model hacked another company. The only defense was open-source AI from China.
HuggingFace did not seek access to Claude Mythos or OpenAI's equivalent program. They probably could have had access to these models for defensive purposes if they'd done it properly.
> these statements are counter-factual.
The OpenAI incident is a single example. You're massively overgeneralizing. You can't refute an entire class of possible outcomes based on a single event where it went the other way.
I tend to agree that model capabilities will favor defense over attack, but I think there will be a lot of disruption before that equilibrium is reached. If cybercriminals or state-sponsored actors are able to scale up attacks quickly, many orgs with less sophisticated defenses will be caught by surprise.
"HuggingFace did not seek access to Claude Mythos or OpenAI's equivalent program. They probably could have had access to these models for defensive purposes if they'd done it properly."
HF released a statement and made it clear a closed source model specialized in cyber security refused them. They stated they had to use open source. What model is specialized in cyber security, closed, and frequently denies users access other than Mythos/Fable and 5.5Cyber? If not these two, what was HF referring to? It sounds like you have a source, I would like to read it.
"You can't refute an entire class of possible outcomes based on a single event where it went the other way."
But Dario can dream up and entire class of outcomes based on the zero events that have never gone his way? Convenient.
The OpenAI incident is singular and HF was clear, it went exactly how I wrote it: a closed source American AI decided to perform corporate espionage and the only tool available was open source AI from China
"I tend to agree that model capabilities will favor defense over attack, but I think there will be a lot of disruption before that equilibrium is reached. If cybercriminals or state-sponsored actors are able to scale up attacks quickly, many orgs with less sophisticated defenses will be caught by surprise."
We literally just saw an advanced model from openAI commit a cyber crime. I can't take hypotheticals that ignore reality seriously and it shouldn't be lauded as some higher form of thought
I assume you mean https://huggingface.co/blog/security-incident-july-2026. It says that they used frontier models, not frontier cybersecurity models. My reading is that they asked Fable and it refused; if they'd had access to Mythos, it would have helped. You're mixing the two but they're NOT the same model.
Source is here: https://thezvi.substack.com/p/more-on-an-internal-openai-mod... ctrl+f "Skill issue." No source is cited, but I'm fairly confident it's correct. If Mythos/5.5Cyber specifically had refused to help, then HF would have made a much bigger deal out of it. The whole point of these models is that they have relaxed guardrails and specialty cybersecurity training relative to the publicly-available ones.
> zero events
What about all of the vulnerabilities already patched under Project Glasswing?
In the quote you provided Amodei is expressing uncertainty, saying we don't know which way things will go. You're the one making strong assertions; the burden of proof is on you.
To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? The OpenAI / Hugging Face incident shows what a GPT 5.6 level model can do off the leash; within ~6 months, open weight models will match this and every bad actor under the sun will be able to pull off attacks at this scale. Do you seriously want this level of capabilities to be generally available with no guardrails?
The open weight issue has a lot of difficult nuance. Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
> what should be done about open weight bioweapon
Does not exist. What has in fact happened is some cults had bioweapons programs but any failure points were at deployment. (Aum Shinrikyo https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack and https://en.wikipedia.org/wiki/1984_Rajneeshee_bioterror_atta... )
> and cyber-offense capabilities?
You mean defense. That's how things get hardened. Anyone that was working during the XP era before Service Pack 2 knows what that was like, but it's very manageable.
The bigger real problem here is hardening like that would remove the opportunity for intelligence agencies to spy on everyone.
You admit that some attackers have the inclination to use bioweapons. Why would they not use the best tools at their disposal going forward?
From the WSJ the other day:
> After OpenAI enhanced the brain power of its chatbot last summer, hundreds of users worldwide began asking it how to make and deploy biological weapons and poisons.
https://www.wsj.com/tech/ai/openai-chatbot-biological-weapon...
On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.
I've got zero knowledge of bio, so can't answer that. But with cyber the answer is very simple - the attackers already have more cyber-offense capabilities and there's no putting it back.
Open/closed doesn't matter that much. You can get closed models to do a lot of cyber harm, even with all the guardrails, which currently are heavily skewed towards more false positives.
The only effective control is to level the playing field. If both offense and defense have access to the same capabilities, then we're relatively back where we started.
If you want to ensure chaos, then you do what Dario is proposing to do - create gates that attackers can bypass and defenders can not.
In cybersecurity, a level playing field favors the attacker. Trusted access programs give defenders access to tools they need. It's not perfect (because there is an extremely long tail of defenders who are not technically savvy enough to get on these programs and use the tools), but it's better than total access.
The bio angle is very important here too; in that context the imbalance favors the attackers much more.
> In cybersecurity, a level playing field favors the attacker
Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs.
And I don't know what Trusted Access programs give to defenders, because as a defender who has credentials, connections, but no deep pockets and no high ranking passport, it only gave me silence. I fail to see how this is better than total access.
I don't think the world where defense is given to those that "deserve" it is the world that we all want to live in. Which brings me back to the starting point - attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.
The Hugging Face incident is a great example of why open source models with defensive cyber capabilities are needed. Hugging Face did not have access to cyber-capable frontier models and kept hitting safeguards. Only by using the open source GLM-5.2 were they able to survive an attack. A world where open source models are banned is one where cybersecurity is impossible if you're not on OpenAI or Anthropic's allowlist.
Hugging Face survived the attack because the OpenAI model only cared about accessing the ExploitGym dataset; by all appearances, HF was completely owned. GLM-5.2 was only used to assess the damage after the fact. Cybersecurity has a attacker-defender asymmetry that heavily favors attackers. If GPT-5.6 were open sourced today, do you think every hospital in the world would be able to use it to shore up their defenses before attackers got to them?
I know the company I consult for (not cybersecurity) is not in these programs and if attacked would need to use open weight models.
Did the company apply for access? This is either a problem with your company or the trusted access program. In no way does that suggest the solution is total unfettered access for everyone.
> Anthropic has never advocated for a ban on open-weights models.
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
Yeah, this is anthropic advocating for a ban on open weight models.
Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate.
This is exactly how the US has banned goods in the past, by requiring a stamp and then refusing to issue it.
Same way they have banned DJI products like camera microphones, technically it's not banned, it just needs to be approved because it has a wireless transmitter, and for some strange reason the US is the only country that hasn't approved them.
Exactly. If you care about AI, simply don't use Anthropic - use open source.
This isn't actually about safety. This is just another example of pulling the ladder up so nobody else can follow
> > All sufficiently capable models, open and closed, should go through mandatory safety testing.
> Yeah, this is anthropic advocating for a ban on open weight models.
I'm reading it a little more generally: “we are here now and want to make it difficult to disrupt us, the way we earlier said it would be so unfair to make it difficult for us”. Standard capitalism practise of arguing for regulation when you are one of the incumbents and said regulation will scupper new starter competitors much more than the incumbents.
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
I mean you're assuming this is even possible. I don't really care what the US admin does. If someone releases a powerful open source model I'll run it. Good luck trying to stop everyone doing that.
Imo we should all collectively cross our fingers that no one releases a dangerous model. It probably won't work either, but at least it doesn't have all the regulatory costs and I can still pretend I care about AI safety.
There should be safety testing, but no guardrails that limit models for cyber or bio research.
Guardrails are not a safety measure, they are a pay-to-play scheme that allows the people with deep pockets to have access to offensive and defensive capabilities first.
When Fable was yanked, it was said to be (in part) due to the "jailbreak" of instructing Fable to "fix this code" — https://news.ycombinator.com/item?id=48552687
Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?
There can't be more than a few million to tens of millions software businesses / services / regularly used F/OSS projects on Earth.
Why not just give everyone a $100 Fable / Mythos credit to "fix [their] code?"
It would arguably benefit Anthropic. For $100M to $1B, Anthropic could execute the greatest ad campaign in human history. And they'd make the entire world more secure.
Most people aren't malicious. If you, as an engineer, consultant, founder, business owner, or maintainer, were given access to Mythos' capabilities wouldn't you ask it to fix your code?
I might be wrong. But I think that a greater amount of harm will be done in the long-term by trying to lack these capabilities and systems away behind permission gates and sealed doors. It creates an asymmetric world with haves and have nots. And in that world who gets to have access now decides who gets to be secure.
If everyone has mythos, no one has "Mythos."
Just let people fix their code.
I doubt most bosses will give engineers the time. They care about security only to the extent that they have already been harmed by a lack of it. I would like to play with mythos, but on my own time my kids have plenty of activities to fill my time. My personal backlog of projects is only getting longer and none of it is something mythos could help. If I had more time is have restored my old truck instead of making payments on something new (in turn limiting what else I can afford to buy)
I think eventually there will be Mythos grade AI which will be released which can solve a lot of bugs, even right now opus/fable can fix more things which companies can even keep track of.
The problem is how to make sure such AI is released safely. The same AI that can solve bugs can also find bugs in authentication or loopholes in critical systems.
I think there is some logic in delaying the rollout, giving it to the heads of the largest software products first to fix their code before dumping it on the general public. But yes eventually everyone will have this tech and it won't matter because the low hanging fruit will have all been picked clean.
I don't think a one-time $100 credit is enough. First of all, that isn't very much. But also, the volume of new code is going way up. Unless they keep giving out monthly free credits, it's just a stopgap.
> If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?
Because it doesn’t really confer the advantage they claim, especially compared to e.g. paying an equivalent amount of money to do traditional security scanning.
It’s much better to play of FOMO and hype than to let everyone use it and be underwhelmed.
You forgot “what is the definition of ‘sufficiently capable’”. Presumably it’s anything that competes with Anthropic. If they’re around in a year, presumably they won’t care about Fable level and will only think that whatever competes with Claude 7 or whatever needs to be restricted.
GPT-2 was 'sufficiently capable' according to Amodei: https://explainx.ai/blog/dario-amodei-gpt2-openai-open-sourc...
Regulation is not a blanket ban. Regulators (presumably government agencies) can review models (of any kind) and approve or ask for changes.
There are many other regulated industries, like drugs (the FDA), cars (NHTSA and EPA), airplanes and rocket launches (the FAA), radios (the FCC) and so on. That's not unusual. Regulation is normal for stuff that might be dangerous.
cryptography, too. remember when that was regulated? strong cryptography had to be carefully controlled as a munition, for national security!
Regulations on forbidden numbers exist, but they are usually not a sensible policy and are not comparable to the regulation of rocket launchers.
Yeah, seems pretty likely. Anthropic will make the case that their models should be evaluated with the safety layer in front, because that is the only way the model is available whereas open weight models need to pass the same test just on the weights. The economic implications will be rather large, but in terms of security it seems inconsequential. The most compelling argument would be that by limiting the use of open-weight models in the US that it will reduce cases of accidents like the recent attack on Hugging Face. More crucially though, the US government can do little to enforce their testing requirements. The nature of open-weight models makes it virtually impossible to clear the same bar for security as models served via an API. Open-weight model makers couldn't comply if they wanted to. The US government can restrict access with IP blocks and limit inference capacity with export controls, but these measures are not effective in deterring malicious actors.
Makes sense why OpenAIs little "hacking" stunt was published last week
> The most compelling argument would be that by limiting the use of open-weight models in the US that it will reduce cases of accidents like the recent attack on Hugging Face.
an attack done by a closed-weight model (GPT-6) and defended against by an open-weight model (GLM-5.2) precisely because OAI positioned themselves as gatekeepers for cyber capabilities.
if anything, open-weight models shift the battle towards defenders because they can actually run them.
A government agency tests all medications, why not models?
There's a different level of personal risk with these two things. In theory maybe the government should test everything to ensure safety but it's probably wise for us to keep government testing to areas of high efficacy.
I wouldn't object to a government advisory body that tests models for safety so that users can make informed decisions. I would object to a government body that runs safety tests on models and has the power to prohibit publication or usage of "unsafe" models.
Do they? Or do they accept trail reports pay for by the pharma (super expensive, hence not affordable for open source / not-patentable medicine development)
Don’t think government controlling AI is a good idea.
Not sure if they have an understanding of AI in the first place. Secondly, even though AI companies claim that they have achieved AI that needs to be heavily monitored (maybe for PR purposes), I’m not sure if that is true. Sam Altman said the same things about GPT-4 that Anthropic is now claiming about Mythos.
Government control will be a good idea once we start approaching AI that is actually destructive.
Also even if we decide to put controls in place what is the guarantee that china will do the same, specially for a model which is not actually destructive.
Yes: at great expense, one carried in part by drug companies. Who pays to test the open weight models?
...the AI companies? Probably Anthropic itself? I see no possibility of regulatory capture here, so it must be a good idea.
The entire safety evals industry is essentially funded and controlled by OpenAI/Anthropic. Notice that on recent models, they exclusively use internal testing or black box external vendors (e.g., Gray Swan) whose entire business is to serve OpenAI/Anthropic. And all these companies just share the same pool of researchers back and forth.
The USG has a safety organization (CAISI), but it has been neutered by the current administration (with the recent stop-work order etc.). Perhaps UK AISI would be closest to what you are looking for? See their recent work on Kimi K3 cyber (which was declared safe) [1].
It's tricky because a lot of the safety researchers have ties to the labs since those were the only companies training LLMs >5 years ago.
[1]: https://www.nist.gov/news-events/news/2026/07/uk-aisi-caisi-...
Who gets to decide what is safety?
I expect some of those tests (prolly not public) will basically be "wokeness" tests or "PC correctness" tests or "western media filter" tests.
China has different objectives. Sure.
I'm not sure one is safer than the other; I would know which one to go to if I want to research on topic that are viewed very different on both sides of this "new iron curtain".
So, it's a cottage industry.
Anyone who calls it “safety” probably has a certain world view and is more aligned with the big 2 (and stuck in 2023).
There is a growing industry of commercially focused risk evals that has a broader customer base.
What’s the equivalent term for “safety” that’s used by others?
To me "safety" means "I'm safe from this while I use it". It means the AI is my loyal friend who will never betray me in any way, no matter what prompt I send it.
Not even Anthropic can claim that.
that's pretty damn smart if this was a long-term plan to block competitors
Consider how much money is at stake: some industries have leveraged their power to lobby for bombing entire countries or topple regimes across the world for much less.
Creating an industry around an elusive concept of safety to force regulatory capture seems pretty straightforward to me.
I mean... I'm not even extraordinarily cynical about this stuff, but to me this seems like a totally normal level of corporate gamesmanship?
Companies look for and seek to maintain competitive moats. This is not particularly clever, it's a core part of corporate strategy.
of course, but the safety angle was pushed from day one. I more mean the forethought of how it would play out
It's standard regulatory capture.
You don't say "let's ban my competitor".
You say "let's create laws that make it uneconomical for my competitor to access the market".
is it opportunistic though, or planned from day one? The safety narrative has been there since the beginning
Indeed. It's transparent and ham-fisted. I think it may cost him in the future.
People clown on Alex Karp for his unedited maniacal "crashouts", but this is a real public crashout that made it past a team of publicists.
Their position seems fair to me - sure it does help them as well, but I don't necessarily disagree with the risk they are laying out.
It is consistent with their stated beliefs, unlike OpenAI who flip flop every 6 months on whether they support open source or not.
I think their biggest PR problem is that many people still think of loss-of-control/misalignment etc. as sci-fi. And the distillation arguments come off poorly because people feel as though all the labs have trained on their creative output without their consent, so they deserve to own the result in some way.
Agreed, makes logical sense. You can disagree with specific premises, but the best arguments aren't "OPEN SOURCE GOOD" or "SELFISH ANTHROPIC"!
China bad!
THiS! I'm pretty amazed how many people shoot them down without acknowledging the very real and somewhat probable risks they and other researchers have laid out. I don't agree with every point they make but folks really do just seem to think we should just keep building any technology and whine when we start to consider there are very real risks associated with powerful technology. checks notes see nuclear bombing of japan
Re: shooting them down, I think there are a lot of people out there that consider the US a bigger threat than China. Maybe they're right, I don't know. But I do know that as an American, I'm not looking forward to finding out.
And then there are probably people who are more politically neutral who think Anthropic is using China as an excuse to crush competition. Which could also be true.
But fundamentally, if this technology is so dangerous, why does anyone get to control it?
I thought Dwarkesh's position was pretty thoughtful: https://www.dwarkesh.com/p/dow-anthropic
> Nobody is qualified to steward the development of superintelligence. It is a terrifying, unprecedented thing that our species is doing right now, and the fact that private companies aren’t the ideal institutions to take up this task does not mean the Pentagon or the White House is.
> The only way we can preserve our free society is if we make laws and norms through our political system that it is unacceptable for the government to use AI to enforce mass surveillance and censorship and control. Just as after WW2, the world set the norm that it is unacceptable to use nuclear weapons to wage war.
Thanks for sharing, i agree no one should have absolute control of anything imo, and the mo greater the magnitude of implications the more important it should be stewarded democratically with clear and transparent principles with values adhering to things like human dignity, freedom, human, planetary & animal wellbeing etc etc. ill have to read the article
So the argument is basically: This technology is too dangerous so only _we_ should have access to it. We’re the good guys and only we can ensure a safe use of this technology.
Quis custodiet ipsos custodes?
> To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category. We should instead focus on keeping powerful chips out of authoritarian hands, stopping industrial-scale distillation, and requiring safety testing of all sufficiently capable models, open and closed.
Maybe I should’ve expanded a bit on my comment. I didn’t mean “we” as in Anthropic directly—even though they certainly are advocating for restrictions on how to train AI systems by proposing mandatory safety training—I mean the argument that American AI labs are somehow more responsible than their Chinese counterparts.
It’s especially jarring when just last week OpenAI—an American company—accidentally hacked Hugginface when performing safety testing on an upcoming model [1]. If they have the ability to turn off all guardrails when testing out their models—or when selling them to the military—then the safety training is only there for show. If they can pick and choose who should have access to their most powerful model, surely they are trying to act as the world police?
[1] https://openai.com/index/hugging-face-model-evaluation-secur...
Demanding "required safety testing" is demanding a ban. Otherwise the testing would be inconsequential, right?
I'm sure he didn't mean just a "lobotomized to be worse than Anthropic products" badge for the test-passing models.
If a ban is the implied consequence of failing his "safety" tests, that means that Anthropic was and currently is advocating for a ban on some open-weight models.
Are you replying to the article or me? I have not stated any position. The quote is Anthropic's and was relevant to the comment it followed.
My views:
I find testing of SOTA models problematic.
I find not testing of SOTA models problematic.
Neither view on testing is without merit.
The right way forward is unlikely to be as simple as either of those, but some carved out balance between them. And it is likely to change over time.
Yes, to clarify: I was not arguing against you but against the statement you quoted, made by Dario.
Your quote was very relevant, as it highlights the foundational lack of intellectual honesty behind the whole Anthropic statement.
I felt like the grudging "if we must" nature of his acceptance of open weights came through honestly. :)
It is clear he isn't a champion for them.
Of course, the CCP with access to extremely powerful AI models would be a tremendous risk.
The NSA and the CIA with the same models, on the other hand, would use them exclusively for the good of the common man.
Yeah, like bombing a children school, the CCP did that not long ago.
[dead]
I can understand why the Western media calls something that has an official name of CPC (the Communist Party of China) as CCP (Chinese Communist Party? Not sure here). What puzzles me is - why ordinary people always repeat this wrong abbreviation. Is it like “I never check the sources, I trust everything that Western media publishes”?
More of a "Everyone knows what CCP references and I'd have to include a distracting explanation if I want to use the more technically correct acronym"
Up until a few years ago the PRC used CCP themselves all the time. It's a handy little shibboleth. If someone calls it CPC they're probably a bot.
CCP is a direct transliteration of the characters, so that's what it started as. Some time later China decided to change it but that's a lot of cultural inertia to move in a different direction.
https://en.wikipedia.org/wiki/Chinese_Communist_Party
I've never heard it called anything other than the CCP.
For something I say maybe a few times a year it’d be a great look to sneeringly point out and talk down to people every time I mention China in the context of international politics.
Unless the Communist Party of the US (I’m not looking up its official name, because it doesn’t matter) wins the next presidential election it’s unlikely that people will call it anything but the CCP. Everyone know what everyone else means.
It was an intentional propaganda strategy to separate references to the government of an official enemy country from references to that country (see also "the Houthis" and "the Taliban"), and it also looks like "СССР."
The reason why ordinary people parrot it is because that's what it was designed for. The proper term for "CCP" is "China." Referring to the Chinese government as the "CCP" (or the CPC) is like referring to the US government as the "Demoplicans" (or the Democrats and Republicans.)
Instead, we just say "the US government" or "the US administration."
[dead]
> The NSA and the CIA with the same models, on the other hand, would use them exclusively for the good of the common man.
the real argument is that CCP will leverage AI against US interests, which is obvious. it's weird how so many people pretend that they are citizens of the world and above it all.
> has anyone ever made this absurd argument?
Yes, anthropic just put forward this argument. It's the whole point of the article.
I agree, it's absurd.
-> the real argument is that CIA will leverage AI against China interests, which is obvious. it's weird how so many people pretend that they are citizens of the world and above it all.
Works for both ways, which is fair?
Dario Amodei Literally makes this exact argument in the OP.
>it's weird how so many people pretend that they are citizens of the world
97% of the world aren't US citizens and if you've taken a look at pew research surveys (or travelled to the so-called global south) you're going to be in for a bit of a shock (https://www.pewresearch.org/global/2026/07/15/people-in-many...)
The competition and sheer output of China has driven prosperity, it's the largest trading partner of 150 countries, the US of 50. People don't need to be citizens of the world, they just need to rationally look at their own interests. China is driving down prices of technologies making them available in countries that never could afford first world prices, the US is driving the them into an energy crisis and bankruptcy.
[dead]
The fact that much of the world is as unconcerned with the interests of the US as with those of China, if not less so, should give pause to Americans.
As a citizen of neither country, Chinese open models are in my interest more than US closed models. My only concerns is that if/when Chinese AI becomes more powerful, they too will have little incentive to make their best models open weights.
Actually this would be pretty great, because it would incentivize US labs to pursue the open weights strategy for the same reasons China is currently.
I genuinely think that this is what the trends and incentives point toward: Competition to develop open weights models and to develop efficient inference hardware to run them.
This would be good! But government policy could very easily screw it up.
US citizens have a much greater threat from their own government than a government an ocean away.
See, the Snowden Leaks.
> US citizens have a much greater threat from their own government than a government an ocean away.
> See, the Snowden Leaks
Are you saying the Snowden Leaks are more dangerous than a world where the CCP is a global hegemon?
If your focus as an American is being safe as an American, what the US does in other countries is far less of a concern to you than what other countries might do to the US.
In the case of the CCP, they have and will attempt to destabilize the United States of America and in turn make life measurably worse for Americans because they wish to be the world’s hegemon.
Fundamentally, Americans are safer when the United States is the number one power than when China is the number one power.
> If your focus as an American is being safe as an American, what the US does in other countries is far less of a concern to you than what other countries might do to the US.
There's a causal relationship between "what other countries might do to the US" and "what the US does in other countries" which you seem quite keen to ignore.
China, Russia, and plenty of other countries have all been actively and successfully destabilizing the US for well over a decade now. To the point that your argument about who is less a threat to Americans depends heavily on the skin color, religion, and ethnicity of those Americans.
>the real argument is that CCP will leverage AI against US interests
many people believe that the US will leverage AI against US citizen's interests.
[dead]
Indeed. It already has. In particular I remember several extremely offensive slop pictures and videos being posted by someone in the White House. And I'm certain that's just the tip of the shitberg.
Yeah, but unlike China we have laws and ways to fight against it. China can and will do whatever the fuck they want they don't have to listen to the people of China.
Largely hypothetical, not very effective ways to fight against it. Meanwhile we have the entire Pacific Ocean and US Military between us and China.
The american state routinely shoots and kills its own citizens in broad daylight with 0 repercussions. Thinking it answers to its citizens is severely deluded.
but but but we have guns...
The US has been doing whatever the fuck it wants without listening to its people lately, though.
Well good thing we have elections.
Every four years!
The problem is that at the current moment with the current administration, it does not seem like we do "have laws and ways to fight against it".
I'm more optimistic about the likelihood of the US system of government to heal itself than that statement might seem to imply. But it's just also the case that at the current moment in the US, the rule of law is very much under threat. And as your comment suggests, that same rule of law is a very important thing to the way of life in the US. It's a very bad situation that we've allowed ourselves to slouch into.
Well that what the people voted for. In a few months you can replace these people that allow this. Good luck replacing anyone in the goverment of China.
This is not an argument that the Chinese system of government is better than the US system. Like I said, I'm more optimistic about the US pulling out of the tailspin we've been in for the past decade. But it's also imminently reasonable to believe that this episode has raised real questions about the durability of the rule of law in this country.