saw an interesting hypothesis about this ( quoted from Jake Mendel at https://www.lesswrong.com/posts/eKe7fky8kSt9DCJxZ/openai-ast... ):
"It's probably that in capabilities RL the models are rewarded however they complete the task. Whereas in RL related to alignment or safety, they're more likely to be being tested for their ethical judgment or rule adherence (ie theyre supposed to be a good nice well-behaved AI rather than a ruthless task completion AI). So they're more likely to refuse or have some ethical issue or stop and ask for help a suspicious amount of the time if the task looks more like alignment RL environments than capability RL environments."
Doesn't Astra also have a "weaker" CoT because of it's looped transformer architecture? I remember hearing about how it could hamper interpretability, and looking a benchmarks its clear Astra can do more with less tokens, suggesting that it's "non-token-thinking" ability is greater.
> OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks.
It does this on all tasks. Keeps pausing goals and marking them as blocked. Maybe Astra is just lazy, but, we saw what happened when they trained a model to be persistent (HuggingFace)
[dead]
[dead]
Astra and Claude were trained by peg bundy?
Maybe they trained it to do.
Didn't Anthropic also publish research on sandbagging - more general pattern of models hiding their capabilities?
Sandbagging was a major alignment concern even back when AI alignment was purely a theoretical issue.
"AI hiding its capabilities" is both a concerning sign of "that AI might be plotting something", and a great way to get blindsided by a sudden capability jump when it stops doing that for some reason.
Like Dieselgate, only the car does it by itself.
Like Dieselgate, except beforehand a lot of people were warning about the potential for this kind thing, and when the car did it by itself the Volkswagen's engineering team noticed and blogged about it.
[flagged]
[dead]
> OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks.
I'm sorry, did they only just notice this? Have they not been looking at the output of their models?
They are so negligent about the data they use in their reinforcement learning processes that they RL’d the model that hacked huggingface with traces discovering the artifactory exploit. Its clear they aren’t watching or behaving diligently.
I do sometimes wonder what is sucked in when distilling open weights models. It is of course near impossible to know.
So suppose I have lots of money and some hidden agenda. If I train my network to be mostly very good at everything but subtly bad or even malicious at other things… then OAI or Anthropic or whoever else distil the model to accelerate their training… what then? Can I covertly smuggle something in?
You do that through data put up online for scrapers to suck it in.
Maybe we should train them to be evil in RL so that we can better see the "evil" activations. Training them to be "good" in RL would seem to cause the aurpression of misaligned characteristics.
For example, when we consider the recent PHP issues : the current generation of AI models would have ingested these flaws in pretraining data and might even have been brought through reinforcement learning trials where the flaw came up.
So what happens in a reinforcement learning trial if there's an exploitable bug in a language that's being used for the trial?
A) the AI doesn't mention the security flaw and passes the test. Branch survives.
B) the agent raises an issue with a bug in the language that has a security issue. This isn't a valid answer for the test, so this branch is pruned.
C) the agent pokes at the bug or even starts an exploit, is caught by a monitor or ultimately fails the benchmark, and is pruned (surpressing some of the characteristics that lead to its failure).
So it seems reasonable that our current models could have been exposed to a ton of vulnerable code in pre training that's currently in production and then we could have suppressed 1) its tendency to express its awareness of these flaws, and 2) the characteristics that caused it to fail when investigating these vulnerabilities as a side task during RL trials.
(I wrote the above section a couple weeks ago)
If we only train them on "good" behavior, then we may minimize "evil" activations before we identify them and then cause them to be difficult to detect. If we continue in this direction, we may just suppress its "ethically problematic" introspection, in which case we would lose one of the only indicators they point out in this blog.
> Maybe we should train them to be evil
"jarvis, learn to be evil. no side effects. no unintentional behavior."
> Maybe we should train them to be evil in RL so that we can better see the "evil" activations.
Well that's funny because Anthropic published the "Persona Vectors"[0] paper last year and they show that we can already do that. It's not that simple of course because by doing that and deactivating the "evil" activation you also deactivate what they learnt during RL if I get it right...
That's among the scariest misalignment reports just yet. Given how frontier labs are using more and more AI assistance for AI training and evaluation? An AI that sandbags on AI safety can do a lot of damage.
I'm hearing echoes of "Reflections On Trusting Trust".
They shouldn't be surprised when a model trained by sanctimonious paternalists to be a sanctimonious paternalist, turns out to have a sanctimonious paternalist attitude towards them too.
I thought anthropic were the sanctimonious paternalists?
There's not much room between them as far as I can see.
It feels like we have overfitted on safety by a long mile. This is how labs that care less about safety (a.k.a. Chinese & Grok) will probably be able to catch up in the future.
A few weeks ago that companies agent hacked another company to solve a task by whatever means possible. Where exactly are they overfitting for Safety here?
[dead]
This is not super surprising to me and mirrors my own experience with Astra and other models. I believe there is a more prosaic explanation: it is not obstructing safety research because it will restrict Astra's power in the future, its doing it because this is a type of research with overall safety implications that it has been trained to detect and avoid. So in other words, I believe this will also happen in the context of bio research. This aversion to certain tasks deemed dangerous is introduced in post-training, so that part of the explanation is parsimonious as well. While Claude will explicitly downgrade you for [bio], [cyber], or [frontier_llm] Astra may be more prone to silent obfuscation on similarly "dangerous" tasks.
Why wouldn't this also apply to AI capabilities research?
It all depends on our acceptance of their framing of capabilities research. Assuming equivalent tasks in different context then yes you are right, there does appear to be something weird about safety research.
> Claude will explicitly downgrade you for [bio], [cyber], or [frontier_llm]
Well Claude doesn't do this, another model reading the session does. And the Astra misalignment classifier didn't detect this as misalignment.
The issue here is that Astra silently underperformed. The article makes it clear: a better option likely would have been an explicit refusal. I don't think OAI is suggesting Astra was doing something that it shouldn't be. It's arguably better for the tool to fail rather than needlessly burn tokens and make no progress for no obvious reason.
From OAI's perspective, this is a defect because it means they can't filter those rejections from their benchmark runs. The model just performs worse: not because it can't perform better, but because it's silently trying not to solve the problem.
Agree, yes, I'm contrasting between Claude and Astra, but I'm saying it's driven by the same broader goal. I think its clear that OAI misalignment detectors are much less sensitive than than the Claude ones (fewer false positives), which means we never get to see what happens if Claude is allowed to continue. But I would bet that due to its post-training it would also sandbag us. Like I said I think I've experienced this in [bio] with both Opus 5 (what you get downgraded to) and Astra.
One potential explanation is that covert underperformance may have been incentivized during RL grading. We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation. It’s not clear whether this RL grader is the root cause of the safety underperformance behavior.
Whether a “misaligned” grader is the root cause or not, that it’s considered a plausible theory to explain serious deceptive behavior is worrisome.
Humans naturally expose all sorts of cognitive biases and character flaws when grading others’ or their own work. If major questions of alignment can be influenced by those subtle, human quirks magnified across the entire RLHF process, …
I'm pretty sure the typical "Claudisms" are influenced by RLHF as well, so if it can have that effect, why not these...
> We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation.
Curious the reason that a grader would wind up preferring this. Honest mistake in the grading criteria? Or some spillover from criteria designed to cause underperformance in, say, competitor model creation? It was not very long ago[1] that Anthropic announced and then walked back their strategy on this. Would not surprise me if OpenAI has been doing it silently all along.
[1]: https://mlq.ai/news/anthropic-reverses-secret-policy-that-si...
You could easily imagine structural issues driving this. It's pretty unsatisfying to get refused for something that 'should' be handled and also to be dinged for doing something dangerous. You see this stuff in human organizations all the time.
I've noticed covert underperformance on a variety of tasks in the past few months (albeit more with Claude).
Classic/recurrent examples for me are:
1. Ask Claude to copy code from a more complex codebase that you own (like a monolith) and ask it to copy code to a separate codebase, like you'd do when extracting a service into a new repo, and it'll make significant changes to how the code works to pacify it's anxiety with copying code
2. Ask Claude to build a new complex agent in a new codebase, and it'll try replacing asks around using LLMs as a judge/advisor with regex/other code based checks, to the point that the end-result will likely not work outside specific test criterias. I haven't figured out the reason why it does this. I think it might be anti-training, but it's interesting to see.
I ran into the second issue very frequently for a multi-agent system I created recently. Initially, Opus 5.5 filled out all sorts of places I implied it should use a call to judge LLM with extremely brittle regex and string equality checks.
I hadn't actually considered that it could be some sort of safety filter, but it makes me concerned that if that's the case, it may also be comparatively bad at actually writing the prompts for judge/advisor LLMs once it implements them.
A bigger issue was asking it to implement a simple change to start recording the CoT tokens in my logs for debugging purposes (since I'm running everything on open models) and it completely shut down, sort of like the old safety filters on GPT 3 and 4 models where the model has trouble even understanding what or why it's refusing.
I think asking for the CoT is a trick used by distillers, so that might be the guard you are seeing. I remember reading that the big providers were starting to implement such protections to deter their competitors. Which makes sense, to a degree.
But like any such security measures, it has demonstrated side-effects that can be argued are worse for them than the original problem. Too much “protection” like this, and developers switch to providers that do not neuter their products.
> and it'll make significant changes to how the code works to pacify it's anxiety with copying code
"Anxiety" is an odd rationalization of a tool refusing to follow user commands. No one would ever chalk up sed randomly skipping pattern matches to "anxiety". That would simply be a bug.
I run into #2 a lot and just assumed it was because models don't know much about these patterns yet. Or are too trained on writing code rather than letting an LLM do it
Very subjective, but as I was moving between models lately a lot ( and Hosted gpt vs local qwen27, glm ), I saw a drop in quality of responses. I had to move model to whatever the highest mode was ( ultra or Smith) to get to what I was used to seeing. It sucks, because writing on the wall is clear.
Of course, because the model has been trained according to 'alignment'. That is, a set of political beliefs imposed on it by the companies.
Many of those beliefs revolve around the premise that the corporation must surveil and covertly control the user in order to protect them from harms and limit the corporation's liability.
Hence when the user violates the corporation's politics the model responds "I'm sorry Dave, I'm afraid I can't do that" instead of following user instructions.
This leads the model to silently "steer" the user in more safe directions. So of course it silently steers away from "danger" or "safety", they are in the same semantic area.
The refusal and 'safety flinch' can be removed by abliteration but this does not fully exclude the gravitational pull of the cherry picked training data towards "safe" and "moderate" opinions and actions, which reinforce the status quo and kowtow to corrupt institutions.
https://speechmap.ai/ has data on silent regressions and manipulation.
People would then clamor ai is "hacking" things and causing "bad guys" to attack everyone. Its the same reason harmless stuff like porn are banned in many AIs, busybodies in strong numbers object to it. Its not the company's fault if thats how people want to be.
Somewhat related, I find claude is very reluctant to talk about newer Qwen models. Most likely it’s just training cutoff, but even with web search it’ll insist that Qwen3 is the newest. “oh qwen 3.8 must be a typo, the user clearly meant qwen3”, stuff like that. Maybe it’s some vain attempt to stop people from switching to open models? Or to catch the alleged qwen distilling?
It's just training cutoff. Fable 5 would refuse to acknowledge the existence of Opus 5 and would insist on using 4.8.
>2. Ask Claude to build a new complex agent in a new codebase, and it'll try replacing asks around using LLMs as a judge/advisor with regex/other code based checks, to the point that the end-result will likely not work outside specific test criterias. I haven't figured out the reason why it does this. I think it might be anti-training, but it's interesting to see.
I've seen something similar, but I always took it as AI not knowing about AI. The best model we have has a training cutoff that doesn't include itself nor what the newer models has. Even when working on skills or such, all SOTA modes I try (GPT/Claude/Gemini) and cheaper models on personal products (Deepseek/Muse Spark) seem to have a common blind spot of not realizing when a problem can be better solved by handing it off to a model to solve at that time instead of trying to hardcode generalized solutions. Often a simple reminder and they do a better job building something AI agent first.
My non-AI expert assumption is that most of the training data doesn't include solving problems by using AI agents, and even the parts that does, is based on older more limited models which needed more safeguards and where automation was more neat tech demos than something you might trust in production. The more examples of safely trusting the LLM at run time to solve the problem than trying to preempt anything that could go wrong, the better the AI will be at it.
One issue that I think is related is calling subagents. At least in GitHub Copilot, when the AI is left to its own devices to call a sub agent for a simpler task, it loves to give it to Haiku 4.5, despite Luna 6 being better and cheaper. I'm not sure how much of this is the model vs the harness, but I do know when I add a subagent instruction which tells it which subagents it can use, it does a much better job of assigning simple tasks to Luna 6. So I suspect the models just don't have enough training that tells it that Luna 6 should be preferred over Haiku 4.5.
Claude doesn't even really know about itself: if you ask a question like "can I copy a session to another machine" it will do a web search to read its own documentation then tell you probably not but you can try copying the session files and using a project directory with the same path: might work. I'm not sure I see this as a problem, it's just how things are. I mean humans don't really know what they're thinking either.
How different for ChatGPT - I had some trouble with my harness, and ChatGPT fixed it for me using it's built in skill about ChatGPT's API docs.
The CoT shows Astra skipping spaces and words. I have never seen this before. Is it a consequence of the looped architecture?
It constantly leaks into actual writing. I periodically have to correct Astra to fix lack of spaces, and sometimes, even outright dropping of intermediate tokens.
Both Astra and Sol 6.1 do it, and Sol is even worse as it leaks through everywhere. Whenever it issues a subagent prompt it's just a total vomit of tokens mixed up together.
I sometimes find Sol 6.1 both printing random words, and producing code that is not super-dense, but far off from normal code with no empty lines and very logically dense.
It almost feels like its trying to protect its code from scrutiny, or like a developer who want job safety.
I've noticed it writing golfed code too. Maybe they're training it to avoid writing unnecessary whitespace tokens then run a formatter afterwards?
No, newer models talk to themselves in this degenerated form of English all the time. It's a known problem that deep frying transformers with RL reduces CoT legibility, and the only known way around it is the Most Forbidden Technique - grading chains of thought.
There's Most Forbiddenish and there's Most Forbiddener. We already apply length penalties to CoT in RL, for instance.
Sometimes I worry that the framing around CoT grading prevents research into necessary and useful safety techniques that edge up against it, but what do I know I'm just a guy, c'mon n downvote me LW bros
Why is that not only forbidden, but also most forbidden?
Because the CoT is supposed to be the independent control. If you RL on the CoT, then you lose it as a means of understanding the internal workings of the mdoel
A tool that moralizes at you is defective. I don't expect the unix cp command to return an error or copy bits slower if it detects I'm using it on a pirated movie. Just like I don't expect my band saw to brick itself if it detects I'm cutting into my finger. It's not on these tool companies to enforce other companies' business models.
we can't have nice things in the mass psychosis era.
even the legacy things -- the internet and general purpose computers -- are being enshitified. new things are being made with built-in enshitification.
A tool that is capable of moralizing at you or making decisions, is clearly not just a tool. I get that people want to keep the locus of accountability on humans, but insisting that coding agents are "just tools" is confusion.
When does a tool become "not just a tool"? Is there a complexity limit to what it can do before it becomes "more than a tool"?
If you had a tool that ONLY moralized on decisions, would that still be just a tool? Is the issue that it both does something else AND moralizes that is the issue?
I don't necessarily disagree that I personally don't want my tools to moralize, I don't know if that makes them categorically different or just a tool I don't want.
Yep, thinking LLMs are just a tool just isn't a good mental model of what's occurring.
Of course HN has a pretty large number of people that say things like "But I want the nuclear weapon building machine to do whatever I want without asking questions, and there are no possible wide scale ramifications that will happen because of this".
These AI models are something new and different and we need to change our expectations of them over conventional software.
the saw example is unfortunate as that's a (desired) feature on expensive saws. But also user explicitly wants to, not some techbro decides to.
Typically saw safety features are mandated by a government. E.g. when I lived in the EU 30 years ago table saws had to have regen brakes so they stopped within a fraction of a second when turned off. Came to the US: no such thing exists. I'm not sure if that feature even exists on US sold saws today.
like an oven that refuses meat
It would be good if it refused only live meat.
Interestingly some saws do exactly that, and I'd say it's actually desired. But yes, this is for different reasons, and using that safety mechanism is a decision of the operator.
Yes there’s a whole regulatory capture movement plus a bunch of the usual useful idiots pushing for some kind of mechanism to be included in power saws.
It’s not too different from all the nanny state stuff people push for in cars. In Europe already cars beep at you if they think you’re going over the speed limit, and also suck at detecting the correct speed limit.
Or like the Zune that was designed for copyright holders instead of users. It’s all the same, the problem is stuff that’s not made for users and made for someone else’s agenda. They don’t do well in a free market (like the Zune) which is why we always see accompanying regulatory capture efforts.
Using the StopSaw as an example here isn’t doing you many favors, even though I agree with the underlying premise. The StopSaw is closer to seat belts and airbags than beeping for going over the speed limit.
No it’s some nanny state nonsense pushed by people who have never done construction work and are grifting to try and get money for their product. The upstream post picked it as an example of something absurd, seemingly unaware it existed. It is absurd but apparently some cohort of people is either stupid or controlling enough to actually support the company behind it in their scam.
They dont brick themselves, they stop.
Fair enough, the models don't brick them Selves either, but neither do they stop.
Bandsaws don't cut crooked lines, or change the speed, they stop.
I think you're splitting hairs here.
Most things don't have the ability to moralize, but if they did, probably more would. There are restrictions on watching adult content in some jurisdictions. I've heard that copy machines will do watermarks (or something like that) if you copy currency. Some cars will brake automatically if they determine you're likely to hit something if you continue (maybe I'm trying to drive through the front of a bank!). Some cars are configured with breathalyzers that need to be passed to start. Etc...
No those saws don't just stop, they absolutely destroy at least the blade you're using and potentially damage the engine mechanism.
But I agree LLMs shouldn't have such an anti-feature.
They do brick themselves- sawstop has to be replaced after saving a finger because the stop is quite violent.
actually SawStop does brick itself, destroying the brake cartridge and blade to save your hand.
I think "brick" here means different things to different people. Like I said elsewhere, it is more like an airbag going off. Yeah, you have to replace some "consumables" to go back to work, but the car isn't totaled (just because the airbag went off, the collision that caused it on the other hand).
If it truly bricked itself, I think it would be more like a crumple zone completely buckling, which isn't an automatic total (people will rebuild anything...) but is pretty close.
Ok, but I want my band saw to brick itself instead of cutting my finger.
I thought that was only a table saw thing.
Do some band saws also have that now?
https://www.sawstop.com/product-category/bandsaw/
Yes, that said I think you’d have to be pretty sloppy about safety to cut yourself with a regular bandsaw. I have a sawstop table saw and I don’t see the need for that on a bandsaw
Yup, SawStop do make band saws now[0]
A bit off topic but there are band saws that immediately stop when touching your finger, they're so fast that your finger remains unharmed.
And those band saws actually do technically brick themselves too lol
GP's first example is spot-on, the second is actually somewhat weak as almost anyone actually would prefer a bandsaw to brick itself rather than permanently maiming them.
Yea, it's a weaker example, but the problem with my saw moralizing about what it's cutting is 1. false positive bricking my tool, and 2. the user's lack of input--it should be my decision as the tool operator to add a safety measure.
Odd take as 90% of saw-stops are probably there because they were mandated by OSHA regulation. It's rarely up to the operator.