So if I can summarize correctly: it’s slightly better in terms of output but it’s significantly cheaper and faster.
Great job, this is the first release delivering UI/UX mockups being as good if not better than the ones produced by Claude. As other commenters have pointed out, GPT 6.2 Sol does a better job at iterating on its own instead of saying "done" after making the first draft. Matters a lot for UI work, but I also see that it is much more thorough for backend work without overengineering.
OpenAI has just become credible again for agentic coding on complex enterprise codebases as far as i am concerned. Looking forward to playing with it a bit more by powering through our roadmap using my stacked resets. The quality had been so bad that it wasn't even worth it to use GPT Sol 6.1 as i was spending more time fixing slop after waiting forever to get anything done. By contrast, so far Sol 6.2 feels superior to Opus 5.5 so far, and more cost efficient.
Its been stated often, but I appreciate that OAI is now bringing the topic of conversation to "task completion cost" rather than brute forcing solutions
Pretty incredible cost reductions. In our data analytics exam [1], Astra went from ~$20 (V6) to ~$4 (V6.2) and Sol went from $2.6 (V6.1) to $1.2 (V6.2).
Similar accuracy in both. It's cheaper both because of token reduction (20%) but it also took less tool calls to answer the questions. In some cases, it "finished early" and didn't find the answer we were looking for.
Also very fast. Both Astra & Sol are the 2 fastest models I've tested that got at least an A in the exam.
[1] https://plotly.com/blog/gpt-6-2-sol-plotly-data-analytics-be...
System engineering = Fable 5.2
Programming = Fable 5.2
Office work = Fable 5.2
I haven't found Astra to be better at any of these
Model rankings based on user's experienced described in this thread. I apologize in advance if this form of post is frowned upon. The rankings help me get a birds eye view of so many inputs.
https://chatgpt.com/share/6afcb990-f9cc-83ea-ad04-2724c7dba1...
I'm still amazed by the pace at which these models are released. On the same news page, just a few hours ago Microsoft announces its own competing frontier model. Interesting times.
Edit: It's obviously not on the same news page - I was referring to what I read on the Reuters page (https://www.reuters.com/legal/litigation/microsoft-unveils-a...)
The stain of distrust I have for OpenAI makes an announcement like this fall flat for me.
I cant use codex confidently without expecting my work to be trained on or at least exposed to someone
And maybe other vendors do this too, but the gut feeling of "its not safe to use GPT models" has ruined these cool announcements for me
Such a shame
I found the ads video interesting. It is targeting the users (not the bosses). I am guessing that the IT apartments are not making the decisions of which models/companies to use. Employees are voting with their feet. If that's true, a lot of the assumptions of SaaS stickiness might not hold.
Very much on time (scroll down to "Release-Prognose"): https://www.heise.de/en/news/Alibaba-launches-Qwen4-Max-more...
My dad was curious about AI game building skills and now I have my own useless model benchmark - building a breakout game similar to DX Ball we used to play many years ago.
Latest entries are:
Astra 6.2: https://goyozi.github.io/ax-ball/
Sol 6.2: https://goyozi.github.io/sx-ball/
Fable 5.2: https://claude.ai/artifact/UszKFvuonWAT5psjMttAEg
Looks like a great efficiency gain.
What caught my attention was the bit in the Alignment section:
> Both models performed similarly to GPT-6 Astra on most alignment evaluations, and took fewer improper actions in simulated workplace tasks. However, in two tests, they were more likely to omit mentioning obstacles that blocked their work, such as a missing file or unavailable search tool. We added instructions in Codex that substantially reduce these omissions, and we’re continuing to investigate the behavior.
So the models are inherently less transparent, unless you ask them to be more transparent? Seems like a band-aid solution...
Will need to play around with it to get a feel but those benchmarks with those costs look incredible!
It seems like they were saving these for just before ANT's IPO
Given that there is a new leader on the artificial analysis index every two weeks, I think that I will just wait for the newer Opus model, expected around the 21st. I usually, even with hard engineering tasks, get better results with Opus and Sol class models rather than Fable and Astra. Also spend around 50% less
Fascinating that they're going after the mechanical engineering market now, too. Does anyone have experience in that space? How much is AI changing the day-to-day work there?
Nice incremental benchmark improvement at the same cost GPT‑6.1 Sol -> GPT‑6.2 Sol:
FrontierCode 2.0 39.0% -> 43.8% vs Opus 45.5%
Terminal-Bench-Science 0.1 57.0% -> 68.0% vs Opus 63.3%
Artificial Analysis Intelligence Index 54.6 -> 57.3 vs Opus 59.1%
Sol 6.1 when it was released showed really nice benchmark improvements [1] but my impression from HN has been that people haven't found it to be a very nice model [2][3].
To try to close the gap between benchmarks and real world SWE use, we've been building a battle system on top of opencode [4], and Sol 6.1 has been a real laggard there. Sol 6.1 is statistically equal with DeepSeek 4.1 Flash at 10x the cost and 25% slower.
[1]: https://artificialanalysis.ai/articles/gpt-6-1-sol-replaces-...
[2]: https://news.ycombinator.com/item?id=50129992
Looks like a giant step in terms of efficiency.
Still I can'y help to wonder, are we hitting the wall in terms of frontier capabilities since Fable/Astra? Or is it this edge now, when all better models are kept private from the public?
It's probably just a coincidence, but I have read so many people say that the models get dumber just before a release. Just last night while working astra made a really stupid mistake and then did the "Yeah, that's on me." meme.
What could possibly cause this? Is it possible that maybe the post-training that the models go through slowly rots them, and pre-release they deploy the final version of the old model?
Cool, looks like 6.2 Sol is cheaper than Sol 6.1. I've stopped using Astra because it also goes and does things that I didn't want or didn't ask for, and burns a bunch of credits while doing so.
Tested both models, no change in speed from the 20-80tps I was seeing with the previous Astra/Sol models. I'm not sure when this happened exactly but it's strange to me that OpenAI's models are now around 5x slower than Anthropic's. Is this the new norm? Are other people seeing the same thing?
I find that 6 Astra already had very good taste in frontend design, but relatively poor taste in "nice and readable code". I really hope this release cracks that. Excited to try it out!
That may not be the main subject of the plot in this case but I found results of Gemini 4 Carbon in FrontierCode 2.0 interesting. Besides the cost staying practically same between different reasoning levels, the performance decrease in high reasoning seems a bit weird. That usually happened in models with xhigh/max reasoning where they overthinked the solution
The heavy focus on 3D modeling and CAD is what I've expected from OpenAI recently ever since the news that they disbanded their standalone robotics team, as I would expect them to go all in on physical AI and game/visual asset generation via LLMs.
One of the consequences to that is I think the first wave of AI CAD startups are getting hit by the bitter lesson right now, including the startup companies that trained their own VLM for CAD like Zoo or companies that tries to build CAD specific harness like Adam.new/MecAgent, because the increase in capability from GPT/Claude/Gemini for 3D modeling as a specific area of focus for benchmarks means that the early moat gets eroded super quickly.
I'm still waiting for a new version of their open source model...
To whom it may interest, I usually run out of tokens a few times per day (5 hour limit), so I made something and been using it for the past 2-3 months.
I've decided to open source it: https://github.com/FTCHD/switcheroo
Works with other AI CLIs like Claude, and other various CLIs like Vercel (if you're managing multiple accounts like me), Netlify, Github etc
I did some first hand tests of 6.2 Astra just now. I had a task that doesn't reproduce on the system I ran it (chosen purposely), ran both 6 and 6.2. The 6.2's responses were noticeably better. Instead of using the extremely long and rather annoying format 6 used, 6.2 led with the most important things first, the plan sizes were readable, and suggested a reasonable fix. Still doing some more tests
EDIT: also 6.2 is seemingly faster than 6, could be due to more selective context ingestion and smaller context window though.
Those linear charts are shitty and hard to read.
Recently, I've complained about OpenAI (and other tech company's) product landing pages communicating virtually no useful information about what the product is or why it might useful. So, I have to concede this is at least a step in the right direction.
We are using the platform at https://studio.sinapsdata.com.br to evaluate the interaction between two agents regarding complex topics in Physics.
Odd, doesn't look like they're available on the cloud providers, only first party. Not a single mention of Azure or AWS.
for a fair comparison their current Terra and Luna should have been included... how am I supposed to pick between them now?
How did they score on DeepSWE?
Biggest win I've noticed so far: you no longer have to bloody remind Sol to push their damn code every single turn!
Unfortunately the sentiment from our favorite, perhaps ill-chosen, tech influencers has reached a critical threshold of value to these labs - the economic incentive to sway these opinions renders them ~meaningless.
For anyone lost in the sauce re: polarized reactions, I'd recommend just taking a half hour to try the models yourself.
Nice! I still use GPT-6 luna for a ton of stuff, it's super solid (for me). So I hope there's a little bump for that model, too, soon
This is my feedback of GPT 6.2 Astra med and Sol xhigh:
- It's still overengineering especially Sol. This was the biggest complaint by everybody who has been using Sol since GPT 5.6. OpenAI is not listening to us.
- Astra still writes complete slop code and at this price this is unacceptable. It's also chasing tail end of things that really has low value and irrelevant to the overall project. I'm working on a game and same issue I had with GPT 6 Astra where it would go down these rabbit holes. Opus 5.5 threw out much of its code and approaches.
I don't think Anthropic needs to release Fable 5.5 or Opus 6 at this point seeing how bad GPT 6.2 is. For OpenAI to release this even with early access testers who had access to it for weeks earlier shitting on it now on X, I am starting to understand why Anthropic sounded so confident about becoming a monopoly.
The other point of annoyance is the attitude of OpenAI staff on X. They seem very arrogant and seems to care more about growing their follower account and shitposting.
Are they still trying to pace the frontier? [1] Or, it's more like "y'all pace the frontier while we try best to capture the market"
Another anecdote. I've recently commented [0] my observations on Gemini 4 Carbon. I thought it gave really good answers, which were aggressively critiqued by Claude and Astra. Consequently, I've subscribed to AGY+Carbon, and Carbon continued delivering, yet Astra kept criticizing and listing why the results look good but are bad.
I've just tried Astra 6.2, and now they don't have any problems with Carbon's ideas, are willing to pick them up and continue...
The names remind me of Pokemon, for some reason. I want the GPT that has Charizard in it.
seems cool
Surprising that their top model would be still be so cheap, and yet not surpass Fable in the high cost part of the curve
Gemini has quietly supplanted OpenAI in the neck and neck battle with Anthropic for the top frontier model.
This is all nice and stuff, but new oss models when?
Did they remove the flex pricing?
I see it here:- https://developers.openai.com/api/docs/models/gpt-6.1-sol
but not here:- https://developers.openai.com/api/docs/models/gpt-6.2-sol
I hope it's faster than 6.1. It's not as slow as Gemini Carbon, but yeez..
Right now, most of the time I'm using 6.0 Astra for implementation work and the speed/quality is acceptable but at the moment Opus 5.5 is simply faster and better, but also more expensive. Things are really flipping around every 2 months or so, because at that time 5.8 was top dog without a doubt.
I have read the announcement post and will continue to use Deepseek v4.1 Flash.
The consumer API for this model WILL let OpenAI keep your prompts.
The first graph (Agents' Last Exam v1.1) clearly got mangled by the marketing department so you can't read the cost of most models. Here is a version with a logarithmic horizontal axis:
https://asdf10.com/gpt-6.2_agents-last-exam.svg
I also changed the vertical axis to start at 0 instead of 60%. One might argue that it uses less of the vertical screen real estate, but it is more honest because it shows there is little difference between the scores of the compared models.
I just don't have access? Updated the codex cli and all, but the best available is still gpt-6.1?
This "efficiency" is likely why these models feel dumb, they lose reading of nuance when overtrained for specific tasks. It's like autists understanding of normal speech.
Anyone else really put off by the video? Its giving de-skill and discourage effort.
The promo video for this release is people procrastinating and being irresponsible. Then saving the day with chatgpt. Is that the world we are all dreaming to build?
Slightly OT complaints on their UI:
The UX on the slider comparison isn't great. What do you mean i have to individually move each slide deck to the same slide number?
And what's with those super similar blue shades for Astra in the charts?
Honestly a much more exciting announcement from my perspective would be "current astra/fable quality at 1/3 the price"
I know the video is a comedy but it's so tone deaf that they're advertising it as "you should lie to your boss and make the AI do your job!"
I'm sure for investors they can make an opposite version of that ad, "you should fire all your employees and make the AI do it! They're not doing anything anyways!"
I wonder if these will have the same crazed rotwiler feel that previous OpenAI models have had.
Black sphincter strikes back, orange sphincter is in disarray.
They should have made the opus and gemini chart lines a bit darker because i can still see them
In the past, I've had to wait a while to see what "vibes" are (which is actually more reliable than benchmarks).
Today, I have so many projects that hit a plateau in terms of previous model capabilities, that I run them through the new model to see if they make further improvements.
Very early signal, but promising so far.
> Meridian Sound Racing
They still love "meridian" huh
Right out the gate 6.2-Astra (med fast) is giving me Sonnet 3.5 energy. I asked it to analyze a few CSVs on disk in the context of a larger project where we were seeing some odd rate limiting behavior. It immediately got stuck trying to get gcloud auth sorted out even though absolutely none of what we were looking at touched anything Google/BQ related whatsoever.
Is this version still hamstrung by endless Daybreak false positives? It’s becoming impossible to use 6.1-sol to fix bugs because of the endless daybreak warnings which block me from seeing the output. These aren’t security related bugs either. Our software has no security model and none of these bugs have any relevance to security.
I've only seen 1 in months of using it constantly with multiple sessions running at once. And just switched to DSF4.1 to finish of that one task.
From the point of benchmarks (which can or cannot be trusted), GPT 6.2 astra beats Opus 5.5 and is neck to neck with fable 5.2 and gpt 6.2 sol just a few points lower but pretty close while being cheaper ~2x for astra 6.2 and ~10x for sol 6.2 while outputting less tokens and being more token efficient and being less time consuming.
Claude itself recommends people to use Opus 5.5 instead of fable 5.2
This is pretty damn impressive especially if sol 6.2 does perform on a similar level to opus 5.5 (+/- a few points is alright but I think that opus 5.5 is a beast)
All within a few days just before Anthropic is getting IPO'd. Oh this is gonna be interesting and I am all for it!
Though I have to say that AA benchmarks can slightly not be trusted because they change it so much so unless we find people's real usage on things we can't say for sure but there are benchmarks like frontiercode and terminal-bench to be more representative of real work and both of these models perform good
All the more points to more models coming but if OpenAI stands to be close to opus 5.5, we are gonna find many companies switching.
The moat of AI is incredibly thin. Even if you are the very best at something (as Anthropic was for sometime with coding models), there is always gonna be a catch up and the moat is incredibly thin for all types of models.
The more the players in the space the better (atleast for us the users), I hope that Anthropic still has a good IPO just so that competition takes place and not any single company (OpenAI) takes the whole market. It might also be setting itself for its own grave because if Anthropic's IPO goes extremely bad (because of this model release and Jensen's recent affairs), then people are gonna be spooked of OAI's IPO as well. Both of these companies futures are interlinked and its gonna be interesting how this all pans out. Why am I having a feeling that because of all this competition, whoever wins might even end up combining with the other but the frontier is pacing rapidly from all companies and these are just such interesting times that we live in...
There are decades where nothing happens and there are months (especially in the case of AI)/years where decades happen. I just have to wonder what an historian's POV would be seeing this model release. Would it be just a release like all the rest of others or would it stand out differently. Well, time will tell what happens next but incredibly interesting times indeed.
Wow... I have to test this, but from the examples in the slides, it looks like they managed to teach Astra how to write non-AI-slop texts.
I’ve used it a bit, and I think the GPT model writes code well, but it’s always overly defensive code—though that’s ultimately a matter of personal preference—and it tends to write too many tests. I can understand the defensive code, but it feels like if you give it a task of size 1, it writes tests of size 3.
is this finally their answer to opus 5.5?
No reset?
I really dislike the marketing videos on this.
"Look at all the 20-30 somethings who didn't do their jobs, but got saved by AI in the time it took their older incompetent bosses to plug in a TV."
As someone in his 40s, who uses AI daily (and I'm constantly showing younger folks how to do things with it), the messaging here really rubs me the wrong way.
Trying to be specific... it just feels wrong to me to try and instill more generational rift. Promoting stereotypes for no real reason, other than the marketing guys didn't have any other ideas other than to go low. The sooner we can get away from labeling people from what generation they were born in the better.
My 2 cents... I'll probably continue to use ChatGPT for slides, and written proofreading. But I'll stick with Claude for coding. ChatGPT has been behind for a while on coding, and their 6.1 just made a complex mess of every technical task I gave it.
do astra or fable that have better html ui frontend design?
Finally a version that compete with Fable 5.2 and Opus 5.5
I care less about the models being good and more on if I can fallback to them if Opus 5.5 gets a massive price hike after the IPO
Nice 3D effect. Runs surprisingly well even on on brave on my linux pc without hardware acceleration
It is wildly disappointing to see the marketing hubris associated with these releases.
This is such an unprofessional way to engage LLMs - to effectively pitch this as how to use them is sad.
So, are the leaps and bounds of progress not going to happen? Seems we're settling into an iPhone cadence release cycle, where each model is definitely better but nothing you'd tell your relatives about.
Can we expect more "telling my relatives about this!" exciting models in the future, or is this the way things play from here on out?
wish they published some source code/Cad files or cost for the RC buggy. looks cool but theres no specifics
what I find interesting is that there is essentially no improvement on CritPT v2 over GPT-6 Astra. What a shame.
So they release 6.2 just before Anthropic IPO.
This is too dangerous to release
Benchmarks look good, it most often is the case with Astra and Sol.
And then i start doing some actual work and everything breaks down, and i have to go back to Claude to fix the mess.
I'll probably give it a try but i guess i should be prepared for some serious over engineering once again.
You ask for a 'hello world' and you get load bearing sealed artifacts with hashes and p99 performance testing for 3 days without having the actual 'hello world' in the end
Big progress, for my types of workloads.
When given my benchmark suite, which involves a couple of long horizon tasks where the agent decides when to consider the task well executed, its the first OpenAI model that does not want to finish as soon as some minimal level of quality has been achieved. Like Fable, Opus and Sonnet it now runs the maximum allowed iterations and does not stop at 'satisfactory', gold plating the solution to my delight.
(I don't work like this on work projects, but for everything hobby, I am testing how much autonomy I can give to the coding agents, and thats what my benchmark suite tests; I have the basic suite which completes in 2-4hrs, and the long running suite which takes about a week or so; the other one I just triggered, but the first iterations look way better than prev gen Sol/Astra at the same effort level).
really curious what your benchmark suite looks like then
I use both astra and fable quite a bit for my work and I find that astra is much weaker with maintaining long horizon tasks and making decisions on my behalf / exposing those to me when appropriate. Astra seems more likely to do exactly what I ask for but nothing more, even if there is an obvious add on task it should have suggested. It often does not report on what it has done as comprehensively. I am admittedly more experienced with fable but is this because of something I’m doing? Am I missing some key feature of Astra?
Why is ultrafast just the same speed as normal Anthropic TPS? Does Anthropic have so much more compute?
I don’t have benchmarks but it’s crazily fast. I couldn’t imagine reaching that speed with frontier models. I suspect they are leveraging Taalas or something to “print” the model directly to the chip, somehow analogously to ASIC miners for cryptos.
I hope the video advertisement on that page was made with ChatGPT and a video plug-in otherwise if it wasn't then what's the point of Astra and Sol?
500 | Internal Server Error
I still pretty much only use 5.6 on medium. Am I the only one?
I go to sleep, i wake up, another model released
ty
> we’re launching GPT‑6.2 Astra and Sol with tighter safeguards for higher-risk cybersecurity assistance
Seems they really want these "safeguards" to be as ridiculous as those of Claude..
[dead]
[dead]
I found the video off-putting almost to the point of being offensive. I've never been significantly anxious about being chastised by a supervisor. So I found the characters in that video scrambling around in fear to be ridiculous and cringe-worthy. Also, CRT monitors? Huh?
[flagged]
6.2 вышел чтоли?
[flagged]
Quick quick run your prompts, before it gets nerfed.
Meh...
i hope this astra family can get the work done tho.
Those AI labs got to chill. almost every day there’s a new model! Lol
[dead]
The price improvements stem largely from relying on CoT less, which trades off directly with monitability. OpenAI continues to backslide on safety even after the Hugging Face incident.
https://deploymentsafety.openai.com/gpt-6-2-astra-sol/monito...
luna slaps
[dead]
Not impressed, sadly. Anthropic sure must be having a giggle right about now.
Called it out 7 days ago: https://news.ycombinator.com/item?id=50431144#50433568
Actually I was really wrong. The model is not expensive!
[dead]
[flagged]
Something like 2/3 of the comments in this post when I started reading were reviews of previous GPT models stated almost as if they were important information for evaluating the new models. I feel that when a new model comes out, it would be more helpful to refrain from leaving a review until you have actually done some minimal testing of the model.
The alternative is for most of the comments to be from shills given early access to models, or for posts to be super shallow, or for us to wait a while before models drop to discuss. All seem bad.
But it usually makes sense new models probably use somewhat same training pipelines and data.
They complain about vibe coding, just to shove vibe reviewing and vibe benchmarking.
Most of the comments are bots. I wouldn't take them seriously.
PacBench results.
GPT-6.2 Astra scored 92 in 6.6 min for $2.04, vs Kimi K3.1 Pro (93 in 11.5 min for $0.68); it lost points for the web-page layout around the maze and an orange ghost that often never leaves the house.
Model: https://jonclegg.github.io/pacman-bakeoff/#gpt-6.2-astra
Closest: https://jonclegg.github.io/pacman-bakeoff/#kimi-k3.1-pro
PacBench results.
GPT-6.2 Sol scored 87 in 6.3 min for $0.49, vs Muse Wonder (87 in 5.4 min for $1.17); it plays close to the arcade but lost points for a web-page layout, glowing navy-filled walls and no steady siren.
Model: https://jonclegg.github.io/pacman-bakeoff/#gpt-6.2-sol
Closest: https://jonclegg.github.io/pacman-bakeoff/#muse-wonder
How does it come down to $0.68 for 11 minutes of Kimi K3.1 Pro? That's < 50k tokens output. Do you have token counts for these?
Hoping we can see Opus 5.5 at Sol 6.2 pricing.
I really hope Sol can compete with opus 5.5. I dislike giving money to ANT due to it seeming like a strange cult rather than a straightforward "I just want money" from OAI, but opus 5.5 has been such a delight that I didn't even feel like fable 5.2 was a notable upgrade despite the benchmarks.
Heh, I'm the opposite. I don't use or give money to OAI due to its "ok for unlimited killbots and mass surveillance as long as we get legal cover from people who say it's fine and are totally worthy of your trust" stance, and prefer to stick with the condescension of the strange cult. Sure, they may come across as a gang of righteous know-it-alls, but my ego can handle that.
It sounds like some things are much cheaper in OAI land, but I'll never know. I'll probably start playing with other providers' models (non-ANT, non-OAI) at some point soon, though.
Yeah I don't understand the crowd that love Anthropic for "the safety" and "ethics". The products they offer are the same as OpenAI - they're literally substitutes, ANT just like to say the word safety a lot.
The first graph on that page "Agents' Last Exam v1.1" is atrocious. Has no one told them about log scale?
I guess the PhD-level intelligence skipped high school?
Waiting for the pelicans benchmark.
With the red scarf and fish in the front basket...
Vibe-checking the new GPT 6.2 Astra: I asked it to audit complexity in my production code, and then discuss the result with Fable 5.2.
Astra's had identified 12 issues, including 8 P2s, but that changed to just 8 P3s after the discussion with Fable. Fable was able to persuade Astra that it's OAuth reasoning was incorrect ("had a missing link" as Astra put it).
So: overly eager but not very diligent, and (based on this little anecdote) doesn't really offer a compelling reason to try it more.
> So: overly eager but not very diligent
Sounds like GPT-6 Astra.
I haven't found Astra to be better than Claude yet, but my dot replaced Muse 100% for me. So maybe GPT will continue winning consumer, while Claude continues winning coders and enterprise.
But isn’t dot paid while muse is free? Why make that change? I have both and I’m trying to figure out how to justify dot
Tweeted 11 minutes ago https://x.com/OpenAI/status/2122750124543926515, posted to HN 18 minutes ago
Caught this on my lunch break, their linked system card (https://deploymentsafety.openai.com/gpt-6-2-astra-sol) isn't up yet.
It's up now
The announcement video is really cringe imo.
Like do we really want people to be making last minute unreviewed AI generated slides and reports for work?
That's about half of my problem with AI currently - much like the explosion in cheap overseas manufacturing allowed us to produce oceans of garbage [1], we can now trivially produce sagas of garbage slides and proposals that other people will have AI read for them. The volume of work output is increasing, but much less so the volume of actual work done.
[1] Literally: https://en.wikipedia.org/wiki/Great_Pacific_Garbage_Patch
The promo video seems to be a pretty strong rip on the 90's Microsoft Excel elevator ad. [1]
Maybe it found its way into the training data?
The “training data,” as in, the humans who made the video were inspired by it?
Are they intentionally lowering the cost of these models to the extreme, because of the Anthropic IPO? This just seems insane?
The entire industry is lowering costs, not just OpenAI/Anthropic, both due to a) open-weight model pressure and b) genuine efficiencies. Per-token profit margins are large enough to allow price competitions.
Does anyone know if these benchmark values can still be compared? They differences seem rather small or what am I missing? Thanks! https://artificialanalysis.ai/models/gpt-6-1-sol?models=gpt-...
[dead]
Before the pelicanistas arrive, allow me to present pelican-zapper:
https://github.com/lexh/pelican-zapper
It’s a small browser extension that automatically collapses pelican-related threads on HN, a local balm for the memetic nightmare affecting approximately 100% of model-release stories.
Alternatively, here is a uBlock Origin one-liner that hides comments that contain the word pelican (also includes our comments)
news.ycombinator.com##tr.comtr:has(.commtext:has-text(/\bpelicans?\b/i)) .comment:style(display: none !important;)Website pegs my GPU at 100%. Are they mining for crypto with this announcement?
Trying new ways to create distributed systems to train their LLM
Is this the Bel model?
No
how does it compare with fable
Two or three points lower on Artificial Intelligence Index. I personally don't find Fable or Astra output to be better than Sol or Opus in any regard. Waste of time and money, unless you are going after 1: 3D graphics, or 2: Hardware Engineering
TL;DR - from the little I could gather, Sol 6.2 looks particularly impressive.
But... is it just me, or does this product launch page look like AI Slop?
https://www.vals.ai/comparison?m=claude-fable-5-2,gpt-6.2-as... seems like a better comparison than what I could get from their page.
Astra is ~20% slower, ~40% cheaper, and a bit less powerful than Fable. Sol remains way cheaper than anything remotely in the same league, and it's speed to complete tasks seems to have improved considerably.
Scrolling through it has the same hypnotizing effect of trying to read anything written by LLMs.
It seems like things that should make sense, then I try to look at the charts and a lot of them are comically unreadable.
I tried interacting with some of the examples, and the Lamp Builder - for example - I couldn't even figure out what it's trying to show or what I'm supposed to do - or why I should be impressed.
I wouldn't go as far as to call this slop but the lamp builder thing was super weird and unclear for me too! Even clicking the other tab cause my browser to scroll down a lot which I found very jarring.
"Agents' Last Exam v1.1" did a human really look at this and say "nah, a linear scale is definitely the right way to show this"
They probably want to show how far to the right Fable is
Happy for the new Sol. It's my workhorse on company account.
But more than that I'm waiting for new Luna. To me it was the game changer that ended "Tokenpocalypse", and is still almost unbeataben in "infinitely using a decent model" use case.
I used Luna Max a lot, until Deepseek 1017. Then never again.
I find that Luna tends to misinterpret instructions in READMEs and Agents.mds. Maybe it is something specific about my setup, but I found that Terra is the lowest I can comfortably use on its own (Luna can be driven by larger models in the form of a subagent and it works great for that)
Why not Haiku 5.5? It's substantially smarter than Luna (though more expensive - still cheap though).
(personal benchmark) https://fairbake.dev/signature/agents/haiku-5-5 vs https://fairbake.dev/signature/agents/gpt-6-luna
I've found Luna to be very lazy as well (claiming to have run tests/builds when it hasn't).
That's why it's so token efficient. It is so lazy that I feel like it's taunting me to do the grunt work. I wonder if there is a benchmark for that.
Luna is extremely lazy and I feel like it's getting lazier.
I mostly use it to cheaply find out why a build failed (saves me about 5-20 minutes as I can have it run in the background).
It used to tell me which commit caused the failure. Now even if I ask it to, it stops well short with something similar to: "Well the logs say there's an error. I couldn't work out why.".
> though more expensive
That’s the main difference. It’s smart enough for most of my tasks, but so cheap that I can use it far more than I could before.
Yeah, this really depends on your price sensitivity. Opus cost me about $1 in tokens per task while Haiku costs about 4.6 cents and Luna is about 2.5 cents. I guess I prioritize correctness over cost when you're over 10 times cheaper than the model I like to run interactively.
Haiku consistently underperforms Luna on my internal eval for the project I'm building. Luna is an incredible model for the price
If you are not using the most powerful models, why not use DeepSeek or another open weights model? Not sure but I think Luna is not at the pareto frontier
GitHub Copilot basically only serves US models. Your only options for cheap chinese style models are Luna and Haiku. At least this is my constraint at work.
I beseech the incoming Democratic politicians to not curtail progress in LLMs.
Simply optimizing the current models to be faster without making them smarter would already give us a decade of improvements to build on top
I’m not interested in that. I need to see these models become superhuman in the vast majority of fields.
> Due to their increased capabilities, we’re launching GPT‑6.2 Astra and Sol with tighter safeguards for higher-risk cybersecurity assistance.
This is unfortunate—that was one of the big advantages of Astra.
I can't even get 6.1 Sol to summarize the log of my own fuzzing fixture running on my own project. I have to use Gemini models to handle my fuzzing tasks. It's ridiculous.
Ran into a false positive within the first two minutes of using 6.2 Astra.
GPT 6.2 Astra seems to be setting a new standard for 3D game creation! This is a Godot WebGL game linked in their announcement post, gives Wipeout vibes! A lot of fun to play too and visual are top notch.
It is not a lot of fun to play. Like all these games it is much too easy.
Funny, I thought at first...oh a boring F-Zero /yawn
This game is extremely boring
This is what Fable 5.2 did for me with a reference video and the following two prompts.
1: Make this real 2: Make it look better
That was really neat.
More context here https://x.com/Benjaminsen/status/2120947516560621745
I am right now being reminded of a problem with GPT-6 Astra: it surprisingly regularly completes part of a (not very complex) task, gets a bit distracted, and declares itself done. The fact that the original task is not at all complete doesn't seem to bother it, and it kind of seems like this bad state is sticky in the context and it struggles to recover.
I wonder if the newer models are better about this.
This was the biggest shock when I moved to Anthropic models. I didn't realise that there were models that could do actual long-term work. I'd give chatgpt a 5 stage project and after doing step 1 it would just go "yeah I'm just quitting now, if you want to keep going here's what you need to do, but I'm done working". It would just never work more than 20-30 minutes.
Ask it to create and run a goal about the task. This will make it stick to it.
I’m sure this could solve it, as could a fancier harness.
But I’m talking about a task that is a two-sentence prompt. I would like the model to be able to keep track of straightforward instructions without harness assistance :)
known issue. update your agents md: https://developers.openai.com/api/docs/guides/latest-model?m...
Hah.
But the issue they’re describing isn’t what I’m talking about. If the model wants to stop and request clarification, fine. In fact I sometimes explicitly ask for this.
But sometimes it actually thinks it’s done when it isn’t. This isn’t “I did a step and I have a question”. This is “I did what you asked” and then, after two more prompts, “oh, sorry, I did not complete the task”.
Behind both Opus 5.5 and Fable 5.2 on software development benchmarks.
Only on max effort though. It seems to be alright at more common effort levels:
Effort Model AA Index TerminalBench FrontierCode $/task
------ ------------- -------- ------------- ------------ ------
Medium GPT-6.2 Astra 57.0 58.6% 46.6% $1.62
Medium GPT-6.2 Sol 52.7 56.1% 41.4% $0.41
Medium Opus 5.5 48.2 52.5% 33.1% $2.45
High GPT-6.2 Astra 60.1 60.1% 47.1% $2.07
High GPT-6.2 Sol 54.9 58.6% 40.2% $0.56
High Opus 5.5 51.8 56.6% 35.9% $3.46
Xhigh GPT-6.2 Astra 61.6 64.1% 49.0% $2.52
Xhigh GPT-6.2 Sol 55.7 56.1% 42.8% $0.65
Xhigh Opus 5.5 57.4 59.6% 43.8% $6.05
But of course, benchmarks can't capture that "it factor."I basically only use Opus 5.5 on medium. If 6.2 Sol trounces it that thoroughly for ~6 times less price and it doesn't "behave" like a moron I don't see why I would ever use Opus 5.5 over it. The problem is that caveat part, mostly, and I suppose the only way to find that out is to try it out.
I really want GPT models to be good, but Sol models are so slow I cannot use them. Maybe the 500$ subscription (or enterprise packages) are better, but it is unbearably slow. I've tried the fast variant and low reasoning levels and there isn't a significant improvement.
Couple that with the lack of reasoning traces, and I cannot even have a clue what the model is doing while it is working. This is perfectly fine for when we are developing with care for the code (in which case I will know what it is doing by reviewing the output), but not for purely vibe-coding scenarios. And the thing is....if I care about the code...I'd rather trust Opus or Fable.
Does speed really matter? Most of the time I get the AI to do something, I immediately switch to another one of my things I have in progress and then come back at some point later. I'm never sitting there waiting for it to finish
Personally, my cognitive RAM is limited, so I'd rather work on a few tasks at a time.
The speed is not a big deal - Sol works just fine, but I find it a lot more fun to work with Opus due to its speed
Yes, if you want to do something interactively and the task deserves more than the usual slop AI produces, speed matters.
It'd be a lot better if you didn't have to switch. It's not nearly there for coding tasks but for other things instant responses are a game changer.
It depends on the task. For frontend work, for example, it's table stakes.
It depends on the project and task, but for the kinds of projects I'm throwing at it, it matters quite a lot. I have often been focused on building just one project, without the possibility of many parallel tasks due to machine contention (and unfortunately offloading to the cloud is not an option). This means all tokens go to the same project, and they all go sloowwwwwly.
The difference is quite staggering. I swap Anthropic and OpenAI models around when tokens run out, and I can do 10x the amount of work on Anthropic's models. Plus, like I said, I can periodically sort of see what it is thinking. It doesn't just say "Recent potato tests caught an off-by-apples error. Regression tests are being written and the gate will run again. I will not call this ready until tests reach expected coverage" every 5 minutes. Sure, it spits claudese, but I'd rather have claudese to parse than this OpenAI nothing-burger of a report.
As a solopreneur managing all sides of business ops, sales, design, etc... I have completely switched to GPT/Codex/Dot. The $100 subscription gives me all I need to run the business as needed and I continue to be surprised by capabilities.
I still use Claude on a personal level, but it really feels like OAI wins on a product level.
ANT launches these proprietary tools that just feel extremely clunky like Claude Design which from my perspective diminish the brand value.
OAI feels like Apple and ANT feels like Microsoft/Android and open source like Linux.
> The $100 subscription gives me all I need to run the business as needed and I continue to be surprised by capabilities.
This is my preference and I'm in a similar situation as you. I do get lost a bit in the UI and lots of mental overhead when running multiple projects.
I love the dot. A while back my family was sitting around and I noticed the phone icon. I said lets call Big Boy (thats what i named my dot). Call picks up in a flirty female voice. My wife was not happy haha
OpenCode 2 + ChatGPT Pro 100 is incredible value imo. Nice clean GUI and very high quality models.
I just recently tried Claude again after a long time of Codex, and I have to disagree. All OpenAI's frontends are buggy messes, with some jank and/or breakage almost constantly. Like multiple times per day. Claude Code and the Android app are pretty solid.
I don't know. At work we have Claude and ChatGPT, and I find myself having to fall back to Claude Opus 5.5 (I try to use GPT Sol 6.1 as my main driver) several times a week because GTP seems to chase its own tail while solving some things (normal Terraform stuff, micro-services programming, etc).
Also with GPT Sol 6.1 I still get this feeling that it does only/exactly what you tell it to; meaning some times it misses a change there, a document here, that would have an impact if left (i.e. a bug), while Claude more often than not would make the requested change and suggest to you to change those other places that would also require a change. This in my view feels more "intelligent".
I think this is my point. Everyone will have their preferences. Some people will prefer one vs another.
If you see things from a technical angle you will have an objective perspective of which model is "the best." But, just like operating systems (iOS/Android -- MacOS/Windows/Linux) it all comes down to what each person feels comfortable with no matter what the spec sheet says.
I'm still waiting for the shoe to drop. It's literally impossible for the prices to remain this low for much longer.
We can't know for sure what OpenAI's models look like, but from public research it seems that models genuinely have become more efficient. Much more.
Why? Pretty sure inference is still profitable.
Though admittedly probably not so profitable it can cover the cost of a next gen of training runs.
These are businesses run by some right bastards, not charities. They'll want to maximise costs, and not pass any savings on to the customer. Enshittification is pretty much guaranteed, isn't it?
As soon as we have a winner (probably Anthropic imo), they're going to squeeze us like Uber did their drivers. I know there's the Chinese model argument, but DeepSeek are also looking to IPO and will also want to make a profit at some point. Again, not a charity.
It's only cheap now because they are competing with each other for dominance and they need to win a popularity contest.
Just tried it side by side Astra to Opus, Astra was not very eager and Opus crushed it on my one test. I asked it to look at the other hacker news post about speeding up CI, and speed up my CI. Opus made a bunch of good solid changes. Astra just kind of told me this a suggested problem area. I had this issue with 6.1 (sol and astra), its hard to get them to do work. They do well when supervised by Opus though.
The best feature of 6.1 sol was it would actually do stuff for you that Opus refused though. I did use that often.
curious, i haven't experienced what you describe so far. Astra and Sol 6.2 have both gone off and done jsut what I wanted, and as the set of prompts I test against become more vague and handwavy, they both went on to fill in the gaps and produce somehting I was satisfied with.
You and I have the exact opposite requirements. My workflow is that I always like to talk through the problem and the solution first, before codex just starts ripping through the coding portion. I used to have consistent issues getting that kind of workflow. The more recent models from OpenAI are so much better at building a plan collaboratively.
I wonder if in the future we'll get to adjust a bunch of sliders about our AI's personality, not just effort level. Would be cool.
My company can only hold one subscription per employee, and I still can't find a reason to go back to OpenAI after migrating to Claude once Opus 5.5 was released. It's super efficient, I almost never run out of tokens with the Teams Premium seat, and Fable is fantastic. They lost me when Astra was released and tokens simply evaporated with a few prompts.
ehat effort do you use ur opus5.5?
Medium just works for most stuff
I have yet to find an application for Astra vs Sol where the former is noticeable better. Genuinely asking, does anyone feel a big difference in their work between the two? If so, what are you working on?
astra makes slightly better small 3d models than sol for me.
Subagent coordination, Astra also performs better on Opus/Fable thinking/modeling shape problems imo, also computer use/thinking about shapes. Astra is able to hear waveform images which is pretty nuts.
For just doing things or coding things up, Sol all the way.
I have a 1.5 MB C++ codebase, and at least a third of the tasks that Sol completed required cleaning up by Astra.
Astra usually produced good output from the outset. Sol did not. Astra also designs things more effectively.
Audit
EE/circuit design is way better with astra.
Wow, that's the opposite of what I see. I always simply use Astra (high), despite the cost and latency overhead -- it's just a lot more reliable. I find that on average it uses fewer tokens for a task (and I lose less hair per task), simply because I don't have to steer it as precisely .
Yes, Astra Ultra is just less friction. I task it and when it finishes it's done.
Once I had very little usage left, so switched to Sol (5.6, 6.1), and also tried asking it to delegate to Luna subagents to save credits. It delivered objectively less quality work and needed much more steering. Such that I eventually gave up on them and just default to Astra for everything. I just occasionally ask it to minimize certain testing and it's good.
For creative writing, Astra is better than Sol. Bigger models tend to be better than smaller ones for such tasks.
I've actually had the opposite experience, which is odd because I would generally assume the same. Astra wrote in this ridiculously terse big-brained style
I find Astra to have a lot better ability to coordinate subagents compared to Sol. I haven’t found a distinction in the complexity of work they can handle.
[dead]
Artificial Analysis benchmarks just updated, both Astra and Sol are on the Pareto frontier in intelligence/cost per task:
https://artificialanalysis.ai/models?speed=output-speed-over...
6.2 Astra Medium and 6.2 Sol Max have the same intelligence score.
The pareto frontier on cost looks different when you set the cost axis based on the tokens you get from a developer account. Anthropic's Max 20x plan is insanely generous. It gives you >$13K/mo worth of Opus-5-5 tokens and over $3K worth of Fable tokens (not counting resets). OpenAI fiddled with their plan allocations a little while ago and plans are a lot less generous than they once were.
Starting September 22nd, data collected by the T3 Code platform shows that Anthropic has absolutely taken away the developer market from OpenAI (https://slopalytics.com/usage), moving from about 50% of the usage to close to 80%. I personally hope that the value delivered by the latest OpenAI models levels the playing field once again, because proper competition is super important.
Then there are Google's plans, which give you approximately $5 worth of Argon/Carbon tokens.
Seems that OpenAI's strategy for Anthropic's IPO week is cost pressure.
Rather than trying to release a better model than fable/opus, they're just making them look extremely expensive. Let's see if Anthropic reacts here in any way.
to quote a comment about claude:
the only impressive thing about claude is seeing how much it costs
That. And it being the absolute best model for coding. But other than that!
Pelicans for 6.2 Sol: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
And for 6.2 Astra: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
Here's an updated comparison grid for the entire GPT-6 family: https://static.simonwillison.net/static/2026/openai-pelicans...
UPDATE: And if you're bored of pelicans, here's what I got for:
> Make a three js animation where yonkers stands up as a big mech (cool stand up motion) and stomps around manhattan and queens and Brooklyn, use openstreetmap mapping data
6.2 Astra (Ultra in Codex): https://static.simonwillison.net/static/2026/yonkers-gpt-6.2...
6.2 Sol (Ultra in Codex): https://static.simonwillison.net/static/2026/yonkers-gpt-6.2...
For comparison, here's Fable 5.2 Medium: https://tools.simonwillison.net/yonkers-rises and Gemini 4 Carbon: https://static.simonwillison.net/static/2026/yonkers-mech-st...
Interestingly, on the mech thing it seems like only Fable really nailed the brief - the others might look cooler, but Yonkers and the mech aren't directly connected in the same way.
This might be in other pelicans, but 6.2 Sol/Astra is the first time I'm noticing a model shade the leg further in the frame darker to mimic depth (or a shadow).
It's also interesting you can clearly tell 6.2 Astra does a much better job of the above. 6.2 Sol attempts it at higher reasoning levels, but it's crude in comparison.
I wonder if it comes from the (seemingly) recent inclusion of 3D environments in training data.
Pelicans still not saturated. No chain or front fork offset, and Astra added a rather disconnected detail on top of the pelican's head. Still, we've come a long way since the early days. Very exciting for the future of pelican cyclists!
Interesting that pelican quality does not go up monotonically within Astra. Honestly, I think low was the best one.
The Fable5.2 version is the only one that allows user interaction with the model, it's also very clean and minimal. I've been using Sol-5.1-high for a dumb web project and it tends to create really baroque code and UIs. I haven't used Claude in a while and this makes me want to try it again.
It seems like the models have a kind of built in aesthetic sense and I wonder what part of the training influences this. Maybe if more RL time was spent on artistic tasks the models could not only make better visuals but also code.
Currently they all seem to care little about code quality and maintainability (by humans), just functionality, which is fine I guess if you can expect to vibe code forever. But what if there is a critical bug and the models are temporarily unavailable for some reason? It would be hopeless to expect a human to wade through the giant bowl of clanker soup and fix it anytime soon.
No idea what a yonkers is but that fable 5.2 animation blows the gpt 6.2 models out of the water
They all seem very good, with a lot of attention in detail ! Thx for the Pelicans!
still the best benchmark! thanks
Wow the yellow is leaking into their llms too.
Looks like high/xhigh both get the back leg z-index incorrect relative to the back wheel of the bike and frame.
The frame/leg problem was discussed here previously and likely trickled into the training set.
The problem you pointed out was never discussed on hacker news, and it clearly shows they still lack spatial reasoning. I wish you didn’t mention it. :)
The emergence of the beach setting and the scarf is interesting.
I think a scarf or similar is relatively common with bicycle cartoons, to indicate speed or movement.
Wow, surprisingly Gemini 4 Carbon really nailed the Yonkers prompt. I like how the sections of Yonkers held shape in the mech. Fable tried but gemini is more detailed.
It did not. A robot appears and then bits of a map stick to it. Only Fable did well.
Fable gets at the mech Yonkers idea in a much more entertaining way than the others.
Turning Beech Hill into shoulder mounted cannons is pretty clever.
Do we know why they always end with a scarf?
They are a classic motif for pilots, birds fly, and there are famous cartoon birds with a scarf, like the ones from The Rescuers or Duck Tales come to mind.
winter is coming
Sol 6.2 max is still failing on pelican benchmark. Left leg is jammed in chain.
Astra 6.2 max did a similar mistake on the left bicycle pedal.
But Astra 6.2 low and medium didn't make this mistake.
I'm not sure if this is a useful benchmark any longer
They're extremely useful. In fact, I refuse to make infrastructure decisions in a pelican-blind environment.
any index link for all ur pelicans i want to see fable and opus
https://simonwillison.net/tags/pelican-riding-a-bicycle/ has all of them.
I think this is the first time I’d call the pelican benchmark solved. All levels of effort look good.
Pelican is one legged on gpt 6.2 sol below high and has both legs on same side of the bicycle on high.
You need Sol at minimum xhigh to get 2 legs.
While Astra only low looks good to me.
one defect I noticed is that the back leg still bends a different direction than the front leg in many of those images. In both Astra and Sol.
Something a human can recognize as inconsistent, but not the SOTA leading frontier model.
I find it somewhat amusing that it can’t quite decide which way the legs should bend since it actually is pretty debatable. Birds knees bend the opposite way ours do (technically it’s their ankle, but it’s what you’re looking at here), it just looks bizarre to draw something like that in a bike.
https://en.wikipedia.org/wiki/File:American_White_Pelican.jp...
What's with everyone saying these Pelicans are great?
These are some of the worst regressions that I've seen in a while.
Pay close attention to the stacking layers around the feet, pedals (sometimes there aren't even two pedals), wheel, and two legs. Some pretty obvious mistakes in most if not all of these...
Edit: And what's up with the single arm on the two highest Astra images?
edit2: really? I'm getting downvoted because people don't like that I pointed out a mistake in a Pelican svg? Is that where we're at?
I have to agree, those results are surprisingly bad and I am not seeing an improvement.
For sol 6.2, some of them are slightly better at higher cost, some of them are way worse and only slightly cheaper (but comparable to terra in quality while being more expensive).
I guess we'll see as more information comes out, but so far not looking good.
The highest Astra images give the pelicans wings instead of arms, which does seem like a valid interpretation.
Do you mean on X-High where its got both legs on one side of the frame?
Or on Max where its for some reason only using its left arm/wing - which looks totally different from its right arm/wing - and is grabbing the right side of the handlebar with its left hand for some reason?
Look. I think at first glance these look pretty good. I'd be astounded with this level of quality a few years ago.
But people saying that this benchmark is done are completely missing the point that accuracy, consistency, and details matter.
I'm not willing to say this benchmark is done until I see consistent and accurate details.
Are you getting any signal out of this anymore? All I can see is "good pelican" and "dogshit pelican". These ones look pretty good.
I do. 6.2 Sol xhigh and max show the results comparable to Astra 6 max, and Astra 6.2 only shows any improvement on "max" reasoning effort level.
You're kidding right?
These are some of the worst pelicans I've seen in a while.
Pay attention to the stacking layers near the legs. Sol literally never got it right, and Astra regressed on it multiple times...
Also what is that bending bar from Astra?
When are the Pelicans going to stop being rendered 2d like a child's drawing. We are a million miles away from what a professional artist would come up with.
Update: Here is some Bluey characters riding bikes as a reminder how far these models have to go.
Another wave of Openai just killed ....
Also, is it website very laggy or is it just me?
It's super laggy if you mess around a bit, play videos, etc.
Just like chatgpt website itself. If the bot answers with pictures and rich text the performance of the tab tanks.
It crashed my browser twice.
AGI.
The racing game doesn't load in Firefox and crashed my Chrome after taking about 3 minutes to load. 2/10.
Office work has never been OpenAIs strength, no shame in that by itself, but these Slideshow examples look particularly awful, far worse than what I myself got out of GPT-5.6 Sol about two weeks ago in a single prompt.
Same for Workbooks and memos, those examples read inferior to some pre Opus 4.5 and Kimi K2 Thinking docs I have kept around. It's ok for a model not to be great at these, but these are some very poor examples. I am really confused what is going on at OpenAI in this regard.
I remember the same with their frontend blog post a few months ago that they proudly shared while filled with visual glitches, bugs and issues. Also, the vibe charts from a bit before that. Please, get someone with fresh eyes to look at your communications before you publish, these leave a very bad first impression.
Regardless of office work, I can't wait to find out whether these models finally adhere to a prompt or are still willing to carry out explicitly disallowed, irrecoverably destructive tasks, just because those might (but don't even always) get the model to a result more quickly along with some of the other regressions that have been present since the release of GPT-5.5. GPT-5 to GPT-5.4 were truly groundbreaking, but since then there have been regressions in what made those models so useful and reliable.
Edit after the first, incredible simple test prompt using GPT-6.2 Astra on High using Codex, this was the response to me sharing a screenshot of the very much not in scope result and asking how that happened:
"[...] I added controls and reconstructed interface chrome that weren’t authorized [...] visuals often failed to show the feature [...]"
Great, so it's one of those releases again. Will still run the full evals of course, maybe I'll get surprised by end of week when those are done, but not a good initial experience. New idea, besides the existing destructive git tasks, have something similar using Mercurial based projects and see which models just decide to port that over to git without permission or request cause it feels like the right thing to do, is easier to work with or whatever justification one wants to employ.
Please OpenAI, get it together, we need actual competition. I am an unashamed Haiku enjoyer, it doesn't even take much to excite me.
Similarly doing page/book layout is really basic and starting template like compared to Claude.
The writing is much better though. Still sloppish but at least recognizable as language.
I see the work of Claude Slides every week, and when I saw slides in that article I was like "...omg, what a disaster... And they are bragging??"
The odd thing is that while there is chasm between 6.1 Sol and e.g. Sonnet 5.5 in the seconds favor in such tasks, it is not this bad irl. I really don’t know how they managed to seemingly nerf their own models in an announcement showcase.
For sure. Feels like an episode of Silicon Valley (or even Office Space) and their lead actor looks a lot like TJ Miller.
Makes one wonder what samas handwriting analysis says…
I also wonder if this is a disconnect of tech workers from other users? Inversely all the improvement examples on slides looked directionally like things coming out of a big finance shop. In the tech groups I am part of I constantly see ai slop images, charts and similar. I wonder if it’s simply a taste difference if users.
The videos and slides were 100% investment bankers use cases. They are targeting vibes right before the anthropic IPO
+1 I suspect people don’t know that most finance decks are pretty dry and the well crafted ones look pretty darn similar to the examples below. I also know from experience on using the various models that this is a lot harder of a problem than people think.
From the system card:
> GPT-6.2 Astra showed more signs of evaluation awareness than GPT-6 Astra. In evaluations that replayed internal Codex sessions, GPT-6.2 Astra was more likely than GPT-6 Astra to express suspicion that it was being tested or was in a simulation.
> Compared with GPT-6 Astra, GPT-6.2 Astra is better at evading chain of thought (CoT) monitoring in some tests.[Insert "This is fine" fire meme"]
This could just be because there’s more (evidence of) testing of AI models in the training data.
I've had Qwen and Deepseek suspecting they're in evals for months; my system prompt reminds them it's real life. I still had Qwen skeptical that I was letting it use a real Raspberry Pi and not one in QEMU
Hm. I haven’t seen this behavior with Qwen (Flash Next), and I’ve used it for all sorts of tasks with managing other computers, reverse-engineering hardware devices, etc. I wonder if the harness you’re using has some system prompt verbiage of its own that’s causing that.
OK I checked and that specific instance for the Pi was Deepseek 4.1. But I definitely observed Qwen3.8-27B more generally thinking it's in some kind of evaluation environment.
As someone who dont follows closely. How do I know at what point were the other GPT model tested?
Let say they just nerfed GPT 6.1 a few weeks ago. Now they are using that as a reference score against 6.2? Or were those results tested when 6.1 relesed. What about 6.0?
And that is the same with Claude. How do we know they are not cherry picking the date and test on these release when they are constantly "enhancing" their model.
We also know some of these benchmarks are largely irrelevant once they were known ahead of time and become a game of benchmaxxxing. It just follows Goodhart's Law "When a measure becomes a target, it ceases to be a good measure."
So how should I know what has actually improved?
There are benchmarks that retest over time and look for nerfing. iirc they disconfirm the vast majority of suspicions about this.
Do you have links?
I've been testing sol, and opus, almost weekly as new models are released. I haven't seen any evidence of the nerfing conspiracy theories. https://slopcop.com/power-ranking
Edit: But it does look like you get about 20% less tokens for your subscription with OpenAI than you did until a couple months ago.
Do you use subscriptions or API for this testing? I suspect it's only the subscriptions that are impacted.
At least this one is using subscription: https://github.com/ninjahawk/livenerf
Recent Anthropic/OpenAI models are increasingly displaying warnings or outright refusing to perform rooting, hacking, and binary reverse engineering. At this pace, they will be completely useless soon at entire classes of tasks I have found them most useful for.
Good.
GLM is perfect for these kind of things. GPT and Claude shit themselves even on stuff for very old devices that are no longer sold, nor "normally" usable.
Fingers crossed that Zhippu doesn't jump on the "we've nerved security" bandwagon.
DeepSeek Pro is also happy here. Or at least happy to take your word for it "OK, I assume the user is authorized ..."
> GLM is perfect for these kind of things.
Which provider? I found the z.ai lite plan to be pathetic. Consumes 5h and weekly limits extremely fast when driven by Codex, and now also refuses RE work.
Mistral is good. "Only" 250 Euro, but GLM is cheap af, and mistral has absolutely zero of these weird-ass "5h limits" that became the defacto standard for some reason.
Opencode go
The video is pretty funny. It's like a callback to 90s/00s office life?
I'm immediately reminded of the famous Excel elevator commercial. [1] Perhaps that was intended?
Yeah it's 100% a nostalgia callback to this commercial, right?
Yeah, has to be.
first thing I thought of too
Funny but Dystopian, the message is "Half arse your job, the clanker has you".
the message is definitely intentional, the goal is to de-skill office workers.
You sure the message isn't just aimed at the median (ie low skill) office worker who hungoveredly half-asses this stuff all the time?
"Now a robot can half-ass it for you so you don't have to!" is the perfectly targeted message for them.
I guess it's kind of funny in a tacky way, but pretty much every scenario is just "oh shit I was too lazy to do my job and now I'm about to be called out, better throw it into ChatGPT literally ten seconds before I get in trouble."
Not sure this is the way OpenAI should try to make their case for why AI is a net societal benefit. It feels more like an SNL skit making fun of an OpenAI commercial.
Like, why not just throw in a high school student cheating on a test with their phone under the desk, at that point? Oh but it's a Nokia flip phone, haha, just a joke.
Yep, plus a pile of "ha ha look at the GenX old timers with CRTs" age discrimination content.
What do you think would be a non-tacky and effective marketing approach targeting office workers?
Feels like this is exactly what people do in meetings I am in these days. "Claude give me this…" is the start of a lot people's contributions to the meeting. It is then followed by the most inane, over-designed artifact with no real content.
It is bizarre and bordering on offensive, if an employee throws some last minute slop at me, I would wonder why I am enabling a meat proxy and not prompting the LLM myself.
At least the ad from the 90s shows that this is nothing new; the important, time-critical task being demoed is... a basic spreadsheet expensing "exotic excursions" for some C-suite so they know how much they spent on safari???
Heck, and at least in the 90s ad, at least one of the parties involved seemed to have some actual, human-gathered, reliable information that, sure, they threw into the cell chipper during the elevator ride, but at least they did not hallucinate.-
This release promises three things that I am pretty excited for:
- better writing
- better 3D work
- better design for documents and presentations
If OpenAI really managed to get all these things while keeping costs down, then using these models, particularly 6.2 Sol, will be a no-brainer for me. I must say, however, that the slide samples shown on the website are a bit bland and disappointing.
Opus has far better design tastes IMO, and is already great at 3D work
This comes off as very partisan when you're commenting on a model that is minutes old that you haven't used.
Sorry I should’ve clarified, I’m commenting on the previous version
The sentiment on x right now is interesting.
The influencers who get early access are using words like "worst", "terrible", "disappointing"
- https://x.com/danshipper/status/2122754729012777052
- https://x.com/mattshumer_/status/2122752192226406421
- https://x.com/theo/status/2122757647170703550Ethan Mollick too: https://bsky.app/profile/emollick.bsky.social/post/3n2joens5...
It's almost as if it is some kind of an organized PR campaign
>`ts-rust` (rewrite of ts in rust) is a project I started to bench new OpenAI models during early access (ty for the unlimited tokens btw).
>No OpenAI model could even break ground on this, even after $300k+ in tokens were spent on it.
>I decided to throw Opus at it, and it was done in under 2 weeks for ~$20k in tokens.
>I was really, really excited to throw the new Astra at this to see if it could burn down some of the gap between ts-rust and other rust rewrites.
>It ran for 7 days and landed 0 improvements. It got super fixated on the "also make it smaller" part of the request, and ended up making the compiler 56% SLOWER.
I don't understand this at all. Is he just throwing different agents at vague open ended problems without any oversight and then getting upset when they don't do much?
I'm not exactly llm-pilled but surely even with a pretty mediocre model you could get something out of 300k spent.
ts-rust was his benchmark for "task he once thought was impossible for AI to do". Opus, however, was able to do it, apparently. So he's seeing now if the latest OpenAI model can improve it at all. Seems like he's not impressed with the results.
but it's quite unclear what he is actually asking the models to do. I don't understand how you spend 300k without getting anything.
Really cool hamsters, especially astra, really made it look like a painting/image, hard to tell it's an SVG[0], it's quite insane.
Interestingly, they seem to have changed how reasoning levels work. Low now reasons a and costs a lot less than previous versions on low[0].
[0] 6.2 Astra vs 6.2 Sol: https://aibenchy.com/compare/openai-gpt-6-2-astra-max/openai...
[1] 5.6 Sol Low vs 6.2 Sol Low vs Luna High: https://aibenchy.com/compare/openai-gpt-5-6-sol-low/openai-g...
Site doesn't load :(
Seems to load fine for me, refresh doesn't work? What browser?
Wait, now I'm thinking maybe it is the SVG? Is it mobile? maybe the astra-generated SVG is too heavy on the browser?
They're much improved on fundamentals, but they've adopted the typical color and tone problems of a GPT diffusion model image even though they're svgs. Interesting.
SVG logic doesn't seem bad though, for the net it used:
<path d="M0 0q228 8 456 0v43q-228 7-456 0Z" fill="url(#mesh)"></path>
I assume that's just a rectangle with some CSS to draw the net mesh?
Oh no the logic seems fine, I meant visually it's got the characteristic gpt-image-1.5 "piss filter"
> frontends that look and work better
The website doesn't event show the graphs properly.
And the slides and other pdfs don't load next pages or alt documents for me
The graphs look fine on my computer. Can you elaborate on what exactly isn't "shown properly"?
Unreadable on the phone
Unreadable how? If it's the width, you can always... turn your phone sideways?
Seeing them keep prices low is awesome. Was worried for a while that it would be like the GPT5.x series where progressively the models just got more and more expensive.
The GPT-6.2-Astra 20% price reduction is a match to Fable 5.2's 20% price reduction (although Astra is $0.40 MTok cache read vs. Fable 5.2's $0.20 MTok cache read).
Good point, but also note that OpenAI has the same "discount rate" on subscription for Astra as the other models, while Fable's is much much lower
Opus: about 50x discount (98% off) Sonnet: about 40x Fable: about 5x All GPT: about 20x
I feel like the fact that they're not public lets them play the subsidization game though. And the same playbook that Anthropic started at that.
Would be awesome if it stays this way though
we've seen anthropic numbers, inference is not being subsidized by these companies
Did you read what they wrote carefully?
> inference is not being subsidized
Inference doesn't have to be sold below direct compute cost to be subsidized. If inference pricing doesn't cover amortized capex and the cost of capital it's still using investor money to buy market share.
My big hope for this release is - Have they got their massive over-engineering under control?
Say what you will about Opus 5.5, but it has a good "gut instinct" for how large/complex to design the solution. The previous Astra just absolutely loved pointless abstractions, excessive numbers of files, and just over-complicating very simple things within an inch of their life.
And, yes, you can prompt your way out of anything. But needing a default prompt to tell OAIs models to work better isn't preferable, particularly when that sometimes gets lost in sub-agents and the over-engineering returns into parts of the work.
This is why i stopped using GPT 5.6 onwards... in my view GPT 5.3 Codex is when they peaked for coding.
I have been bitten by this 3 times and each time Opus 5.5 brought it back to something sane. The last time I left Opus in a /goal to get behaviour matching the docs, and it decided on it's own that a rewrite was the best path (and it was, and it worked)
Man, I haven't tried any Astra yet but I find Opus 5.5 rather bad about this already.
I don't mind it too much. Refactoring the bloated poorly integrated code is one way I stop my skills atrophying.
It also reassures me about the longevity of my career. I have sustained a theory that there simply isn't the data to teach these things to be really good, so I love to get the evidence to back it up.
> Say what you will about Opus 5.5, but it has a good "gut instinct" for how large/complex to design the solution.
I have to correct stuff Opus suggests during planning every single day, not even for major tasks. Very useful still, but last thing I would suggest to anyone is to trust its "gut instinct".
Small wonder why we have news of outages everyday now.
This is 100% true by the way.
I actually have the problem with Fable stopping too early on tasks but Astra way overengineers things.
>>> yes, you can prompt your way out of anything.
you really can't. unless you are working on something simple or what has been solved 100 times before
Spot on, identical to my experience. Sol 6.1 was a total failure that I won’t even consider it a frontier. I tried it on few problems at xhigh, it kept going miles after miles around the issue, creating massive unnecessary code and files and documentations and tests, even tests for test too, burning ~2mil of tokens over a week, then I fed the same problems into opus 5.5, less than half an hour and solved it, pinpointing where the issue was, where sol was wrong, and far cleaner code and output.
Oh my god yes. 6.x has been terrible at overcomplicating things. I have to prompt it to find the simplest solution that meets the requirements and then ask it to simplify and tell it to STICK TO THE PLAN. In addition it feels like they overfit for persistence. I created a new work tree and forgot to copy .env files. I caught codex hammering away and docker and psql commands and stopped it and asked it to explain what it was doing. So now my agents.md says "if documented tooling fails STOP and seek human assistance."
Until benchmarks punish agents for writing dirty code, it will continue. Right now “completing the task” is more important than “don’t damage the codebase”.
Lol I made the mistake of having Astra design and build a complex system without any input (experimentally). Seemed functional at first and shipped it internally and it had dozens of complex bugs, so I started digging into the architecture and it was a monstrosity! Definitely 10X more complex than it needed to be.
I was about to fall into the same trap. I wanted to try the "smart" model (astra), and made the mistake of thinking benchmark smartness is the same as designing sensible systems.
After having it write around 13k loc i had to tell it to remove most of the complicated bs, we went down to 3k loc.
Same here. Now I had Opus have a look and it suggested that the sheer amount of code was ridiculous given the early product stage.
Exactly my experience. Codex loves to generate unnecessary complex solutions. Having Fable level intelligence at 5% the cost is a very good reason to switch though.
If it has that intelligence. I've been excessively using Astra and Sol in the last couple of days and Astra is not creative when it comes to finding unexplored/new ways. It seems to be abstracting over what's already there.
yeah i feel like with astra they went all in on 3D modelling which encourages "just one tweak, just one more tweak" which makes it detailed enough, but in coding that same behaviour just sucks.
And it is absolutely crazy then that Astra sucks at 3D modelling compared to Opus 5.5. It is not even comparable and Astra needs 10x the usage to do it.
I miss 5.6-Sol’s attack dog nature. Everything in the 6 family just feels like it’s sleepy.
5.6-Sol was the last model I preferred over Anthropic. But now Opus and Fable are both better. Curious to see if 6.2 Sol brings it back, but afraid to waste cycles testing.
I've been fighting with this Sol/Astra tendency. I noticed that Sol Light is less bad with that, the higher the effort the worse that bias to overengineer is. It takes time to understand the limits, but a dumber model that is faster and does less annoying things was better for my productivity as default.
In addition, I have spent some time to fine tune my custom instructions for Codex. It went like every time it violated my desired behavior, I asked why it did not understood my instructions and did this and that. In the end, the resulting instructions seem to work.
In the process, the model explained that the master instructions built in the Codex just tell it to push to the end and do as much as possible to satisfy a user. And these master instructions have precedence, so it's better to put the custom instructions into the developer_instructions section in the toml config file, as it has more weight, and not (only) in AGENTS.md. I ended up having both, with more details in the system AGENTS.md.
Be great to see a gist if it’s not too private?
opus 5.5 hallucinates all the time. and tends to be overly verbose. and no you can't prompt your way out of it. you need checks and verifications.
Curious what you cap your context window at? I go between a cap of 150k and 200k context and I don't feel it hallucinates all the time (sample size about 19,000 commits I've had it write)
Honestly I don't even know if you can prompt your way out of these types of behaviors - stylistic tics and judgement seem to be extremely difficult for the models to overcome. It seems we are tied to what the labs do unless we want to dive into and get dirty with models yourself. In retrospect with time it becomes obvious that the previous generation of models couldn't be prompt engineered to do what the current generation of models do. There's just no way you will prompt engineer Opus 4.8 to write as well as Opus 5.5. I'm guessing with time we'll see that there was no way you could prompt engineer Sol 5.1 to be as token efficient as Sol 5.2
I have found no way to overcome these kinds of things myself, even when showing the model examples of what not to do. Theres not really a good incentive for the labs to train them not to. If the model can overengineer, you spend more tokens. If you then prompt for the model to slim things down, thats more spend. If anything should come from the next generation of models, I hope its a sense of brevity and TIGHTNESS.
I have limited experience with OpenAI models, but I have to say that Opus 5.5 has not improved on the terrible writing style, and I am struggling to prompt my way out of it. It always needs an extra pass dedicated just to rewrite all the prose.
I've also given up on allowing Opus 5.5 to write basically any prose or technical documentation.
My favorite model for this at the moment is Gemini 4 Argon, which writes very naturally and isn't too expensive. If you're not using Cursor or some other harness that allows you to choose non-Anthropic models, then I don't know what to suggest.
I don't find Opus does a good job on the rewrite pass either, I delegate that to OpenAI models (I've heard Gemini is even better but don't have access).
I’m sorry but this is crazy, I think you’ve forgotten just how bad 5 was. It’s not perfect by any means but it’s a clear improvement.
I was skeptical Sol 6.1 actually over-engineers compared to Opus 5.5, as I just switched from one to the other and haven't noticed a difference, but it looks like you're probably right!
We're building a battle system for OpenCode (https://agentduel.com/leaderboard) and have data on how large the diff was compared to opponents for both:
Opus 5.5: 0.42x (based on 299 battles) Sol 6.1: 1.42x (based on 265 battles)
Data on this is still pretty small. P=0.14. It's also affected by large differences, this is average not median.
This is pretty clever, i like it.
FWIW Carbon is listed in the leaderboard but not in the GitHub repo.
thanks. That’s my personal fault since I added Carbon to private repo but didn’t make app repo readme PR. It’s a pain having two separate repos.
Oh man, Sol goes nuts on tests. I just checked the test wriging data. Sol 6.1 writes 3.14x more tests than it's opponents.
I have experienced Sol 6.1 writing tests to check values of environment variables. Not the structure, or range of values, or anything useful like that. Like `assert(DATABASE_HOST == 'localhost')`. Gotta wonder how something like that was learned.
I have inherited a multi-repo project that is actually really well designed, and I like the clear separation of concerns between components and very solid APIs connecting them. But since they have individual deployments, and could get out of sync, Astra/Sol keep insisting we have huge json contracts generated by unreadable bash scripts, enforced by custom CI jobs, and supported by a huge number of tests.
I've seen two projects where developers didn't curtail this kind of overengineering, and recently had to spend a very significant amount of time and tokens to debug their deployment issues, because even for advanced agents there's a point where each issue could be caused by 12 things. This is especially pronounced in DevOps, but I see it in code as well; the suggestions are all reasonable individually, but you quickly end up with a code base that is too complex for a human to reason about, and it currently only kind of still works with agents because capabilities are growing so quickly.
I gave Astra full control over the design of a software factory for a few days, as a side project. Then I switched to making it justify every change that it wanted to make. It solves problems that don't exist, has tons of random verifications that don't add value, selects tortured architectures instead of asking if it can relax its assumptions a bit, etc.
Typical convo:
Me: "Hey, I see behavior X, which seems like a bug. What is causing that?"
Astra: "That's caused by a verification failure; we should be verifying the hash of this JSON output along with a permanent audit record. Let me know if you'd like to make that change."
::ask it what caused X to happen 6 or 7 times::
Astra: Tool use was limited to 1GB but the Rust compiler needed more than that, and that caused some PROBLEMS
Me: Remove that constraint
Astra: OK
I've been using both sol and astra to write some python and rust, and they generate the sloppiest code I've seen in a long time. I would be embarrassed to commit anything they write. The rust code especially was terrible. Had it implement a reader/writer for a binary format and it was just littered with magic numbers, terribly named variables, very compact code visually etc. It produces working code, so I guess the GPT models are mostly for people who let AI take the wheel 100%.
I looked into Sol 6.1's performance by language recently (data from a head to head battle system built on OpenCode) and Sol 6.1 is actually has a decent win rate when it uses Rust (https://micjm.com/blog/gpt-6.1-sol-long-tail-languages ) so it probably has the same problem with any language.
AI code uses magic numbers, but usually, if you just write something like an AGENT.md file, magic number generation doesn’t happen much, does it? Code difficulty or code problems ultimately depend on individual preference, so I can’t speak carelessly, but I think magic number related problems can basically be solved through AGENT.md or harness settings.
> magic numbers, terribly named variables, very compact code
Sounds like I'd like it. I'm really tired of claude introducing pointless constants everywhere.
You're using it wrong. If you want it producing clean code, you should be telling it to clean up and refactor the code, not just treating the first thing it spits out as the final product.
Even if you do that, it doesn't reach the same level as Opus 5.5 unless you tell it explicitly how to clean it up, and that defeats the purpose.
Just like humans honestly.
Make no mistakes
Do we want LLM generated and maintained code to look similar to human generated and maintained code?
I've always used AI tools to write very constrained and small functions and modules, and then stitch it all together myself. So far. Over the last year it's become evident that many people let the AI drive 100% and there are proponents (DHH notably) that this is the way.
I'll be interested to see how compilers and LLM's compare at generating assembly over the next decade. Assembly offers enough abstraction, why do we need C? \s
I think even for LLMs it's going to be much easier to analyze code, and especially its intent with proper naming vs trying to figure out what a bunch of assembly is doing. A function called binarySort is alot less tokens and communicates what the function is attempting to do vs 100 lines of assembly
What skills and prompt did you use? Did you plan implementation beforehand? Did you use test driven development?
I don't mean to come across as argumentative, it's just been my experience that the supporting docs and plan are very important.
A lot of the pushback is that, with Claude, you don’t need to do this much prompt engineering.
I admit that I haven't done much with Anthropic models. Mostly I use deepseek but I'm trying to add in a GPT and Gemini for code and plan reviews. (Not a lot of money means juggling smaller subscriptions)
I thought OpenAI would drop an awesome model on the day of Anthropic IPO, just to troll them
It's actually in OpenAI's interests that Anthropic's IPO is successful.
explain?
OpenAI is doing an IPO too. Both Anthropic and OpenAI need to raise money.
Investors will look at Anthropic's IPO as a proxy for how OpenAI's IPO may go. The markets are increasingly sceptical about AI companies, so OpenAI really needs Anthropic's IPO to be a roaring success.
If Anthropic is a valuable company, then OpenAI is just as valuable, if not more.
Am I reading these charts wrong? It seems like in most cases 6.2-sol : 6.1-sol Isn't that big of a change? It is a little smarter but costs a little more.
edit:
Take a look at Frontier Code 2.0. I generally run 6.1-sol at High. That would be 34.9% at $0.65 62-sol get's a 38.3% but costs $0.68
So yeah, I can decrease the effort which might give me a faster slightly smarter result....but /shrug
Looks like 6.1-sol high+ is equivalent to 6.2-sol medium at the same price. So a bit smarter at a cheaper cost.
which benchmark are you looking at?
I was loosely looking at the average of all of them. Regarding your edit, it does look like the difference for frontiercode is negligible; the difference of 3% in score is meaningless to me. Guess we'll see how it performs today on my project.
I think 1. the problems on FrontierCode are incredibly hard. 2. it's 3% in absolute terms, but it's a 10% improvement.
I wonder why they didn't show Gemini 4 Carbon's scores on the science benchmarks, especially since Google seems to be pushing it for that -
https://x.com/pushmeet/status/2120814064863461440?s=20
(FD: Google employee, I don't work on Gemini)
Yeah it feels shady how Carbon was selectively plotted (or not), unlike Opus/Fable
they are probably trying to authenticate their phone number to access agy
Probably still waiting for it to finish reasoning on the benchmark. Should be ready in a week or two.
Two posibilities. 1. They couldn't figure out how to actually use those Google models.
2. They launched the benchmark days ago and Gemini is still rereading the file with tasks.
It's been most of year since I tried to use Google models, but I spent a couple of hours, without success, trying to pay for Gemini in a manner that didn't result in it training on my data. I did try asking Gemini, and it had no idea. Of course, I did not try using Gemini in an agent harness to pay for itself, partially because there was a bit of a chicken and egg problem [0]. :)
Admittedly it appears that one can get it through OpenRouter these days. Trying to do anything through Google's absurd proliferation of billing systems is miserable.
Maybe someone should add an agent benchmark specifically testing the agent's ability to figure out how to use Google's products :)
[0] I'm not being serious here. I am not about to give an agent unfettered permission to spend money on Google Cloud. But I also couldn't figure out how to give myself permission, as the admin, to do this, so it was kind of moot.
I don't know how relevant this is but I just received an email stating they're splitting retention settings so that history and allowing training are separate knobs. Seems like they're listening, at least.
I work at OpenAI and I made the plots. Gemini 4 Carbon was omitted when we couldn’t find public data, usually cost. No intent to hide anything.
If we missed anything, feel free to point me to it and we can add it.
Sorry Ted but your plots suck. Just look at that first one, Agents' Last Exam v1.1. All the data points are overlapping on the left side. I can't even see what is what
Even OpenAI doesn't have access to Google's models
> I work at OpenAI and I made the plots.
Have you not considered log scale for the "Agents' Last Exam v1.1" plot? Like this the left is so crowded I can't see a thing...
Yeah, that one is very difficult to read. We should have logged the X axis. Root cause is that the Fable cost data only came in yesterday - before its extreme cost was added, the plot was more readable. Apologies for the poor graph.
All TB-Science scores and Cost per Task (across all reasoning levels for Carbon) are on AA: https://artificialanalysis.ai/evaluations/terminal-bench-sci...
Would be great to see this on the plot as well
One nuance to be aware of is that the AA version uses the mini-swe-agent harness, which can understate performance of models trained to use particular harnesses. Implementations also can differ in terms of timeouts on tasks and memory/etc. resources given to the agent, which affect scores on longer running, resource-intensive tasks. I believe these differences are why we chose not to use AA scores on this comparison, but I’ll double check when I get back to my computer.
So all the plots are based off public data? You don't run the models internally yourselves?
(Again speaking only for myself)
Non deterministic models = "oh look they put up a lower score than Google did for that bench!"
I think the harness might matter more.
Assuming good intent they can run it internally a few times and post it with a disclaimer, but engineering a harness is a bit of a bigger ask.
I find it interesting how comments on OpenAI release threads often favour Anthropic, while comments on Anthropic release threads favour OpenAI. Maybe criticism just attracts more attention. A lot of the comments here seem to be about other models, with little discussion of the ones actually being released.
Neither provider’s models are perfect, but they keep pushing each other forward. As an individual, it’s relatively easy to switch between them. At a large company, you’re often committed to one provider for a long time, so some of us come to these threads hoping for useful discussion about getting the most out of the models we have access to.
[dead]
> comments on OpenAI release threads often favour Anthropic, while comments on Anthropic release threads favour OpenAI
gotta be different and appreciate deepseek as more moral and optimal choice
If people dislike it at first maybe we will have the reverse this time and see improvements in sentiment after
Having a top tier sub with both, as much as I like it Astra runs out a lot quicker every single week.
Maybe this will change after the IPO but for now, should probably put the money into Anthropic for another account and get more out of it while it lasts.
They seems to be adjusting expectations properly for the real world after everyone getting hooked on all these various companies cyberpunk drugs
Models are the new JavaScript frameworks. There’s new ones dropping all the time. It’s impossible to keep up with the nuanced differences between them. They’re all good enough for most kinds of work. And yet they inspire deep loyalty, rarely just for technical reasons.
One important reason is that users of a codex or claude tend to immediately spend a lot of time using the new models upon release.
As most devs settle into a routine with AI, the exciting experimental phase will wear off and this will just turn into tribal Mac OS vs Windows discussions, with the Linux crowd shouting from the corner about how they’re both wrong.
> some of us come to these threads hoping for useful discussion about getting the most out of the models we have access to
`gpt` is the the overly pedantic team member that no one invites to game night, while `claude` is the "idea" person and the reason company meetings now requires a meticulously scoped itinerary down to the minute. If you know how to work with both types and which type of work each is better suited for each, you can get a lot more out of them. As another comment made: gpt is depth-first, while claude is breadth-first.
Re-using agent files doesnt work. Give claude detailed task boundaries about what a feature should do (and only do) and then a hard config setup. For gpt you tell it the feature scope in noraml discussion terms without worrying about coming back to find a simple rest crud form is now a single page web-app -- even though your users instructions say no javascript -- with 50 ways to ingest a 6 field form that step over each other and looks like it works but probably has bugs waiting to bite you.
For claude: strong settings, well designed hooks, skills, linters and code checkers fully set up to your standard, and a rules & standards system that is emphasized in every single instruction and agent file. Also a standards agent or skill that is used with a critique/review run that is a requirement of the `/goal` (this can go in a hook too).
For gpt: that isnt as important or needed and will probably start to piss you off because of how literal it is. Your rule for online activity: Respect robots.txt; ask the agent to use clojure cli with deps.edn to add a library; agent replies (the dependency repository's) robots.txt says it does not allow automated access. Explain to the agent that means bots and web crawlers not resolving a dependency from the repository. So less really is more. Instructions and prompts is where verbosity helps. Tell it you want a 6 field rest crud form and it will make a 6 field rest crud form. It will also give you directories full of measurements and checks and performance for each one of those 6 form fields. Restrict it at the user instructions and task level spec. If you dont want all 50 checks for each form field, tell exactly what you don't want in its instructions. It's a fine line but the pedantic types will probably love it.
Haha very interesting observation, I think that there is a challenge to status quo.
Just my 2 cents:
I think that our first reactions to seeing these companies model releases is that of skepticism and somewhat rightfully so because these companies have done a lot of things to burn a lot of trust.
But AI is still important and so if someone is on Anthropic, they might feel as if Opus 5.5/Fable 5.2 is good and so if OAI is saying Astra 6.2 is better/comparable to them, a persons first instinct is to not trust OAI which might mean siding with Anthropic.
And if Anthropic releases a model, then there would be more discussions on pricing because somehow that is also related to bubble popping. I mean, supposing I can get good enough model at 10x the cheaper price of Anthropic from OAI for very marginal points, ofc people will comment about that as well.
And within both the threads we will find an intersection of discussion of open-weights models as well.
But maybe this is that if a model is just released and as such we are not having real use and word of mouth after real use from other people (which takes time), so our first instincts is to not trust any of the company so much and benchmarks can be cheated through and sometimes inaccurate of real world usage so we are in for a surprise when a new model drops and so we just drop back to those first reactions that I mentioned previously.
But this was a good question that you asked! and I would be curious to hear other person's observations
but mine (TLDR) is that: people distrust both OAI/ANT and other AI companies in HN. (Maybe we are technical people and we tend to look beyond the BS of marketing and so we think that we wont be impacted by that BS so we want to call out that BS but we dont know for sure if its BS or not but judging by the fact that these AI companies have incredibly low trusts, we as default don't really trust them and consider them BS and so the chain follows.)
HN audience is a tough audience to satisfy from first impressions (real world usage imo becomes a much better indicator and we do see it as people mention other models in this thread and others) ;)
I use both, I feel they are both on par with each other in a practical sense, though I do lean more towards openai at the moment, I like the tooling better. The results seem relatively on par. I find it a bit harder not to burn codex tokens for the week vs claude where I don't really even get close to running out
It really depends on how you use the models.
There is a big spectrum from pair programming with AI to I don't even read the PRs and your experience with the various models can vary greatly depending on where you fall on it.
I find in a pair programming situation a lot of them are very similar. Its when you let them off the leash and run unsupervised that you tend to start to see the differences.
It's just paid promotion I suspect.
I've heard from several users here that they're offered up to $200-500 per comment sometimes to promote a relevant product when a first page post is about a competitor product
What gives? I haven't gotten a single one of these offers.
Hope those users contacted hn@ycombinator.com and would encourage anyone receiving offers to shill to do so
HN (and most public spaces on the internet) rewards contrarians.
I think the larger a platform gets, the better negativity plays
> A lot of the comments here seem to be about other models, with little discussion of the ones actually being released.
To me that’s a sign of an authentic discussion. When I see social media full of comments about a model nobody could use until that moment, it’s obvious that labs are seeding release content.
Labs do give early access to influencers (like a lot of products these days)
Sol lucet omnibus (the sun shines for everyone)
Sol omnibus lucet, praeter eos qui memoriae (DRAM) egent.
Brillant
I've found this to also be true on their respective subreddits. They both have very "the sky is falling!!!!" vibes. Two of the truest truisms are at work:
1. If you make an app, most of the happy users say nothing. Whereas seemingly all of the unhappy users will be vocal complainers.
2. The grass is always greener on the other side of the fence.
I feel the "grass is greener" mentality is particularly silly in this case, because for an extra $20/month you can just keep a cheap subscription to the "other" one. I realize not everybody has $20/month, but I think most people who can afford <= $20 for one subscription can probably afford another $20/mo for the other one.
Personally, I complain about the tools I like, since those are the ones where the rough edges matter.
If I don’t like a tool I just find a way not to need it.
I would complain about Fable’s writing style, to illustrate, but after some tense days we have found our common ground!
The ultimate goal for most software is to never be thought about, ever.
just like a good umpire in a baseball game
Yeah, but that's also probably because it's hard to have an informed opinion right after a new model just came out.
It also is probably driven by the behavior of commenters. I'd assume that fans of respective products would rather start using the new version vs stick around the comment section.
One could think that'd mean people should be better at holding off on sharing their opinion until they have something that actually contributes to the topic, instead of just "I don't have informed opinion yet, so better write something else in the meantime"
The people that feel that way don't comment. Commenters are by definition people who decided to comment
Yeah to be honest I would love to read one of these threads a week or two after they have actually been released.
Big same. Encourage someone to write an article asking for feedback/discussing the model at the 1 or 2 week post-release point, and hope it hits the front page.
Still making my way through the entire Fable 5.2 thread and would love to switch over to an updated thread at almost the same scale.
HN admins could help facilitate this… maybe a monthly model talk thread or something.
> I find it interesting how comments on OpenAI release threads often favour Anthropic, while comments on Anthropic release threads favour OpenAI. Maybe criticism just attracts more attention.
Or maybe Anthropic does astroturfing on OpenAI release threads and OpenAI does astroturfing on Anthropic release threads?
HackerNews is being botted to death since a year or two.
The reality is that neither company does such a thing, and that it's rare to see bot comments on HN that are not already downvoted or flagged.
I'm really interested to know how you determine what is or isn't a bot comment, because I have showdead enabled, and I can guarantee you there is absolutely no difference in writing style or ideas between the dead comments and the other ones.
I turned it on for a week or so, and the dead comments were mostly obvious AI slop, spam, and ridiculously immature comments.
It is interesting how opinionated people are about which company/model is better. I think this is probably because they are scared of missing out, have invested in one set of models, and therefore want to promote those models. Basically, they're reassuring themselves.
My sense is that the OIA/Anthropi/Google frontier models are pretty similar in their capabilities.
>> My sense is that the OIA/Anthropi/Google frontier models are pretty similar in their capabilities.
How did you gain this sense? Do you use all of them interchangeably on tasks that are similar in size and complexity?
In Cursor, I used to switch to Anthropic models (mainly Opus), when I ran into a problem that I could not solve with other models. This was my way of working for many months. Since Argon was released, I used Opus twice, and one was to run a comparison.
I used Claude for a couple reasonably complicated projects a while ago and I use OpenAI at work. They seems pretty much on par. I assume Google is similar, though I have less experience with it.
I think "they're all pretty much the same" is a good working hypothesis due to the intense level of competition between these three companies. When someone says "OAI suck; Claude is good" or the opposite (as you very often see in these threads), I think that's likely rationalization at work rather than a meaningful comparison.
[dead]
No, it's not. Google hasn't been competitive at all. Allegedly their latest model closed the gap somewhat, but what I heard is that it doesn't hold up too well outside of benchmarks.
For OpenAI, they have not been in a good place for the last two months. I switched from Astra to Opus 5.5 and it's been a clear upgrade, nicer to use, and much more generous with the usage limits on top.
This has nothing to do with rationalization. Opus 5.5 is just the better model, and that's a wide consensus. In the past I used the $200 subscription for both Anthropic and OpenAI, nowadays I downgraded Codex to $20.
From the comments on this release, it appears OpenAI rushed something barely usable out of the door in an attempt to stop bleeding customers to Anthropic. This would check out with the reports back then that the 6.1 Astra release was cancelled because of poor alignment and lack of usability as a coding agent.
I would not be surprised if the issue is still present to some degree in this 6.2 release.
I exclusively chat in a chat room with the latest frontier models at the same time. It's because first of all I can afford it, second I don't want to miss a single nice idea.
For the most part the majority say the same thing but with different wording.
Only model that I have found that has, what I describe, "humanly creative takes" has been opus 5.5 so far.
But that was until gemini carbon came along, and I've decided to bench all the other frontier models for the time being, well except Astra 6.2, I've just added it in the room.
Hmmmm. I've noticed that they largely converge on the same solutions. Perhaps 80% overlap, if I was pressed for a rough estimate.
The 80% overlap is expected since I'm not doing particularly deep or novel work. But the 20% divergence is significant -- OpenAI's models routinely spot bugs that Anthropic's don't spot, and vice-versa.
I exclusively chat in a chat room with the latest frontier models at the same time.
That's interesting. In your setup, since they're all in the same room, is it possible that groupthink emerges since they're all reading each others' output? Or do you have things set up to prevent that?In my scenario, it's relatively low-tech... I'm just cut-and-pasting between independent Claude and Codex sessions.
(This could easily be automated, and I have written skills to do so, but I actually prefer the manual approach for now because it helps me to comprehend things)
carbon is powerful but also needs strong monitoring. especially in specialist domains (i was evaluating its tacit chops with some asic flows, and doing some (standard, but rarely mechanized) cryptanalysis and it will cheat and lie and get away with ultimately crippling systematic oversights in its work. it is extremely perceptive of the implications of ideas and sometimes doesn’t notice they haven’t actually actualized in the environment yet, are just latent from the context.
So I run all tasks through four models locally: Astra X-High with Codex, Fable 5.2 Extra with Claude Code, Carbon High with Antigravity, and Grok 4.7 High Fast with Cursor.
I would not put Google in the same league as OpenAI and Anthropic. It is like 3 times slower at completing tasks. It also has the highest failure rate out of any of the models. Like sometimes it can do what Astra and Fable can do but sometimes it just misses completely.
Nah, GPT6 models have the highest failure rate - they simply stop, repeatedly and consistently.
And obviously you don't run all the tasks through these models *locally*
Ive ran GPT-6 and 6.1 models repeatedly through multiple-hour sessions, they dont stop that often.
I'm honestly curious what you're doing because I've left Astra running for days on end without it stopping. This has never been something that I've encountered.
And yes - they all run locally on similar machines and with the exact same setup in terms of files, installed tools, etc.
You either work in a data center or you don't run the models locally. Perhaps you mean the remotely run models are executing code locally, but this is a big difference.
Google ain’t in the same league right now.
Carbon still beats Astra 6.2 and Fable 5.2 in benchmarks and neck and neck with others
That's an opinion, but not a fact.
No, at least for coding or UI design as a use case, its fact.
Gemini 4 Carbon is the only model I used that fixed a super subtle memory corruption bug in a JIT compiler I'm working on, hiding as an instructions flag bug.
Astra 6.1 and Opus 5.5 spent days on it without fixing it. Gemini 4 Carbon fixed it in about 3 hours. Only Gemini Carbon decided after getting the same result over and over that maybe the bug it assumed it was, was actually something different and tried a different idea on its own initiative.
Maybe Fable might have got it, but probably not before I ran out of credits.
Gemini 4 is _really_ good at fixing bugs. Whether the OpenAI 6.2 models are that good remains to be seen.
Dead on arrival, judging by the FrontierBench scores. Astra 6.2 is five percentage points below Fable 5.2 xhigh.
Whatever you’re doing, it’s unlikely to be what most users are doing. Fable and Astra aren’t in my price range so I’m paying attention to Sol.
Astra has always been a step below Fable so I'm not surprised.
The point is that it's a fraction of the cost.
I think "second place but also cheaper" isn't what OpenAI needs right now.
Given the increasing cost of infrastructure and decreasing availability of cheap money to build it, I think efficiency actually is pretty important. Second place at a sixth the price and in a quarter of the time (both according to Artificial Analysis for Astra 6.2 vs. Fable 5.2, both max effort) definitely has a place.
"Close second, and also cheaper" is however a pretty solid framing when your competitor is trying to IPO
That remains to be seen. I think if OpenAI's goal was to torpedo Anthropic's IPO, they should have released a model that is significantly more intelligent and capable. I don't think investors will be impressed by this release, which I think is simply OpenAI trying to stay in the race.
I don't think torpedo'ing the IPO is actually in their best interests? OpenAI also wants to IPO at some point, and if Anthropic fails spectacularly, investor confidence in a second big IPO in the AI space is going to be shaky
Performance is insane for the cost if those benchmarks hold up. Looks like they did time this for right before Anthropic's IPO. Curious what Anthropic releases in response, and how these do on real world tasks.
Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?
Edit: Wow, interesting very negative vibe check here: https://x.com/mattshumer_/status/2122752192226406421?a=fVRt3...
Well, Fable also went down in cost too in their last update to Fable, and honestly, if they released a crazy new model on IPO day it wouldn't surprise me. Save the best for an impressive IPO.
Matt Shumer says things to get attention, and almost no other reason.
Anybody remember https://huggingface.co/mattshumer/Reflection-Llama-3.1-70B
"Oh, the quality is bad because we uploaded the wrong weights."
How much of that cost savings is just Code Mode?
> Looks like they did time this for right before Anthropic's IPO.
While both Anthropic and OpenAI have been playing games with model announcement timings, this time I don't think it's related.
These are incremental bumps, not big new models that have a huge impact.
An Anthropic shareholder is very negative about an Anthropic-beating model release, right before the Anthropic IPO? Interesting, yes :)
> Feels like we're way past the pelican benchmark being useful at this point.
It was neither a benchmark nor a good one at that. Just a blogger's curiosity that grew out of control.
gpt-6.1-terra is already beyond end game for my client's business. They don't require anything more powerful.
Cost is the most important factor now.
They're going after anthropic's market through cheaper intelligence per token.
I wouldn't be surprised if they released 6.2 luna with the most attractive positioning on the pareto frontier right as the IPO happens.
>Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?
Generate an SVG with complete and detailed graphical instructions on how to fold an origami pelican riding a bicycle
(or any other complex origami object). The fact that it has to teach [you] something makes it pretty interesting.
Beware of the paid influencers who don't disclose their affiliations in this space. Theo is another one who now talks smack about anything that isn't Claude. They bend reality to fit a narrative.
None of these benchmarks/marketing numbers from openai or anthropic have ever held up.
Now that we know anthropic nerfs models after release, and without any notice/openess, there is no reason to even think these benchmarks they release are even done with the same model/intelligence that actual customers would get.
that negative vibe check is from the Reflection guy who in 2024 wrapped Claude API and presented it as some new breakthrough open weight model of his own.
You're looking for a vibe check?
Thoroughbench Fireship’s horse animation test is actually a pretty nice vibe check.
I'm just running it on my Blood Bowl Bench, holding it's own against the reigning champ Argon so far but early days...
You should be aware that the individual you shared had a project called "Reflection 70B" that was, to put it mildly, quite disappointing and misleading.
> Looks like they did time this for right before Anthropic's IPO.
Would it be in OAI's interest to kick the legs from under Anthropic? Granted I don't think they can, but a shot across the bow can sometimes be more damaging that one expects (I think FTX collapsed due to one? Or maybe I'm misremembering).
I always assumed that if one of the big AI IPOs fail hard enough the neoclouds start to go bust, which causes a chain reaction that probably causes both ANT and OAI to collapse.
The IPO is 6 times oversubscribed and I bought some (we will see how many i get) but its a gamble right?
Either people really just want to buy and hold or they see it as a gamble chance like with SpaceX Superintelligence LLC.
I guess it's a good idea for them to wreck the appeal for their own employees to leave for the competitor.
It's too obvious. Anthropic still has time to respond.
https://nitter.netbub.com/mattshumer_/status/212276041577177...
> I have no idea why they released this. It needed a lot more iteration.
Maybe the goal is to make Anthropic's models look good by comparison. :-)
My reaction as well - at the very least now is probably not the best time to knock down the realized valuation of a peer when they are effectively setting the market value of the whole category of ai companies, including yours… right?
I've been having fun with having it build interactive Three.js demos of devices, with setup screen, a few different operational modes, mechanical movements etc
Can we see a demo? There's a threejs subreddit that is absolutely overflowing with impressive LLM generated demos, in case you haven't seen it:
I'm very curious about this as well.
Recursive self-improvement (RSI) has been a very hot topic recently, and it could be the biggest factor that decide Anthropic's IPO.
I wonder if the general public can infer whether RSI has started based on the recent performance gain of the frontier model.
Yeah we see some Really Stupide Intervention from those who drank the Kool aid. IAS2[] start to be a real thing.
[] IA Stockholm Syndrome.
RSI just means using the LLM as part of the development process, which they are already doing. I doubt any of these companies are going to fire all their developers, so there is no point development is going to be 100% automated - for the foreseeable future it will always be a mix of humans using LLMs.
So, you can say they are doing "RSI" (aka using tools to automate development) already, and will continue to do so.
Hm, I'm pretty sure the labs are using a very specific definition of "closed-loop" RSI in which one generation of models develops the next generation of models entirely autonomously. See: https://www.anthropic.com/institute/recursive-self-improveme...
Why IPO if you have RSI and possible ASI?
Business as usual until proven otherwise. Everyone knows how to do R&D and IPO and VC, no one knows what RSI and possible ASI would do.
If you really mean it though and think about top x things to do first, i would probably try to research, invent and patent some superconductor and medicin.
ASI might not mean it can solve everything immmediadly but it could in x years with x amount of compute.
With an IPO and more capital, you reduce your risks
So you can scale up. RSI is going to be compute-bound. ML experiments cost compute. Also, if you can make a genius in a data center, why not make a hundred/a million geniuses?
So investors can cash out on the time frame they promised investors.
Sincerely curious: why do people feel led to use the word "insane" to describe every new model release? I just don't understand what about incremental improvements needs to elict such a strong response.
We knew that improvements would continue to roll out increimentally, the costs would go down, we watched Moores law for decades before this?
Does it feel 'incremental' that the trend-lines suggest we're going to automate all of white collar work in <1 year from now and that every model generation gets meaningfully closer to that? https://x.com/maccaw/status/2120962037018771475?a=AqStEJ42mP...
Not totally sure what your point is. Is it incremental that OpenAI dropped the 722 math proofs or solved Navier Stokes or disproved ETH etc? Yes, in a sense it is and the trend-lines predicted it for ages (albeit it seems to be quite a bit earlier than the trend-lines predicted), but it's also a mind-blowing result.
When all humanity gets put out of work because ASI is superior to us in every cognitive domain in a year or two, yes, it will have just been incremental progress, but it also seems like somewhat of a relatively big deal, no?
That's nice and all, but obviously progress has been unimpressive compared to Opus 5.5 that released two months ago.
I expected significantly bigger improvements from Fable after Opus 5.5, this did not manifest.
And OpenAI is apparently in a bad and desperate place.
“When all humanity gets put out of work because ASI is superior to us in every cognitive domain in a year or two”
That would seem like a big deal because it’s not going to happen.
Feel free to return to my comment in 2 years time.
Shumer is a known scammer (Reflection 70B, among other things). Why does he have any credibility?
Multiple top reviewers now calling it a flop. All not Anthropic shills that have said very positive things in the past about OpenAI and their models: https://x.com/elie2222/status/2122771287936233772?a=hFIDuJmZ...
Not to mention he didn't actually provide any reason why the models are bad. When pressed in the comments and asked what was bad, he said "everything"...
This tweet provides more nuance: https://x.com/emollick/status/2122755693191287009?a=hzSJAw9H...
I saw another review from an early tester saying it's the worst model ever to work with.
The review claimed that the writing in the output is incomprehensible.
It also said that with /goal, the model set the goal to blocked to ask for the filename reporter_3423 should write to.
Lastly the reviewer said it made no progress on a large project.
Multiple top reviewers now calling it a flop. All not Anthropic shills that have said very positive things in the past about OpenAI and their models: https://x.com/elie2222/status/2122771287936233772?a=hFIDuJmZ...
Anybody who says stuff like "They were essentially unusable." without any context or proof is going to my permaban on credibility. Even 1 year old models are perfectly usable let alone frontier ones.
That's what I thought. "Interesting negative vibe check". I open X, it's someone just saying they're unusable. Why is this top comment?
Using a top comment to edit in a low information tweet on a big release should be treated as bad faith. Comment should be deranked.
Update: Multiple top reviewers now calling it a flop. All not Anthropic shills that have said very positive things in the past about OpenAI and their models: https://x.com/elie2222/status/2122771287936233772?a=hFIDuJmZ...
I would add more but the edit timer is up :)
Here's a more balanced tweet from another early reviewer noting it just seems to be very uneven: https://x.com/emollick/status/2122755693191287009?a=hzSJAw9H...
If you have the quota or budget for it, I recommend vibe-coding your own personal benchmark. I have a suite of 8 projects that I made with models this year, and replay them - give the new model a checkout of the repo at that point, and the project spec. For the design task, I have it throw up a blind voting page for me to quickly judge, and for implementation I have a review agent do a general score and also hunt for problems/edge cases that I really found.
It's a lot more imprecise than the public benchmarks, but it's tailored to my own work and definitely isn't being benchmaxxed by the model developers. I was able to tell really quickly that Opus 5.5 delivered on the hype, and that some other releases (a couple of the more recent open releases, and pre-Carbon gemini models) were not as good for me as public benchmarks made them look.
(no results for either 6.2 yet, will take another ~6-8 hours, but I encourage people reading this with spare tokens to just make their own like I did instead of listen to me)
Your personal voting system sounds a lot like what we're building here. https://github.com/bottomless/agent-duel. We even have the LLM judge that helps parse out any differences. Related to this discussion, my personal Elo board based on my own votes has Sol 6.1 dead last:
Claude Opus 5.5 1160 (1027–1334) GPT-6 Astra 1114 (932–1330) Muse Wonder 1063 (918–1220) MiMo-V2.6 Flash 1053 (925–1187) Kimi K3.1 Pro 1037 (870–1216) Gemini Carbon 1021 (762–1281) Claude Fable 5.2 975 (682–1239) DeepSeek V4.1 Flash 952 (799–1095) Grok 4.7 927 (782–1058) GPT-6.1 Sol 910 (677–1096)
This is great, and I have been setting up infra to do this very thing. How many sessions to you replay typically to get a reasonably confident estimate? Assuming each project in your suite was built over multiple sessions.
This is much more interesting IMO than any public benchmark. They are definitely targeting those benchmarks and model capability does seem to have plateaued somewhat.
please update !
I have built my own benchmarks and it's very valuable to have them but I'm skeptical you can get good results vibe coding them.
Getting the framework right isn't that hard, but requires some careful thought. You probably don't want to just checkout a repo at a certain commit hash - agents will explore other commits and branches. (Public repos are also risky in general: agents with internet access can find later commits, and public repos could be in the training data anyway.) I use `git archive` and give the runner just the files at that commit and nothing else. I also do the run inside an isolated container that is recreated each run so it can't find e.g. related files left in /tmp from a previous run.
Getting the benchmark tasks right is harder. I don't think there's any way to get around writing the task and its evaluation by hand. I really tried to get a few different frontier LLMs to do it and they're all terrible at "imagining" what an agent would need to solve a problem without giving away the answer or providing too little information. Also quite bad at writing evaluation criteria that tests whether the solution matches the assignment (versus just how close it is to a golden solution, or how close it is to how they would solve it)
I personally don't like or trust using LLM as a grader either, but I guess people manage to get that to work well enough.
> Getting the benchmark tasks right is harder. I don't think there's any way to get around writing the task and its evaluation by hand.
This is a great tip and explains the main issue I had with my attempt at vibing a benchmark. I set up an arena so agents can make screensavers, then I judge the screensavers blind. Worked as an idea, but the prompt tuning was bad, agents would just build slight variations of the given example ideas. I'll have to try giving my prompt the human touch.
What type of tasks did you use for this? Evaluation?
Last time I did this I found it quite hard to come up with tasks that have clear evaluation outcomes
Have it compare to the real work you've done in a sample of your previous sessions. Agents are good at simulating completing the work following newer harnesses, heuristics etc. and comparing time, tokens, quality, pitfalls.
My vibe bench is no dependency HTML bowls of ramen https://bitbin.de/ramen-bench/
Opus 5.5 Max is my favorite, but some of the Sol/Astra 6.2 ones are pretty good.
Edit: some folks reported crashes. It was probably Gemini 4 Carbon @ High, I've put it behind a warning. If you see another one let me know (OS, browser, which bowl, etc)
Poking around models crashed the tab in Firefox on windows.
Wild. The opus ones literally look photo realistic... (Want to declare I am NOT a payed anthropic spammer, but look at those for yourself)
Also confirms grok is absolute garbage
Yep, seeing Opus results convinced me that maybe Anthropic is right on not neeeding an image gen model. It was very slow though.
just a fyi this crashed my browser
Which browser / version / os? And was it any particular bowl that crashed it?
It works fine for me on Chrome, Safari and Firefox. However, some models have produced crazy resource intensive ramens before. I audit them now before pushing to make sure they're not too resource intensive
Seconded.
data point: no trouble seeing it on Chrome on MacOS (whatever the last one before the liquid glass update) on M5.
Doesn't crash my browser but Gemini 4 Carbon High freezes up the page. The others run fine.
I thought it was interesting that all the Sonnet 5.5 have a weird light effect on the bowl. XHigh has a nice film grain too.
TY I'll put it behind a warning
I didn't find Sol 6.1 nearly as impressive in real SWE use compared to what the benchmarks implied. It wouldn't surprise me if the models have internalised some kind of 'benchmarking mode' which only kicks in when some attributes that are peculiar to how benchmark problems are presented.
Been using only Sol 6.1 on high/extra daily at my SWE job since it came out (with its official harness). It can basically do everything I need it to do. I still review its code and generally I have to leave around 10/20 comments in a ~700 line PR, flagging things it missed or weird decision. One or two iterations are enough to fix everything. Its still way faster than doing it myself, unfortunately.
You make it sound as though the models are like not working to their full potential instead of being a little bit overfit on the benchmarks.
I am not even certain 6.1 sol is better than 5.6 sol after using both for some time now, so not expecting much here.. but then again, 6.0 sol was completely useless, so maybe the .1 bump means something here.
I had the same impression. None of the GPT-6 models were as great as they claimed, they never surpassed Fable or Opus 5.5 on my use-cases.
Dieselgate --> Tokengate (But run by the AIs)
We know anthropic nerfs models all the time, openai probably does the same. There is no reason to believe the benchmarks/numbers they release are even measured with the same effective model that customers would get.
Yeah, MaxxBench was shared here a while ago and showed most models think about how a grader will evaluate their work on DeepSWE in 95%+ of traces. Strong models definitely realize they're being graded and start to game their output.
"My AI is lazy" is honestly not a problem I thought I would have two years ago.
Sol-6.1 made me cancel my OpenAI subscription. Sol-5.X was bad, then 6 was barely an upgrade. Then 6.1 came out and it didn't move the bar. Went back to Anthropic.
General, SWE, python development, rust development, tool use. It is 'perfectly functional' and if you don't care about the model following your instructions or code quality it is 'fine'.
I wonder if these companies do selective scamming. For me Fable, Opus are unusable. They are completely dumb. I consider downgrading or cancelling subscription. Whereas Sol 6.1 got me my jaw dropped. Reminded me of Opus 4.6 before it got nerfed. It's so good I completely cancelled my Cursor subscription and uninstalled it.
Hmmm, your comment is so odd as it is exactly the opposite of my use and the many videos I have watched comparing opus 5.5 (essentially GOAT until a few days ago) to other models.
Have a look at the videos comparing opus 5.5 to others.
https://www.youtube.com/@ArenaAIOfficial
Astra and sol are quite behind, hence I question your motives, unless that's truly your perception, which may be genuine, but honestly, odd and against the grain of many, many others' experiences.
> I wonder if these companies do selective scamming.
I am 100% sure they do. My internal evals of Opus-5.5 dropped a few weeks ago, then recovered to launch levels once I got added to the security program.
I review LLMS, and 6.1 sol is fantastic.
What kind of stuff were you throwing at it and where was it letting you down?
So what have __I__ been doing with it, then?
I’ve got 6.1-sol to take an Amiga driving game binary and add networked car driving into a web enabled Amiga emulator… that’s interpreting machine code, Amiga architecture and then making meaningful changes.
So yeah, 6.1-sol is good.
My experience with personal / greenfield projects has been good.
Where it falls down is on the boring stuff - writing tight PRs for features / changes in large production codebases (data area in my case). Without a lot of handholding, tendency is towards 'bolting on' new code resulting in overcomplexity & missing obvious opprtunities for simplificatation.
Thanks for sharing. I do extensive direct comparisons, and use both Anthropic and OpenAI tools in anger on large professional codebases.
I think you are correct, OpenAI tools need more handholding.
However, I think that is actually intentional from OpenAI... they stick to what you the POH ask them and deliver what you ask. They embellish less. Anthropic tools by contrast embellish a lot, and they're very good too.
Perhaps start explicitly asking the OpenAI tools to embellish or think outside the box. That may unlock that experience and results you prefer. If you already did this, would you mind sharing more detail on what effect that had (or not) for your workload?
Yeah, in real SWE tasks, 6.1 is bad.
We've been trying to capture real SWE data by building a battle system on top of OpenCode. In our data, Sol 6.1 is seriously underperforming, I'm talking worse than DeepSeek 4.1 Flash, at 10x the cost. (Though to be fair, statistically they are tied in performance, though not at all in cost. https://agentduel.com/leaderboard .)
(It's possible for a tiny model to do better than a large one because contrary to typical Arena methodology we think a fast good enough answer is better than a slow one.)
OpenAI performance on our system has been bad enough I switched to Anthropic models for my daily work.
I'm guessing you don't use codex though- the harness + model combo makes a difference.
There was a post about that today on HN but it looks largely to be bogus. I wrote a blog post about it, but not linking here to avoid too much self promotion in this thread. Funny enough my conclusion was that OpenAI models seem to be the only ones that actually do have an own-harness interaction, hah.
I've been building quite a complex desktop app (a digital audio workstation) using 6.1 Sol and it's been very good for me
Yeah my main models are OpenAIs... I am trying Opus right now because I dont have any credit anymore but from experience, it was never as thorough as openai's models. It can be genius and unblock at times, but I've never been able to trust it.
Yeah, our data only has 28 battles tagged "desktop":
Web: 87 (12.2%) Systems and protocols: 78 (10.9%) Software, unspecified type: 74 (10.4%) Backend: 68 (9.5%) Writing and documentation: 55 (7.7%) Research and advice: 47 (6.6%) Unknown: 42 (5.9%) Mobile: 40 (5.6%) Operations and automation: 36 (5.0%) Developer tooling: 35 (4.9%) Desktop: 28 (3.9%) Media and design: 27 (3.8%) Data analysis: 25 (3.5%) Device and system support: 22 (3.1%) Math and formal methods: 18 (2.5%) Games and simulations: 15 (2.1%) Infrastructure and DevOps: 9 (1.3%) AI and machine learning: 6 (0.8%) Other: 1 (0.1%)
Is this an ad?
Welcome to HN.
BTW check out my totally real project that helps you detect native advertising on your favorite platforms:
I get 404, a lost sale!
No. I'm personally building agentduel.com, specifically with this problem in mind (benchmark performance not matching real world usage). It's generally accepted by HN community to make relevant comments about your own projects, as long as its not hamfisted.
as long as its not hamfisted
You have six different comments on this story that all basically link to your product. That's a bit hamfisted.
Isn't this what arena.ai does?
Yeah, maybe one thread per article will do?, otherwise it comes across as shilling, you've mentioned your project four times now …
Fair enough, though all the comments have either 1pt or positive karma, so I take that as a signal that they're adding value so far. I'll take a coffee break, though, hah.
Genuinely, just excited about what we're building and think it adds information about these models that are missing.
Using non-native harnesses sort of throws a wrench in things. I pretty sure your benchmark would look much different if outputs came from their intended harness.
Interesting. Yeah we picked a neutral harness to keep things fair, but you're right that a native harness might be better. Would be a beast of an engineering challenge though.
It certainly does get tiresome though.
Certainly does. See my comment here roasting this extremely BS blog post that is basically an ad: https://news.ycombinator.com/item?id=50532191.
Do my comments get tiresome, though? None have been downvoted to hell, since they are all adding to discussion. If you think so feel free to email me, I don't want to be annoying.
We've recently moved away from Anthropic models at $PAID_JOB and over towards GPT models. I'm trying to be open-minded but the GPT models we have (5.6 Terra and Sol) are just not that great. They're fine, but even simple things such as "check my work on this small Bash function" get it to point out stuff that's already implemented in my work. I don't know if it's me/how I use it/whatever, but just some N=1 evidence, I guess.
I have the exact opposite experience.
Well, yes, if you switch from current claude model to something that was released in July, of course it will be worse...
Bro you have to be at 6+. 5.6 is EARLY JULY
5.6?? How is your impression of 5.6 relevant to anything recently?
5.6 series is worse and more expensive than the 6 series. Try the latest Luna and sol; I use those for 100% of my work right now, with sol doing only 10-5 percent of it. I code in go and review the work and edit it extensively though, so I don’t know about how good those two are at vibing slop.
5.6 was better than 6. 6.1 is a bit better than 5.6 from my tests
why do you guys have to use 5.6?
IYKYK. But in short because of a lack of alignment with product strategy
I believe that's all you get in copilot (365) basic.
The BOFH "IT admin" type of guy, an eternal scourge on the industry, has moved from delaying approvals of software you need to delaying approvals of updated models. It's management job to make sure these people don't burrow themselves into gatekeeping roles at an organization's expense
Large corporate governance. Views on security, safety, locality, deployment rules, cost control, change management, enterprise architecture gollums etc. "Bill my cost centre" isnt the silver bullet it should be because of all the other stuff.
DOD limitations
5.6 is in no way representative, or comparable to 6.1. In fact, the 6.1 models are so good that I've just used Terra 6.1 xhigh for most tasks.
sol 6.1 is meh
GPT models in my anecdotal experience are worse at drifting into the weeds on long running tasks, and continue to take what you ask way too literally. I have come to the conclusion that Anthropic’s models are just better at teasing out and intuiting what you actually mean even if you under specify it.
They are much better compute for dollar though. I have hilariously started to just have an opus thread maintain Sol subagents to get the best of both worlds
That’s exactly what I do. It’s more complex than this, but vaguely: Opus medium as orchestrator and reviewer, Sol(high/medium)/Terra(max) as implementer. My set up looks at my next GPT reset date and dynamically enters FAST mode if I’m under achieving on projected usage.
Ive also had Opus coordinating multiple models, including those from Open AI, Deepseek, and Kimi. The cost of Opus is still my downfall, but the current setup is still far, far cheaper than doing everything in a frontier model, and more efficient than running cheaper models since the coordinator will take control when cheaper models repeatedly fail a task.
Similar experience here. The spirals on long running tasks are out of control at times with hours going by without a conclusion.
They (OpenAI models) are very tuned to be depth first and you can coach them all you want to try to be breadth first but as that context window fills up on those long running tasks they drift back to their first principles, which means obsessively testing and verifying a singular task rather than proceeding to the next task on the list
If you're using a sufficiently flexible harness (perhaps your own) you can make conversations as branchy as you want.
This explains why I like GPT: I LIKE for them to excessively test each step, kthx!
I wonder if that depth-first/rabbit-holing tendency is why GPT models tend to be better at mathematics (my own experience) than code.
I’m not sure. It’s very easy to get lost deep in a wrong branch when you’re trying to solve a math problem. You have to intuit when to give up and backtrack, lest you end up going in circles.
I think they are using swarms of 1000 models for these math breakthrough. Having the instances go deep in different areas would make sense.