Rendered at 06:09:51 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
HeavyStorm 4 hours ago [-]
Excellent idea, most terrible execution. Comparison are completely subjective, problem space is too simplistic for today's AI, the resultstable simply ignores that a face isn't a cube (therefore, gpt 5 shouldn't have 100% success) and the retry is uneven. Also, given the random nature of AI, sampling once each model isn't very scientific.
This feels like a kid trying to do science. The will is there, but lacks experience.
sinuhe69 3 hours ago [-]
It said “3D-looking Rubrik cube”. Maybe your cube looks different but I’m pretty sure for everyone else, the GPT result doesn’t look like a 3D-looking Rubrik cube.
hyperhello 4 hours ago [-]
It’s still science if you say “this appears to be a specimen of X” even if you don’t do a genetic test. Things don’t automatically graduate to science either by repetition or by formal verification. What makes a rubics cube is obvious enough that you can pass or fail.
looksjjhg 3 hours ago [-]
Show us how it’s done then, talk is cheap
reenorap 9 minutes ago [-]
How can grok create a coding LLM at the same level as OpenAI or Anthropic when they don’t have the same amount of AI talent as the other companies by an order of magnitude? Is it really that easy to train a coding model like that?
inemesitaffia 9 minutes ago [-]
Cursor
wwind123 2 hours ago [-]
Half year ago I tried to use Codex, Claude and Gemini build the same scripts to automate various things on my machine. Claude was the clear winner back then, making the most reasonable assumptions, presenting results in the easiest-to-read format, writing runnable script with minimum dependency. Half year later I think Codex and Claude models have both advanced a lot, but Gemini is still lackluster. Gemini could catch problems when reviewing Claude/Codex's design plans and code, but it's hard to make Gemini make complex plans or implement complex code by itself.
sgk284 2 hours ago [-]
If you like this kind of comparison, we have an arena of 52 apps one-shotted across 21 models here: https://arena.logic.inc/
I keep it pretty up to date (tomorrow Grok 4.5 and Sonnet 5 should be pushed).
theturtletalks 1 hours ago [-]
Really nice site! From your experience, what’s your go-to model for nice storefronts?
kristopolous 2 hours ago [-]
I want this with smaller models as well like Gemma 4 or Qwen 3.6
sgk284 1 hours ago [-]
Awesome - will work on getting those in.
dadoum 5 hours ago [-]
I tried to one-shot the first test (the Rubik's Cube test) with LucidQuery's Swift model, to test it, as there are not much benchmarks about it and that they brag a lot about it, and I was pleasantly surprised to see it achieving a result similar to Grok 4.5 but in one shot (there is the same issue that if you scramble twice the solve button does not work anymore, but it got it in one shot).
Though it crunched most of the free quota, 47111 tokens, so I couldn't make multiple attempts.
jeffgreco 5 hours ago [-]
So strange to write a whole post with Claude giving the best results and Grok consistently the worst, but awarding Grok the winner because at least it did the worst fastest?
singingtoday 5 hours ago [-]
GPT was the worst on the Rubik's cube
trollbridge 2 minutes ago [-]
GPT-5.5 isn't really a fair comparison to Fable or Opus. GPT-5.5-Pro would be a better comparison (and yes, I know how much more expensive it is).
I really wish they'd thrown in something like GLM-5.2 into the comparison.
cuvinny 3 hours ago [-]
They gave grok 2nd try on that one. Now one shotting a webpage is a dumb metric but if that is what you are testing it did the worse.
GaggiX 5 hours ago [-]
Grok did not render anything, they had to prompt it again.
mlmonkey 5 hours ago [-]
I am 99% sure the post was written by AI
ks2048 4 hours ago [-]
Blog post idea:
We made Grok 4.5, GPT-5.5, and Claude write a blog post about using Grok 4.5, GPT-5.5, and Claude to build the same apps.
jszymborski 5 hours ago [-]
The honest takeaway: this is 100% written by an LLM.
trollbridge 47 seconds ago [-]
Pretty much any harness with a halfway decent model should be capable of running the experiment including making the screenshots, summarising the results, and so forth.
I sort of do this whenever an interesting new model comes out.
mlmonkey 5 hours ago [-]
That was the honest giveaway ... :-D
Crisco 4 hours ago [-]
I did a quick skim and the usage of phrases like "snappy stylist" and "speed-and-value monster" were what instantly stuck out to me as AI. I decided I probably didn't need to actually read the article after that.
walrus01 4 hours ago [-]
This is the real unlock of the speed and value monster.
I am trying to figure out how many LLM converged on a writing style that resembles a LinkedIn MBA true believer. Maybe because there was just such a sheer mass of corporate-speak drone writing out there in the wild in the training data set?
But more seriously, is there a firefox extension that 'skims' the text body content of a page and puts some kind of "this was probably written by AI" meter, gauge, number or indicator in the top menu bar adjacent to the URL bar? It could even be color coded in various shades from green, yellow, orange, red. If there isn't, it sure seems like something that would be good to have.
hattmall 3 hours ago [-]
It's kinda logical. Most people, individually, have a somewhat unique writing style. So if there's one set of writing that's very formulaic and consistent and you build an averaging machine it's going to converge on that formulaic style because everyone else's writing style is going to be much closer to n=1.
red75prime 1 hours ago [-]
Kenyans who provided the data for RLHF liked that style. That's the most of it. And a transformer is not an "averaging machine", it's a prediction machine.
somenameforme 2 hours ago [-]
I'd expect that most corporate speak made it past their data curation, while normal people speaking normally was probably scrutinized a bit more heavily. The PC stuff is also probably quite terrified of a lot of adjectives. For instance I just used normal as an adjective, and that can be a hyper-loaded term if somebody's obsessed with trying to interpret things in the most absurdly bad faith way imaginable. By contrast corporate speak tends to have obnoxious lingo, but lingo that can't really be spun too much. Even for one of those guys, mental gymnasticing 'snappy' into being a secret dog whistle's going to be pretty hard.
userbinator 2 hours ago [-]
Either that or a human that has started writing like an LLM, having been "trained" on LLM output itself by sufficient exposure.
kristopolous 2 hours ago [-]
I'll actually put in spelling errors and grammar mistakes intentionally these days to show human
dwa3592 5 hours ago [-]
i will give you the remaining 1% because i felt the same way.
walrus01 4 hours ago [-]
But just think of the amazing advancements over the past 12-18 months in fully automated slop-posting! We've reached a new high water mark.
dirteater_ 2 hours ago [-]
> The receipts: speed and cost
I don't get why cost per reply is at all relevant here?
Why do so few who attempt comparisons actually compare dollars per task.
NewJazz 2 hours ago [-]
I think these were all one shots, so it was 1 reply per task?
baxtr 2 hours ago [-]
Tokens per task might be the better choice
unanonymousanon 2 hours ago [-]
Could you elaborate on why tokens over dollars as a metric?
Computer0 2 hours ago [-]
Why do I care how many tokens were used? 50,000 or 5,000,000 all I care about is the output quality, speed, and the cost.
millsau 19 minutes ago [-]
Claude seems to have the best text generation out of all of them.
steve-atx-7600 3 hours ago [-]
I have not used grok 4.5 yet, but the other pictures match my experience doing anything graphical with the other models that it cracks me up. gpt 5.5 has no design sense whatsoever. It cannot even make terminal output not look terrible. I've asked it to use colors and formatting in various ways and got goofy randomly colored output. opus 4.7 and later seemed to have an inuitive design sense by comparison - 2d or 3d. Fabel 5 is just rock solid.
Yes, subjective. But it matches my repeated experiences with these models for what it is worth.
sehw 20 minutes ago [-]
>look guys, we burned money! Upvote pls.
JoyfulPanda 1 hours ago [-]
Grok failed the Rubik's cube. I pressed Scramble twice and then solve and it didn't solve the cube. Opus did.
owenbrown 2 hours ago [-]
I’m spending a significant portion of my day waiting for agents to execute.
What’s more interesting to me than time-to-first token or latency is the time it takes for the agent to execute, from starts to finish, excluding when it’s waiting on a human.
singingtoday 5 hours ago [-]
Love the idea, I think more complex games would show the gap in ability better.
Do it again but this time get them to make a multiplayer online Jetmen REVIVAL game. Online play is key, because it's very complex. Jetmen is a good game for this since it has physics and customization that's complex enough but still simple.
paxys 5 hours ago [-]
Why not wait one more day for GPT-5.6?
foxfired 5 hours ago [-]
If we wait for the next models, we will never test anything because there will always be another model. Like the Ai Scotsman:
> "Nay, laddie, that’s no’ the real AI Scotsman! He’s grander still! More powerful! Just wait for the next model!"
looksjjhg 2 hours ago [-]
I think he’s being sarcastic
faitswulff 5 hours ago [-]
Also throw in GLM 5.2 for good measure
m4rkuskk 5 hours ago [-]
That will be in the Part 2 article.
acters 5 hours ago [-]
I worry that GPT 5.6 will be heavily restricted and have the same feature to fallback to another model like Claude fable 5 does all too often. That fallback shenanigans mess up actual benchmarks and I don't like it.
fluidcruft 5 hours ago [-]
And why not Sonnet?
Kuyawa 5 hours ago [-]
I'd like to see the comparisons with DeepSeek, Qwen, Mimo, Kimi and GLM
Kuyawa 4 hours ago [-]
I just did the tests, Mimo and GLM delivered working cubes but GLM was the only visually perfect with smooth movements and great effects.
Isn’t the number of turns most important? Some agents take repeated input, while others can mostly one-shot what I’m looking for.
brainless 2 hours ago [-]
I get the point of this demo but if instructions are clear, tech stack related resources are available, then the models do not differ as much.
I use different models all the time. And mostly lower cost ones. I do not know how people write software these days, but I have clean instructions, usually in Epics and they have Tasks.
I have been using DeepSeek V4 Flash for much of my coding in https://github.com/brainless/akar for example. Planning is mostly done by Qwen latest (in opencode) or Sonnet.
For my commercial, client work I use Claude but barely use Opus. Sonnet does most of the work. For a recent project, I went through a 35 page PRD in about 4 weeks, that includes client calls, changes, Ecpi/Task generation, a massive test suite, deployment.
kelvinjps10 4 hours ago [-]
Chat got ones were slow on Firefox mobile
thorum 5 hours ago [-]
Interesting that all four models converge on such similar designs, for such short prompts.
HeavyStorm 4 hours ago [-]
They were trained pretty much on the same data.
krauq 5 hours ago [-]
Too nice to Grok, if there are really cost savings it should say how much each of the three demos cost so we can judge if it's worth the lower quality (probably not). The time to complete each would also be interesting.
RickS 5 hours ago [-]
Barring the retry thing, n=1 on all models? Am I misreading, or is this a joke?
Variance in quality on these things is so, so high.
Madmallard 2 hours ago [-]
Could probably do this with a much older model given that it's something that probably has thousands of github repositories for source code to do so.
RazorBucksICO 3 hours ago [-]
“The honest headline:”
Written by Claude. Ugh. If it’s worth publishing, it’s worth proofreading, folks.
Yiin 3 hours ago [-]
especially when using /humanizer is one prompt away
nrightnour 4 hours ago [-]
This is disgustingly biased. The conclusion is that Grok holds its own?! There was zero evidence of that.
deanalyzer 2 minutes ago [-]
Yes, I wonder how the verdicts would hold under a blinded test. This analysis read like the authors going out of their way to be supportive of Grok.
luciana1u 56 seconds ago [-]
[dead]
yashthakker 2 hours ago [-]
[dead]
maxdo 5 hours ago [-]
Tried at work , this release def a moment I will remember. My work is not the same . The model is the first model that offer exactly as I want :
For hard tasks , that needs precision I will wait and pay expensive tokens
For everything else , query data , logs, rolling out releases , I’m using grok and it’s much better vs other tools and much cheaper too .
Tostino 4 hours ago [-]
Who the hell tries something that's been out a few hours and says "My work is not the same"?
This feels like a kid trying to do science. The will is there, but lacks experience.
I keep it pretty up to date (tomorrow Grok 4.5 and Sonnet 5 should be pushed).
Though it crunched most of the free quota, 47111 tokens, so I couldn't make multiple attempts.
I really wish they'd thrown in something like GLM-5.2 into the comparison.
We made Grok 4.5, GPT-5.5, and Claude write a blog post about using Grok 4.5, GPT-5.5, and Claude to build the same apps.
I sort of do this whenever an interesting new model comes out.
I am trying to figure out how many LLM converged on a writing style that resembles a LinkedIn MBA true believer. Maybe because there was just such a sheer mass of corporate-speak drone writing out there in the wild in the training data set?
But more seriously, is there a firefox extension that 'skims' the text body content of a page and puts some kind of "this was probably written by AI" meter, gauge, number or indicator in the top menu bar adjacent to the URL bar? It could even be color coded in various shades from green, yellow, orange, red. If there isn't, it sure seems like something that would be good to have.
I don't get why cost per reply is at all relevant here?
Why do so few who attempt comparisons actually compare dollars per task.
Yes, subjective. But it matches my repeated experiences with these models for what it is worth.
What’s more interesting to me than time-to-first token or latency is the time it takes for the agent to execute, from starts to finish, excluding when it’s waiting on a human.
Do it again but this time get them to make a multiplayer online Jetmen REVIVAL game. Online play is key, because it's very complex. Jetmen is a good game for this since it has physics and customization that's complex enough but still simple.
> "Nay, laddie, that’s no’ the real AI Scotsman! He’s grander still! More powerful! Just wait for the next model!"
GLM is the clear winner:
https://chat.z.ai/space/t19sx5kvw631-art
I use different models all the time. And mostly lower cost ones. I do not know how people write software these days, but I have clean instructions, usually in Epics and they have Tasks.
I have been using DeepSeek V4 Flash for much of my coding in https://github.com/brainless/akar for example. Planning is mostly done by Qwen latest (in opencode) or Sonnet.
For my commercial, client work I use Claude but barely use Opus. Sonnet does most of the work. For a recent project, I went through a 35 page PRD in about 4 weeks, that includes client calls, changes, Ecpi/Task generation, a massive test suite, deployment.
Variance in quality on these things is so, so high.
Written by Claude. Ugh. If it’s worth publishing, it’s worth proofreading, folks.
For hard tasks , that needs precision I will wait and pay expensive tokens
For everything else , query data , logs, rolling out releases , I’m using grok and it’s much better vs other tools and much cheaper too .