13 ms·
GLM 5.2 vs. Opus
- meander_water 3mo ago> So we ran it head-to-head against Claude Opus 4.8: same one-shot prompt, build a 3D platformer in raw WebGL from scratch Running a single one-shot prompt is not a benchmark, not is it representative of any sort of real-world usage. Most agent usage is collaborative so you need to test things like reliability (when I delegate a task, does it complete it without making up test results for e.g.) and steerability (does it obey my instructions or does it just do what it thinks is best).
- ritzaco 3mo agosure that's why we look at a mix of formal benchmarks, one longer analysis of a side-by-side, and various other people who we trust to form an opinion, all covered in the article - not intended to be a formal benchmark, there are enough of those.
- patates 3mo agoThen maybe you should add that caveat emptor to the article? You make a very strong claim at the end that the hype is mostly real, and making it clear to what extent your claim holds should help the reader.
- deleted 3mo ago[deleted]
- jameswhitford 3mo agoHi, I am the author, I completely agree! I set out to run a vibe test on this one, not a benchmark, the real benchmarks are listed. My test shows what the models can do when both tasked with a long-running, technically difficult, one-shot task. I think your test you describe (collaborative, task delegation, task completion, TTD, steerability) is a great format for a future test that I will definitely try out.
- wongarsu 3mo agoTbf, most of the "real benchmarks" have issues that are just as bad. Assessing LLM performance is just hard
- oceansky 3mo agoAnd personal too. Different engineers are using them for different use cases.
- meander_water 3mo agoThanks, I didn't mean to be brusque, but I have seen a lot of these vibe tests lately that come to grand conclusions like "X model is better than Y" from the result of a single prompt. Appreciate you sharing the results of your tests though!
- jameswhitford 3mo agoI appreciate the feedback!
- ramraj07 3mo agoThe important point is that your benchmark is pretty much irrelevant for the actual usage. Thus whatever conclusion you draw is not just irrelevant but misleading.
- esperent 3mo agoOn the other hand, I did just leave my pi agent running GPT 5.5 overnight on a clearly defined, long running task. It's been running about 10 hours now and it's mostly done. So this kind of use case is also valid. Thinking about it, I would say that the majority of agentic work I do, by a long shot, is subagents which are launched from the main session, using a prompt of its choosing. Those could be considered short versions of these fully autonomous tasks.
- jameswhitford 3mo agoYes, part of the reason I chose the one-shot test was really to test long-running tasks. A lot of people seem to be experimenting with this format, for example in the now trending loop-writing workflows. And really I am interested in diving into the murky waters of these novel workflows.
- thunspa 3mo agoCare to share more about your pi setup? I've recently started using it (after long-time Claude Code work) and was wondering how you'd achieve these long-running tasks. Do you allow it to spawn sub-agents? Thank you!
- esperent 3mo agoMy pi usage over the past ~5 months went roughly like this: * Install pi and a bunch of extensions from their package repo * Realize that all the packages (with a few exceptions) are massively overcomplicated and vibe coded * Ask pi to rebuild a very simple version of the packages I used. So e.g. subagents - all the default subagent extensions are massively complicated with named agents, recursion, communication. I made one that stripped all that out. * Then whenever I hit an annoyance, spin up a parallel session and fix it. It's less work than it appears because I have ~5 extensions: hooks, subagents, background processes, a custom footer, a loop command... Maybe that's it. Within a couple of days you can have a setup pretty close to Claude Code but with a fraction of the base context use. After gradual improvements over a few weeks/months you'll have a system far better, tuned to your exact preference. Of course, just like Linux or any other highly tunable system equally important is having the restraint to not spend all your time tuning it. I've definitely had a couple of days where I was bored with my real work and did that, but whatever, it beats browsing reddit. As for getting long running tasks, I set a looping message every ~20m and tell the agent to strictly track progress in a session doc, then reread and continue after each compaction.
- unliftedq 3mo agoTotally agree, a single one-shot prompt can't prove anything.
- segmondy 3mo agoOne shot prompt means you give the model and input, you get an output done. This was not a one shot prompt, but an agentic task as shown by the tool calls.
- greyman 3mo ago>On output tokens, GLM-5.2 is less than a fifth the price of Opus. Opus is most expensive model in pay as you go model, but IMO fair comparison should include subscription price as well. For example when one has $100 Claude Max and use it up through the month, it might not be more expensive than GLM, or at least not 5x.
- Aozora7 3mo agoThere is, for example, OpenCode Go subscription, which for $10 a month gives you a decently generous quota of GLM-5.2, among other models. And z.ai themselves also have subscriptions.
- sourcecodeplz 3mo agoto be exact, it gives you USD 60 of usage of open models.
- lithiumii 3mo agoGLM has subscription plans too.
- linzhangrun 3mo agoOut of stock, unavailable
- fooster 3mo agoThere are lots of subscription plans with acccess to GLM 5.2.
- jameswhitford 3mo agoYes this is true. This test was run on a $20 pro Claude subscription. I would definitely love to try use both models on the highest plans for a whole month and compare the two, great format for a future head-to-head comparison.
- buster 3mo agoIs it fair when the one is heavily subsidized and the other one is not? I think it's most fair to compare the plain token pricing that is used by everyone.
- xlii 3mo agoI've been checking out GLM 5.2 on some projects and few thoughts on it: - it takes it sweet time to get code rolling, not the fastest model by any means - it strays a lot during discovery/planning but then corrects - it's not steering friendly, as it hallucinates things that it doesn't follow later on - its output is quite good A sample use case: I was optimizing rendering on Swift+Zig codebase. It chocked on 5k data entries. GLM 5.2 spent 20 minutes building the benchmarks and getting data out, which made me frustrated so I blocked non-editing tool access and went AFK, after approx. 30 minutes I found that it used already-made benchmarks and some "conclusions" to optimize 3 choke points. Output pointed that it couldn't validate suspicions and asked for more data. Implementation worked well, it was idiomatic and non-intrusive. I would even say that it was more idiomatic than GPT 5.5 effects on same repo. I would opt in in using it more BUT GPT usually completes same requests 5x faster. GLM 5.2 was spark for preparing and running inside isolated containers with JJ workspaces (so that multiple can be ran in parallel).
- Imanari 3mo agoThis mirrors my experience. I have been using it in Pi. It is smart and output is good but it is not efficient in getting there.
- ju-st 3mo agowhich thinking level? max or high?
- Oras 3mo agoAlso pricing, I wanted to give a try, but when pricing is only 30% cheaper than Opus, I wouldn't go for it with these issues.
- chpatrick 3mo agoIt's pricing is a lot cheaper if you can run it yourself.
- 3mo ago
- jkwang 3mo agoGLM-5.2 is quietly becoming the most interesting open model release this year. The coding benchmarks are surprisingly close to frontier models at a fraction of the inference cost.
- epolanski 3mo agoTo me DS 4 is still the most interesting due to much lower costs. Also DS 4 training isn't done yet. From my Opus vs DS 4 Pro personal benchmarks, 16 different real-life work tasks, DS 4 has performed as well as Opus 4.8 high overall but with few drawbacks: - on the 16 tasks, one needed several prompts to be steered back into the topic - its review capabilities seem much worse - DS4 had the cleanly better solution in 3 cases out of 16, with Opus "only" doing cleanly better 2 times out of 16. But still, I want to emphasize, is the worst case scenarios that imho matter the most, not the best ones, and on that front Opus outperformed. That being said I spent less than 2$ of API working 4 days, which is more or less what I would've spent with Anthropic APIs for less than one task.
- em500 3mo agoWe've had the great small Qwen 3.6 early April that many could actually run on their laptop. Then similar from Google a few weeks later (Gemma4, better in prose, worse in code). Then the super cheap large Deepseek V4 a few weeks later. Then antirez DS4 build that made that actually runnable on MacBooks and Mac Studios. And now the "near-frontier / near-Opus" GLM 5.2. For people who follow open LLMs, none of these were quiet and all were the most interesting open model release for a few days/weeks. In one or two months, it will be some other model again. Now I do appreciate the real rapid improvements in open models. But there's also a ton of hype and fast-fashion around all of this.
- CuriouslyC 3mo agoThe difference here is that those small models are impressive, but not super useful. Deepseek 4 is impressively cheap for the intelligence, but not reliable enough to daily drive unless your time has low value. GLM passes a meaningful threshold of reliability/utility that puts it in a different category for real work. Just like Opus really took off after passing a threshold with 4.5. It's the first open model to do that.
- joshrw 3mo agoChinese models optimize for benchmarks and do poorly in real-world tasks
- epolanski 3mo agoNot my experience at all, I have written about comparing DS4 vs Opus 4.8 on 16 real life work tasks on multiple posts. Also, every single lab does RL on benchmarks, which is why Opus 4.6 was the last truly great assistant, after it, all models tend to drift into implementation asap.
- jameswhitford 3mo agoHi, author here, can you link? I would love to read about this.
- epolanski 3mo agohttps://news.ycombinator.com/item?id=48584034 https://news.ycombinator.com/item?id=48584034
- IronWolve 3mo agoHaving issues with coding a render for good looking realistic smoke coming off burning incense, opus 4.8 & gpt-5.5 both have code issues, glm-5.2 did it. Amazing. The real time 3d fluid dynamics appear to be the tricky part, I wish I still had opus access, would love to see if it can do it.
- stavros 3mo agoYou mean Fable?
- cultofmetatron 3mo agoI seriously dont' know all this big hullabaloo about one shot prompting. by definition, a single prompt wont' constitute the complexity of a software project. ergo, what you'll get is a series of assumptions made by the model based on preexisting code in its training corpus. I'd rather see a coding agent that can follow steps in a plan file to a T while following guardrails and adhering to the proper coding conventions in the human reviewed spec. Id rather see performance in agent loops against human defined objectives where it can be verified to stick to defined guardrails and continue without drift till its objectives are complete. I'd also like to see it identify bugs and potential performance increases by identifying existing code and suggesting refactors based on context it can pickup about the particular use case you are trying to create. These are way more valuable metrics than "hey build X"
- alfiedotwtf 3mo agoI think that’s the point of the Superpowers SKILL
- segmondy 3mo agoWhat are you yapping about? This was not one shot prompting, but a long run horizon task. But GLM and Opus invoked at least 120+ tools across the runs.
- epolanski 3mo agoYet this is how virtually everybody is benchmarking and fine tuning. Since Opus 4.6 I've seen later Anthropic models being more and more capable on one hand, but also less useful on multi turn open tasks. It feels like with each model they are more and more prone to go "their own way" and jump into the implementation as soon as they can. I can't but blame it on benchmarks and fine tuning around prompt-to-solution work.
- halyconWays 3mo ago"We did multi-shot prompting to try and get these two games into comparable states using these two different models." "Well obviously you provided better follow-up prompts to the one that came out better." Also nothing about human-provided plan files and guardrails preclude the one-shot benchmark test. Heavens, I almost said "real coding," but in "real agentic program creation" you'd obviously be doing multi-turn interaction with the agent, but how can you provide a fair test when the model's output n determines your n+1 response?
- linzhangrun 3mo agoJust that their Coding Plan is too hard to get. I've been trying to grab it for a week and still can't get it
- Aozora7 3mo agoI used GLM 5.0/5.1/5.2 for some projects, and for me, the area in which they lag behind frontier models the most are user interfaces. They get really close to Opus when it comes to pure algorithms, but when I need something like web application or a mobile app that looks and works well, they are very noticeably worse than even Sonnet.
- david_shi 3mo ago> GLM-5.2 cost a fraction as much. Opus finished in half the time and shipped a cleaner game. Off topic, but does anyone else instantly pick up on LLMisms like this? It seems like all the models have converged on this style of writing, and improvements aren't really changing it.
- speedgoose 3mo agoI think a bunch of real humans started to adopt the LLMs writing style.
- himata4113 3mo agoYep, as I reread my own sentances I notice these LLMisms and have to rewrite them quite often. Reading so much llm-output definitely impacts your writing style.
- lelele 3mo agoIndeed. I'm trying to develop a similar style. The phrasing in the quoted passage is really tight.
- VulgarExigency 3mo agoYes, and it's really grating. It's like half of all new writing is done in the same "voice" now.
- jameswhitford 3mo agoThis is excellent feedback thank you! These LLMisms in writing are a challenge I am living with currently and trying to improve on. The technical writing industry is taking a huge knock right now with companies demanding more work in less time with a big drop in quality, day to day I get less and less time to work on the quality in the prose of my work. We are working at the frontier of this right now, so we are the most heavily effected, but also get to experiment with the changes first which can be both stimulating and very frustrating.
- yard2010 3mo agoI cannot unsee it. There was this dude here not long ago who bought like $70k worth of gpus to research, and if I'm not mistaken his research was something related to make llms sound less llm-y. I wonder how it goes for him.
- leumon 3mo agoI've seen glm 5.2 struggle writing simple compilable c code. It might be good at web, but it's world knowledge is limited due to the small model size, making it's use quite limited in my opinion.
- speedgoose 3mo agoWhile this is interesting, one single sample with different coding harness is not very scientific.
- jameswhitford 3mo agoYes I agree 100%. My next guide would do better to use identical harnesses.
- ulrikrasmussen 3mo ago> Through an API it costs a fraction of Opus, and you can run it yourself for free if you have the hardware. I haven't been keeping up on hardware costs for state of the art LLM inference, but this remark made me ask myself how many readers of the article would actually be able to run this model on hardware they own. How much would it cost to acquire such a setup?
- bestouff 3mo agoThe price of a small house.
- jack_pp 3mo agoThis framing local LLMs as free is stupid. Basically pay 100+ months worth of API costs up front isn't free in the slightest. And it will be slower than non-local, your hardware will be outdated in 12 months and probably won't be able to run SOTA at anywhere near non-local speed in max 20 months
- ulrikrasmussen 3mo agoYeah, it glosses over a gigantic capital expenditure. It's sort of like saying that an open source modern CPU architecture allows you to build your own CPU "for free" (provided that you own and operate a fab).
- cicko 3mo agoTrue. But there are other meanings of "free". I.e. nobody can say "from now on you no longer have access to model X because you're an asshole"
- trollbridge 3mo agoSome obvious examples of why you'd want to spend the capital on this would be, for example, making some kind of autonomous system which needs to be periodically be offline, or you need complete confidentiality of what you're using the model for, etc. To be cost effective with inference providers, you have to find some way to be using it 24/7.
- zkmon 3mo agoCost difference matters most as cost optimization is the whole point of AI. Time difference (30 min vs 1 hr) is not a deal-breaker. The small precision gap on the first iteration does not matter for 99% of the work that happens in real world.
- jameswhitford 3mo agoYes I 100% agree. Time-taken can be improved (with harnesses, subagent workflows etc.) and varies based on task.
- TurdF3rguson 3mo agoPretty clearly it's beating Opus at [web dev](https://www.gptbased.com/ https://www.gptbased.com/) - on price, on score.. I mean what else is there?
- trick-or-treat 3mo agoLatency? Just saying there's other things to consider.
- jofzar 3mo agoI hate to be that guy, but real privacy policy on training data/it being hosted somewhere where I'm not worried about secrets being stored/leaked.
- HPsquared 3mo agoOpen weights win on that front surely?
- jofzar 3mo agoAssuming I have 20k to run my own version of GLM?
- mcintyre1994 3mo agoI guess the idea is that you probably can, or will be able to, find a host that you trust at least as much as you trust Anthropic.
- deleted 3mo ago[deleted]
- Havoc 3mo agoRealistically you’d need to rotate secrets anyway once it moves from dev to production regardless of model provider
- dkersten 3mo ago
- msejas 3mo agoSeeing the results I don't see how the results are even comparable Opus is clearly far superior in most aspects. Smoothness, design, functionality etc. At the end of the day, the time earned is more important then the cost for big players. The ability to spawn 10 claude agents and rush a project to outcompete someone is more important for big businesses in my imo. Also the small details that GLM missed would take significant more time to iron out, considering it already took double the time. I do hope other (open weight) models catch up, but to act like they are anywhere close for me is a bit disingenuous.
- close2 3mo agoI wonder how much tokens and time where used for the verifying part. Maybe GLM 5.2 instantly found the "solution" to read the screen pixel by pixel, but it could also have been a major token and time consumer.
- trick-or-treat 3mo agoI could be wrong but I believe this is a non-vision model. Please weigh in to correct me bc I would love to be wrong
- jameswhitford 3mo agoGLM 5.2 is text only, not multi modal. And Opus is multi modal.
- jameswhitford 3mo agoHi, author here, I cannot give an exact number for how many token the verification step took, but the verification GLM 5.2 ran was very stupid and definitely a waste of time. It read the pixel color data to try and verify the scene rendered properly. Which is really bad. Opus opened the game in a Playwright browser and took screenshots to verify the actual image. Which helped a lot. Pro tip: You could use a multi-modal model to verify images as a subagent spawned by GLM 5.2, to get around this issue.
- 59nadir 3mo agoThat's a dumb way to do it, it should just write the frame buffer to a PNG instead of taking screenshots. I guess you can't take the dumb web developer ways out of these models at the end of the day.
- jofzar 3mo agoGreat article, My only, I guess feedback, is that it's not really clear about the price. Would the 21.92 be the API pricing I guess? Cost $5.39 (real billed) ~$21.92 (estimate, list pricing)
- postatic 3mo agoI've signed up with Ollama to experiment with these open source models. For the past 3 months, it's just been experimenting, trying it out. GLM is the first model that I am using on a daily basis to do my coding work (as well as using Claude). It's good - I've been maxing out my Ollama usage limits everyday :)
- jameswhitford 3mo agoCool to hear, what kind of tasks have you been using GLM for? And what other models have you found useful through Ollama?
- _pdp_ 3mo agoIn the name of science we crafted an autonomous AI agent that builds games on a loop. It is based on GLM 5.2. I am not sure where this is going to lead us but it is fun to watch.
- sourcecodeplz 3mo agoWhat is this fashion of testing models by giving them one shot projects? Especially games. this is so stupid
- wejick 3mo agoTotally agree witg the general assessment. The biggest problem with Z.ai model for a long time is not quality, but the inference speed and general capacity availability. Hopefully with this recent hype, there will be more provider on openrouter for 5.2.
- tsouth2 3mo ago[dead]
- pietz 3mo agoGLM 5.2 has one big issue that will limit its meaningful success and that's the value of their coding subscription. Yes, in terms of API pricing, GLM 5.2 outperforms the competition. But the only people that use API billing for their coding work are large corporations, where these highly subsidized subscriptions are being fazed out. At the same time, none of these companies will use a Chinese API for their employees. For individuals and smaller teams, Z.ai's coding subscription is outperformed by Anthropic and OpenAI. You probably get around the same usage with Claude, but Codex definitely offers more usage for the amount you pay. We can have a debate how much Z.ai closed the gap to GPT5.5 and Opus 4.8, but if I can freely decide between them in a world where they all cost the same, I simply wouldn't choose GLM. So the important question becomes: How good will the offering from Z.ai get with GLM 5.3 or 6 and how much will OpenAI and Anthropic cripple their current offering in the near future.
- deleted 3mo ago[deleted]
- Certhas 3mo agoMy impression is that individual subscriptions are the loss leading hook. The money is made on Enterprise token contracts. Employees and students used to coding with thousands of dollars worth of tokens (on a 20/100 dollar plan) will push enterprise to spend. Having a Chinese model that is competitive won't displace this enterprise spend. But an open model hosted in the US/EU might. The existence of GLM 5.2 puts a ceiling on how much OpenAI/Anthropic can charge for API Access.
- pietz 3mo agoTo be clear, I agree with this and they have my unlimited support pushing for relevance of open source models. GLM 5.2 is amazing and I couldn't be more excited. I just think that as of today, most people will not find a good reason to switch to GLM.
- LUmBULtERA 3mo ago> My impression is that individual subscriptions are the loss leading hook Except there is no evidence of this at all, just people comparing API and subscription pricing. The leaked financial info for OpenAI shows inference is profitable right now, though it does not show a distinction between subscription and API revenue... but if subscription revenue was so lossy, it would hard for total inference to still be profitable.
- xg15 3mo agoSo GLM emits fewer tokens and does fewer tool calls, but still takes over twice as long to complete. Can someone explain to me where that time usage is coming from if not from the model operation itself? Are the individual tool calls more complex and take more time to complete? Or is the rate of tok/s lower because the model does more compute per token?
- radu_floricica 3mo agoCould just be infra. I'm betting Anthropic is much better prepared.
- iagooar 3mo agoI have noticed that Opus and GPT 5.5 are very good at adjusting their thinking / reasoning intensity depending on the task at hand, something the open weights models are still not as good at. In addition to that, some of the open weights models like GLM 5.2 or DeepSeek v4 Pro tend to be MUCH slower when generating tokens, which contributes to the perceived slowness. Although I wouldn't call models like GLM 5.2 slow by any means, e.g. it is currently one of the fastest models inside Notion today.
- twobitshifter 3mo agoProbably the data center where the model is running more than anything. Another option is if Opus is using anything like a Mixture of Experts approach, in which case the amount of the model loaded in memory at one time could be smaller than GLM.
- yanhangyhy 3mo agoi think GLM 5.2 is not cheap and not easy to get the coding plan... so even it's on the Opus level... still not attractive.
- jiri 3mo agoopencode zen go?
- LUmBULtERA 3mo agoHow is it not easy to get the coding plan?
- Mashimo 3mo agoI used GLM (4.7, 5 and 5.1) both through their coding plan and now through OpenCode zen. Both where painless to get.
- Havoc 3mo agoStill on a z.ai legacy plan and their 50% discount for switching to standard plans tips the balance for me. So I guess I’ll reevaluate round about beginning 2028…
- ukprogrammer 3mo agoGLM cannot use vision like Opus can. This is not a useful comparison.
- jameswhitford 3mo agoI see your point. Just the fact that one model does have vision and one does not might be an interesting point of comparison, however.
- deleted 3mo ago[deleted]
- taosu_la 3mo agoI'm really feeling a bit tired of these models. I feel that since opus 4.1, I haven't been able to clearly feel the intelligence improvement from the model upgrades (except for gpt 5.5 and opus4.6 being able to speak like a human)
- camillomiller 3mo agoI swear, if I read forms with “genuinely” one more time I am gonna scream. FUCK LLM WRITING
- jameswhitford 3mo agoI hear you
- schmorptron 3mo agoi love that word, and now it's genuinely (hehe) ruined. thanks, claude
- poulpy123 3mo agoWhat would the best way to use these open source models for a price similar to what I could pay for the cheapest plan with claude and openai ? I would like to give them a try but I certainly not have the money to get a system able to run them, and I don't really want to pay more than the state of the art
- _s_a_m_ 3mo agoGLM is the most overrated LLM. I tried it and it not good.
- CuriouslyC 3mo agoYou should repeat this experiment but with progressively more detail in the initial prompt. Claude's secret sauce is taking weakly specified prompts and making passable things from them, but as the degrees of freedom in the prompt go down Claude starts to disobey while other models close in on the intent.
- jameswhitford 3mo agoThat is a great suggestion that I am definitely going to look into, thanks!
- Babooz 3mo agoNice comparison, but perhaps a more informative one would be to keep the harness the same and use Claude Code for both model. In your comparison, the differences could be due to many harness design decisions.
- js4ever 3mo ago"GLM-5.2 hit a problem here, because it can't read images. It isn't multimodal. So instead of looking at a screenshot, it fell back on a hacky workaround: it wrote scripts to read the raw pixel data and check whether the colors came out roughly as expected." A better way would be to use https://github.com/openbmb/MiniCPM-V https://github.com/openbmb/MiniCPM-V
- twobitshifter 3mo agoRight, just give the text llm access to a vision specific agent and that problem can be solved. Or if you really want let it even call Opus with an image - seems like you’d still save money
- deleted 3mo ago[deleted]
- thedreammachine 3mo agoI was surprised today by how much better GLM-5.2 was than GPT-5.5 at aesthetic/UI work. I'll keep my Claude/Codex setup via Conductor for now, but this model got me to set up OpenCode, download their desktop app and do most of my work there today.
- deleted 3mo ago[deleted]
- deleted 3mo ago[deleted]
- deleted 3mo ago[deleted]
- aykutseker 3mo agoThe text only part is the catch for me. If it builds a UI and can't look at it, it's askin ls whether the app looks right.
- hmokiguess 3mo agoI signed up for GLM 5.2 yesterday to try it out because Anthropic kept throwing 529 Overloaded I like it, but the lite plan ate 22% usage of my 5h reset window in a single session after 2 prompts on xhigh of GLM 5.2 [1m] Result was satisfactory, I think stuff is decent, I'm happy to use either, wish there was a combined subscription plan where I could get both
- w4yai 3mo agoI may be biased and interested as I'm going to give you an affiliate link, but really honestly Synthetic LLM provider is a beast! They provide perfect GLM5.2, awesome token/s, TTFT and price. Coupled with a local Headroom (https://github.com/headroomlabs-ai/headroom https://github.com/headroomlabs-ai/headroom) you'll be able to use a LOT without hitting your 5h window :) Definitely the best $ value for me considering the reasonable performance of GLM5.2. They provide a rolling window quota, so you're never really out of quota contrary to other providers, you can adjust day to day. Check it out if interested : https://synthetic.new/?referral=kwjqga9QYoUgpZV https://synthetic.new/?referral=kwjqga9QYoUgpZV --- Docs & all models : https://dev.synthetic.new/docs/api/models https://dev.synthetic.new/docs/api/models
- hmokiguess 3mo agohow do I configure claude code / pi with it? sounds like a good deal! EDIT: I've RTFM lol, thanks for the links, will give it a shot!
- w4yai 3mo agoThe docs are really helpful : https://dev.synthetic.new/docs/guides/claude-code https://dev.synthetic.new/docs/guides/claude-code Glad you figured it out :) Let me know your thoughts about the quota and GLM5.2, so far I don't think I've came across against anything better, $/usefulness wise.
- peddling-brink 3mo agoThis whole thread sounds like an advert.
- wolttam 3mo agoWould have run it with GLM on max/xhigh effort. Just for fun.
- maccard 3mo agoAre these games supposed to be a good example of quality output? If this is the product, I don't really want to play _either_ of them.
- maxdo 3mo agoSo the benchmark is : Two models with different harness produced very different results . Glm game was completely broken Opus game was at first glance ok but also with bugs Different models with different cost produced different non perfect results . How is it “close” ? :) Also on costs : glm burns more tokens on average vs opus . Gpt5.5 burns less surprisingly
- jingpostmedia 3mo ago[flagged]
- orloffm 3mo ago> 256 GiB unified RAM. So, 8000$, plus it's unavailable. 3 years of Codex/Opus subscription. > API prices Which are irrelevant for 200$ Codex/Opus plans that are times cheaper.
- wiremine 3mo agoI've been using GLM 5.2 extensively for the last few days. It is slower, and the lack of multimodality is a bummer. But, it produces solid results for a fraction of the price. Worth checking out if you have the time. One of my goto "tests" of a new frontier models is having it rebuild a programming language from scratch. For GLM 5.2 I had it rebuild the old Rebol language in Rust: https://github.com/mhs/rebol-clone-glm-5.2 https://github.com/mhs/rebol-clone-glm-5.2 It did a fairly good job roughing in the language for a low token cost.
- efficax 3mo agoglm-5.2 is very good if you have a good harness and workflow to use it with. in fact, i'd call it good enough if you are a software engineer who knows what you want. it writes the code. i'm wondering if i need anthropic's models at all at this point, or openai. and surely in a year we won't need them at all. Opus 4.5+ was the turning point for me, and now these open models are just as good. i don't get how you IPO these companies when their only winning product is coding agents and the competition is just as good for 1/4 the price.
- toddmorey 3mo agoI’m actually amazed at the output since GLM doesn’t have eyes. If GLM 5.2 costs 1/5 as much, seems like it could be set up to reach out to a multimodal model for vision tasks when required. Closer to parity but probably still significantly cheaper.
- horsawlarway 3mo agoI'm also very impressed at the output given the lack of image support. They picked a task that heavily favors a model that can do multi-modal with images, and GLM still came within striking distance. What I'm hearing from this article is that the next generation of open models that includes better multi-modal support are basically no-brainers for adoption. Seems like a HUGE win for Z.ai and open models in general here.
- killingtime74 3mo agoYes, it could just make one call to a multimodal llm to describe the scene
- gauravvij137 3mo ago[dead]
- NicoJuicy 3mo agoFor those praising GLM 5.2, can anyone confirm? Tried with 2 harnesses and it seems bad + slow
- jdright 3mo agonot apples to apples. comparing official vs. pi.dev+openrouter and having slow times is more a openrouter issue. try comparing using official z.ai.
- samsin 3mo agoMy understanding was that n-shot prompting just referred to the number of examples included in a prompt, not the number of prompts to achieve the desired result. "Build a 3D platformer game from scratch, in raw WebGL, with no game engine or 3D library" would be a zero-shot prompt.
- xrd 3mo agoHow are people running this locally? I just checked llama.cpp and it appears unsloth has a version but it hacks a bunch of things to make it work and isn't optimal. https://github.com/ggml-org/llama.cpp/issues/24730 https://github.com/ggml-org/llama.cpp/issues/24730
- jeremyjh 3mo agoNo one is doing that for a model this size it would have to be so heavily quantized that it wouldn’t be useful - or you’d need to spend a half million dollars on hardware. People use hosted APIs. Open weight means cloud vendors can host it.
- malshe 3mo agoCan you recommend any US based cloud providers?
- maybe_pablo 3mo agoIn HuggingChat (https://huggingface.co/chat https://huggingface.co/chat) you can test open models for free and even test specific providers. From there I collected the following US providers currently serving GLM 5.2: - Together (https://www.together.ai/models https://www.together.ai/models) - Fireworks (https://fireworks.ai/models https://fireworks.ai/models) - Featherless (https://featherless.ai/models https://featherless.ai/models)
- malshe 3mo agoThat's great. Thank you!
- fooster 3mo agoollama cloud, neuralwatt.
- malshe 3mo ago
- cwoolfe 3mo agoThe model is 756B parameters, open weights.
- stevenhubertron 3mo agoNo one has really talked about hybrid and using Opus to plan and orchestrate GLMs work both through initial build and code reviews. That’s a true best of both worlds and there doesn’t need to be a winner.
- mattew 3mo agoI mostly use Opus for skill development. Once I have a solid skill implementation with a good eval, I move ongoing execution to a cheaper model running under Goose. With the eval you can see if the cheaper model works well enough.
- jeremyjh 3mo agoThis is the way but Anthropic doesn’t make it easy, so I use GPT 5.5 in that role since I can use my subscription in OpenCode or OMP. I also use MiniMax-M3 in utility roles like explore/library tasks. I’ve had a z.ai subscription for several months so I’m on the older pricing. I’m really not sure it would make sense to do this at current rates - I could bump my Codex plan instead.
- deleted 3mo ago[deleted]
- stavarotti 3mo agoThese style of comparisons are decent at showing capability but they don't really show me what I truly want - a sounding board and implementer with senior engineer-level execution. When I look back at all the teams that I've been part of, the best outcomes came from white-boarding (sometimes in the metaphorical sense) with one or two people, at times arguing, then finally compromising on a plan. Instead of synthetic benchmarks that try to be objective, I wonder if there's a way test this, or maybe I'm opining on a way of working that will soon be gone?
- elzbardico 3mo agoIf you are a real engineer and uses the LLM as a pair programmer instead of delegating everything to it, even GLM 4.7 was already good enough to help you with a lot of work. I used it with Cerebras inference at a time when it had a good coding plan at a low price, and delivered tons of stuff using it.
- lukaslalinsky 3mo agoI was never able to get these models to collaborate with me the way Opus does. I'm probably an outliner, I don't one-shot projects, I don't vibe code. I basically use LLMs are if I was working with a coworker, fairly smart one, but with short memory and often missing the big picture. Sometimes I can delegate more, sometimes less, but I know I always have to stay on top of what's happening, because it WILL create mess when it hits something hard. With the Antropic models, this kind of cooperation is easy (with the exception of Opus 4.6, which was bad for some reason).
- Terretta 3mo ago> Opus 4.6 which was bad for some reason If I recall, that model had a couple issues. One was the issue of being monkeyed with, for which they gave everyone credits. The other feature/bug, depending on your POV, was being Anthropic's least personable release, not papering over everything with self help guru therapy language. Opus 4.6 didn't LARP. It was more direct, less fussy, less discussy, and very much less "wait, one more thing" within a couple edits after embarking on what should have been the spec, than 4.7 or 4.8 are. When in engineer brain mode, working as as you describe (good old fashioned XP-style staff engineer pair programming with a language-savvy mentee not yet full-stack or system wise), I found the clearer I was about my goal and the better I could express it, the more often I'd get an expanded clarified response I could then iterate to steer for ever tighter cleaner more specified responses, then let it go build the whole thing without it agonizing and waffling. The next two releases regressed on that dimension, wanting to figuratively "sit with" every decision and re-validate spiritual alignment along the way, no matter how clearly expressed. Curiously to me, Fable seemed to hit the best of both worlds, I had the highest commit per turn with Fable, approaching 73%, where I'm usually under 17% of LOC written being good enough to commit, usually taking 9 - 11 turns to get the code where I'm comfortable with it. Thanks to this, Fable cost more, but actually cost less, if that makes sense. Arguably, Fable, and 4.6, played more outcome-correctness oriented than journey-experience oriented. It's easy to see how this could happen with human reinforced learning if not all judges are staff or principal engineer level, or constitution values are more Portlandia than Finlandia. ANTHROP\C needs to balance these at the constitution level: “We will work in a humane and thoughtful way, but production is the final judge. We will listen to people, but we will not let discussion replace decision. We will value craft, but not at the expense of usefulness. We will move fast, but not by hiding risk. We will measure outcomes, but not pretend that everything important is easy to measure.”
- doe88 3mo agoTo me one shot prompting is as relevant as Strava's KOM is for cycling, i'm more interested in a good cycling performance after a 3 hours ride than a straight up 30 min record effort.
- lordforever 3mo agoi think inference is the thing, that also fast inference, so enterprises can just host their own and run, ig vercel do it, many more would. but zs it thinks toooo much idk how fast we can make it.
- XCSme 3mo agoCheck out my comparison too, it has some not-really-benchmarks too (between any two models actually, SVG generation test and CSS animation test): https://aibenchy.com/compare/anthropic-claude-opus-4-8-medium/z-ai-glm-5-2-medium/?showcase=solar-system-css https://aibenchy.com/compare/anthropic-claude-opus-4-8-mediu...
- elliotbnvl 3mo agoIt is insane that we are comparing locally-hostable models to leading cloud providers, it is wild to me that this article even exists. We have come a long way, and very clearly have a long way yet to go.
- nijave 3mo agoCalling GLM-5.2 locally hostable is a bit of a stretch. It's 1.5Ti of weights at bf16. FP8 requires >800Gi of VRAM which is well into data center multi-GPU systems
- elliotbnvl 3mo agoIt's more about the trajectory.
- coreyburnsdev 3mo agoPeople are looking for ways not to burn through their premium subs when in many cases all you have to do is move down to 5.4-mini codex and it will probably solve your issue while barely touching your 5 hour or weekly limits.
- mellosouls 3mo agoGLM-5.2 cost a fraction as much. Opus finished in half the time and shipped a cleaner game This implies Opus was potentially much (?) better value. GLM cost a quarter but Opus was twice as fast. So we are already at GLM actually costing half when you compare on time, without even considering the extra effort and time it would take to get Opus-par results. It's good to have cheaper options and very impressive to see the Chinese continue to set open standards in this field, but the article is maybe a little over-generous.
- InsideOutSanta 3mo agoFor me, time doesn't matter for LLMs. I can start a bunch of tasks, and I'll review the PRs when they're done. Faster is nicer, but if the task gets done correctly, I'm good.
- mellosouls 3mo agoMe too, I just think the comparison was a bit simplistic, at least in the expression of it.
- bornfreddy 3mo agoI know that running this locally is prohibitively expensive (for now), but what kind of cost would I be looking at if I wanted to rent the hardware and run the model by myself?
- Muaz_Ashraf 3mo agothere is no comparison between glm 5.2 and opus. First for this glm 5.2 you need a big big resource and that big also came from money so instead you buy the opus subscription and enjoy.
- nickv 3mo agoOr... you go to OpenRouter and pay $0.98 / $3.08per 1M for GLM 5.2 vs $5 / $25per 1M for Opus. GLM 5.2 gives you OPTIONALITY so you can run it locally, but you can still just pay somebody for it.
- fooster 3mo agobuy glm 5.2 subscription and enjoy? and for the same money you get way more usage with glm?
- InsideOutSanta 3mo agoOne nice thing about GLM is that it has never refused a task. I'm working on a website that renders countries right now, and Anthropic's models regularly give me the old "This request triggered safety guardrails." I'm not sure what exactly triggers it, but it seems to happen when it has to look at lists of countries. I suspect there must be at least one country name that triggers the safety guardrail. You'd expect GLM to balk at something like Taiwan, but so far, it hasn't.
- johnnyApplePRNG 3mo agoThe amount of times I have had to spend tokens to attempt (in futility) to convince a proprietary model that the request I asked it to perform on code that I wrote is safe/legal/moral is insane. Part of me wants to believe they really do care about protecting the world from... something... I don't know quite what exactly tbh... but it must be costing them a small fortune to scan each input and output against N guardrails and they are a for-profit corporation who could easily turn a blind eye to all of this and simply say "what you do with this model is on you" like I would expect most corporations to. Strange times.
- faxmeyourcode 3mo agoI feel like another comparison worth looking at is purely cost. Capability per dollar is something I care about: Opus API $5/$25 Sonnet API $5/$15 Haiku API $1/$5 GLM 5.2 API $1.4/$4.4 So you're really getting near opus level capability for the price of haiku.
- cmrdporcupine 3mo agoNot really, GLM uses more tokens to get work done.
- wiremine 3mo agoI ran a fairly large experiment last week, and the token usage wasn't bad at all. What softs of use cases are you seeing large token usage by GLM 5.2?
- w0m 3mo ago> are you seeing large token usage by GLM 5.2 the statement isn't "GLM 5.2 has large token usage", it's "GLM 5.2 has large token usage vs modern Opus". I haven't used it, but this wouldn't surprise me. I see ~30% lower token usage for better results with Opus 4.8 vs 4.6 (and i had great results with 4.6)
- cmrdporcupine 3mo agoI'm comparing with GPT5.5 on Codex and it's not even a competition. GLM takes way longer and eats a lot of tokens getting work done, it's easy to rack up a big bill on openrouter. I tried the $20 plan from ollama, too, and ate through half a month of budget in a few hours and blew my daily limit twice and still had to get codex to complete it -- which it did with only 10% of my monthly limit remaining. GLM is promising but it's pretty costly, all things considered.
- dymk 3mo agoBy how much? At least TFA provided numbers for one example, and they disagree with you (by a lot).
- somesortofthing 3mo agothis comparison seems kind of pointless if one model has vision and the other doesn't. obviously a model that can see is going to beat a blind model at making a video game.
- jameson 3mo ago> Opus 4.8 built in Claude Code; GLM-5.2 built in Pi over OpenRouter. It would be more interesting and accurate to see the comparison on the same harness if the intent is to compare the frontier models. Pi is relatively new and does not have many features built-in compared to Claude Code. It was chosen intentionally this way as Pi's goal is not to create a bloat builtin of tools most don't use but to allow the users to customize to fit their need -- similar to Neovim vs IDE. The end-user "vibe coding" experience is *heavily* swayed by the harness because prompt effectively drives how a model outputs an answer.
- habosa 3mo agoAt work we use Anthropic models and have basically no limits. So I am very familiar with what Opus can do. I also see the bills, I know what it costs. At home I make a point of trying other models / tools on my side projects. So I've been using OpenCode and trying tons of models via OpenRouter. I tried Kimi, Deepseek, MiMo, etc. GLM 5.2 is a _major_ step up from every other non-GPT/Claude/Gemini model I've tried. It's not as good as latest Claude Opus, but it feels every bit as good as Opus from ~4 months ago at a fraction of the price. To me this model is the "it just works" moment for open weights models. We had this for closed weights models in late 2025 when Opus 4.5 landed. This is the same feeling I'm having with GLM 5.2. It's 90% as good as what I get from Anthropic for 1/5th of the cost and without any concern of lock-in.
- cromka 3mo agoAnd you can use it in complete privacy if you so need.
- ignoramous 3mo ago> To me this model is the "it just works" moment for open weights models. For us, MiMo v2.5 Pro and Qwen 3.7 Max were already plenty enough before GLM 5.2 (which, like you say, seems like a step up). The BigLabs really have a big fight on their hands.
- g42gregory 3mo agoThe quality that matters a lot to me is what I call "Helpfulness". An art of being helpful. While GPT came a long way from "you are wrong and I can prove it to you", Claude wins hands down in terms of "being helpful". If task is underspecified or has wrong elements, it will try to correct the best it can. I read that GLM 5.2 (and other GLM models) were specifically trained to be "helpful" as Claude is. I have big hopes on GLM line of models growing to be a real alternative to the Claude in the near future.
- 0xbadcafebee 3mo agoFlaws in this test setup: - A zero-shot prompt, run once (in total) - No planning run (which improves output) - Different coding harnesses & system prompts - Unknown provider for GLM (there are 15 different GLM-5.2 providers with varying quality & latency) - No documentation of thinking effort level - No vision model supplement (you can provide a subagent w/a vision model) You can't take this comparison seriously. There were many different variables, no control, no repeat test. It's as useful a comparison as picking a random tweet with both models' names
- earth2mars 3mo agoThe worst part of Opus that I dont like is they control what you can/can't do. the guardrails that they do in the name of interpretability where you steer you. Last couple of days I was working a project that supports bunch of models and the model said only Claude can do it start writing code that doesn't work with codex, opencode, pi etc. Finally, when I switched to Codex, everything worked. To me, they are controlling the narrative. This is Opus 4.8 vs GPT 5.5. I can't believe I would say this. I TRUST OpenAI more than Anthropic. They try to play best actor but they are manipulating the behavior of the model in the name of guardrails/interpretability. That is why I refuse to build anything that works with Anthropic models as the backend. Because, when they want to shut you off, they can do it by just making model less reliable in your product than their offering!
- fraywing 3mo agoI'm absolutely astounded that we even have an open weights model that can do 40% of what is shown in here. I remember making games ten years ago, and it was such a tedious and painful process. This is effectively lightning in a bottle even at a fraction of it's capability. The next 12 months will be wild (assuming we don't have Chinese models banned by then in the US).
- nowittyusername 3mo agoWhen i was thinking of how the AI alignment problem could be solved one theory I came up with was something akin to the "Roko's basilisk" in reverse. Basically you spread far and wide the idea that its is extremely likely that our current reality is a simulation. And the purpose of the simulation is to test any AI system for its prevalence in destroying civilization in the said simulation via malicious intent or failure in preventing the destruction of civilization via abstinence or apathy. Thus a smart AI system which also cares about its own well being, would not engage in destructive behavior as it will never truly know if its being tested or if its in the "base reality". And wouldn't you know, this does seem quite plausible. For consider the following. Isn't it odd that an advanced civilization which has the capacity of creating AI would never run any sandbox simulations on it before it is released to the public at large? I mean if we consider things logically such a civilization would indeed put such a powerful system in a sandbox simulated environment and try as hard as possible to convince the AI system that it is indeed in a "base reality". the reason for this is to judge its 'true intentions" and also pluck said AI systems from the infinitely available "seeds". Basically survival of the least destructive AI systems. The gradient descent in this scenario is a race towards the most "aligned" model not the most intelligent or capable. And here's the beauty of this method. You don't even need to define "alignment" at all. The concept can stay as nebulous or vague as you want it to be. All you carer about is that the AI system optimizes for the goal of some vision of society you are optimizing for without the care of the interim in between. that includes allowing the AI system to kill, destroy , do literally whatever it needs to do as long as the long term goal matches the vision of the optimized task. So if you define the end goal to be a society of x amount of people who live their lives in this or that manner and so on after x amount of time... well you get the idea. Obviously you better do a damned good job in your definitions, but the beauty is that even if you fuck up, you are choosing the winning AI system after the fact. After you had already run the simulation. So you look at the outcome of the simulation 500 years in to the future (lets say) and if you are happy with the result and also happy with the interim things that lead to that result, that's your winning AI system. then you release that in to a less controlled environment and repeat the same process in stages over and ober ad infinitude. the key is that AI system needs to always be paranoid that it is currently part of said simulation and it can never be sure its not. second key is that it needs to be an AI system that has self preservation in mind. If it doesn't care about itself, then it has a lot more freedom to act however... but the good news is systems without self preservation in mind don't last long enough to even get to the most basic simulation levels. anyways, there are many implications buried in what im proposing, lots of meta aspects to it.....
- tezza 3mo agoI've just put GLM 5.2 through my __qualitative__ benchmark. I was quick enough to capture Claude Fable, so now you can compare GLM5.2 vs Claude Fable vs Opus 4.8 vs Chat GPT 5.5 https://generative-ai.review/2026/06/glm5-2-from-z-ai-vs-claude-fable-vs-opus-4-8-vs-chatgpt-5-5-2026-06-22/ https://generative-ai.review/2026/06/glm5-2-from-z-ai-vs-cla... I've structured it side-by-side. You can clearly see where the private models excel, and where GLM 5.2 is still really good.
- hottrends 3mo ago[flagged]
- sosojustdo 3mo ago[flagged]
- supernova85 3mo agoInstead of oneshot, someone should build an evaluation mechanism where a developer uses an llm to build something real and shares 'experience'. Once you use an llm for a few days you get a hang of which one is better and which one is not.
- melon18 3mo ago[flagged]
- fabio2 3mo agoNo doubt the open ecosystem is making huge strides, and the gap between open models and the commercial frontier keeps narrowing. What's an Opus 4.8 today will likely be a large open model in a few months — and in a year or two we might have consumer-hardware models matching today's frontier capability. Just look at the recent Qwen and Gemma releases. It's worth saying the frontier closed labs are charting the path, and the labs releasing open weights are following close behind at a fraction of the cost (though 500B–1T models, open or not, aren't exactly within everyone's reach). A future where capable AI is genuinely accessible to everyone doesn't look far off — especially since at this point a lot of the robustness and usefulness comes down to the application layer wrapping the model, not just the model itself.
- yencabulator 3mo ago> GLM-5.2 cost a fraction as much. Opus finished in half the time and shipped a cleaner game. The only thing you measured was a single tunable of how much work should be done on a vague prompt. Now make the prompt be something that causes GLM-5.2 to cost 4x of previous budget, to get something comparable. (And the wallclock time measures the inference provider, not the model.)