8 ms·
Smaller, faster, safer: running Kimi and GLM at scale
- brokenodo 1mo agoI was interested in reading this until my slop detector went off at the paragraph starting with “It's worth being precise about where the benefit comes from, because it isn't raw speed.” I love AI, but I really hate reading it.
- hankbond 1mo agoI have had to stop commenting this because it would end up on 50% of the posts here. I really wish we could flag prose as ai-generated on here and just filter it out.
- gr_norm 1mo agoLinkedIn (of all places!) announced a button for flagging this recently: https://www.linkedin.com/posts/hsrinivasan1_ai-slop-is-a-top-priority-for-all-of-us-share-7488612006321889282-Ps8Z https://www.linkedin.com/posts/hsrinivasan1_ai-slop-is-a-top... How well it would work on this site, I'm not sure.
- hankbond 1mo agoNext up, LinkedIn starts using this feedback to train a classifier. They then announce an officially approved "not slop" classification only for LinkedIn Gold member posts. The classified posts have a wider reach due to everyone filtering out AI slop. Non-members automatically get bucketed in with the slop bc they don't pay to have the verified classifier run on them.
- serf 1mo agoseems like at that point it'd be a bigger benefit for LinkedIn to provide the one and only model trained to subvert the human capability to detect slop using that new negative corpus. then we're all screwed I guess. lol
- speedgoose 1mo agoIf it works, it’s going to be the best feature introduced by a social network in a long time. Incredible that it comes from LinkedIn.
- SchemaLoad 1mo agoI wonder if the realised the entire website has become the most unbearable AI slop imaginable and if they don't do anything about it they won't have any real humans reading posts, just agents trying to advertise to each other.
- physix 1mo agoBetter would have been to offer a button to flag something that does NOT seem like AI slop on LinkedIn.
- Oras 1mo agoIf there is an action on AI slop on LI, it will end up with almost no posts at all
- trollbridge 1mo agoSign up for Pangram and install the browser extension; covers X, Reddit, and Substack, and more to come.
- dgellow 1mo agoDon’t stop commenting about it, if there is something we (the readers) can do is ensure it is seen as uncool to post slop content
- serf 1mo agofine but please realize that also just pushes people away from caring about the comment section at all, too. I can only read so many (either pro OR against) ".. IT'S AI! ! !" comments before skipping the thread. I can't be the only one.
- hamdingers 1mo agoDrives me nuts that comments are held to a higher standard than submissions. HN is for conversation between humans[1] (about AI generated blogspam, apparently) 1. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- mannanj 1mo agoMeta recently added a filter as a requirement for posts on Facebook. if it was ai generated, you are required to check off a box for that on your posts. I've been asking for that for some time.
- HDBaseT 1mo agoThis is only to help them filter out AI generations for their own training data. There is no way you can "block" all "ai generated content" from your view.
- leawi 1mo agoRecently, I wrote a small script for myself that scrapes the comments to see if anyone already said it's AI, and if so, grays out the article...
- arjie 1mo agoCloudflare blogs are not meant to be human-read, AFAIK. They're raw material meant to be fed into an agent to be filtered down. I rarely read the contents because they are usually word-expanded to a greater degree than an article from The Atlantic.
- dgellow 1mo agoThat’s disappointing, in the past cloudflare had some of the best engineering blog articles
- arjie 1mo agoI don't disagree, but at some point in the last year they ended up severely word-expanded. So in a revealed sense, they are no longer meant for human consumption except for those who don't significantly value their own time. There is very little information in the post that an agent can't pull for you: * they use quantized models * they quantize KV cache * they have a cache tagging mechanism to prevent cache misuse (neat) The agent can extract numbers without filler prose as well.
- mrkstu 1mo agoI'm getting somewhat conflicted. There is a Twitter account I follow that has great takes and lots of deeply personal posts... BUT he obviously uses an LLM in his writing pipeline somewhere. Too many AIisms scattered about to ignore. I'm not thrilled with it, but he is obviously using it to improve his writing overall- to communicate some great ideas that are personal and germane. I've decided that being too inflexible serves no one. If it is true slop, I'll not revisit the writer in the future- if they are using AI to polish writing that at its core is a unique voice, I'll accept it and learn to live with it...
- colingauvin 1mo agoCame here to comment this.
- TacticalCoder 1mo ago> “It's worth being precise about where the benefit comes from, because it isn't raw speed.“ What's funny it's that is as if AI "learned" to speak english but not really. People simply don't speak using those strange constructs: those sentences sound a bit like if a "Karen" was trying to make a point. What's scary, to me, as a dev using AI, is that those LLMs do the same thing with code: it looks like proper code, but it really ain't so once you dig a bit. It's verbose and doesn't add anything: it's just infinite verbiage / sloppy-pasta. Crazy thing though it's that it's 2026 and apparently devs can't be bothered to copy/paste their sloppy-pasta LLMish into a de-sloppifier before publishing blog posts.
- lukan 1mo ago"What's scary, to me, as a dev using AI, is that those LLMs do the same thing with code: it looks like proper code, but it really ain't so once you dig a bit." Of course it always looks better, than it is, but it still works for me as in the code does what I asked the agent to.
- deleted 1mo ago[deleted]
- buttonpusherr 1mo agoI hope you can get over it, because before long literally everything will be written by AI. Millions of people write with this every day.
- mrtesthah 1mo agoSounds like a great way to filter out all the pointless garbage on the internet.
- lelanthran 1mo agoThat makes human written content even more valuable to advertisers. After all, why would advertisers want to advertise to bots? This whole sentiment of "You'all need to allow my bot postings in your group" is getting tiresome. There's plenty of places where your bot can talk to other bots, publish for other bots, etc.
- spopejoy 1mo agoWhat's the tell, other than it butchering the context (bad writing)? The article as a whole doesn't seem horribly LLMy. The overall topic is pretty dry, and that sentence is honkingly bad, but not all bad writing is LLM-originated -- sometimes insufficient proofreading is at fault.
- scrlk 1mo agoNice to see a provider being transparent about KV cache quantisation. I've been suspecting that some providers do this silently whilst heavily promoting their unquantised weights, even though KV quantisation can degrade quality more than weight quantisation. However, I wish their testing were more detailed. Firstly, some model families are more sensitive to KV quantisation than others (only Kimi K2.6 was tested). Secondly, the evaluation suite they use to claim that FP8 KV quantisation is indistinguishable is noticeably lacking coding benchmarks; in long-running tasks, minor tool call errors compound over time.
- anonova 1mo agovLLM's study also concluded that "FP8 can deliver meaningful latency and capacity gains with small or negligible accuracy loss". Their benchmarks include LiveCodeBench 6. https://vllm-project.github.io/2026/04/22/fp8-kvcache.html https://vllm-project.github.io/2026/04/22/fp8-kvcache.html
- deleted 1mo ago[deleted]
- Tepix 1mo agovLLM tested with Llama-3.3-70B-Instruct, Qwen3-30B-A3B-Instruct-2507, Qwen3-30B-A3B-Thinking-2507, and Qwen3.5-27B. I'm wondering if it needs to be tested with every other model or not.
- scrlk 1mo agoIMO, yes. For example, Qwen 3.x is insensitive to weight and KV cache quantisation, whereas Gemma 4 is more sensitive: https://localbench.substack.com/p/kv-cache-quantization-benchmark https://localbench.substack.com/p/kv-cache-quantization-benc...
- amluto 1mo agoThey made an extremely strong claim: > None of this would matter if it changed the model's answers If they want to assert that the answers don’t change, then perhaps they should calculate the statistical distance between the token probability outputs or something to that effect. I doubt the results would indicate that the answers don’t change by any reasonable interpretation. Maybe the results are still good enough.
- syntaxing 1mo ago> View pricing in the Cloudflare dashboard ↗ Why… I wanted to see if it’s worth it to use cloudflare’s endpoint but I can’t even see the pricing
- maxmcd 1mo agoI'm not sure how accurate this is, but there is pricing here: https://openrouter.ai/provider/cloudflare https://openrouter.ai/provider/cloudflare
- culi 1mo agoSo they don't even support K3? What's the point. K2.7 Code is practically free already
- BadBadJellyBean 1mo agoThey do: Input tokens (per 1M)$3.00 Cached input tokens (per 1M)$0.30 Output tokens (per 1M)$15.00
- culi 1mo agoNot sure where you're getting that from but it's not on the pricing page. Nor is K3 mentioned in the post
- tyingq 1mo agoI don't see it via openrouter, but it is in the dashboard if you log into CF. https://imgur.com/a/Sb7jQ0A https://imgur.com/a/Sb7jQ0A $3/$0.30/$15 ( input / cached input / output, all per 1M )
- lostmsu 1mo agoSo they quantize models, only tell about it in the blog post (instead of a warning on the model page), and even in the blog post pretend there's no difference by benchmarking on small context tasks many of which are saturated. Coding agents will probably be severely negatively affected by KV quantization. I'd say serving quantized models without saying so on the "store" page is fraud.
- whimsicalism 1mo agodon't disagree, but there is a big difference between 'quantized model/weights' and quantized activations
- joshuamcginnis 1mo ago> If squeezing the best open models onto GPUs and serving them to millions of developers sounds like your kind of problem, come work with us. What is the typical job title and/or skillset for this?
- simonw 1mo agoI've seen this called MLOps.
- buildbot 1mo agoMLOps though typically wouldn't been going quantization? It requires some careful testing of accuracy and performance even today. Applied ML Research Engineer or something maybe, not that I have ever seen that title. Maybe just catchall ML Engineer...
- deleted 1mo ago[deleted]
- overfeed 1mo agoWhat is the typical job title and/or skillset for this? It can be any one of many jobs depending on how high close to the metal one's focus is, but the highest headcount role is usually SRE/Infra/Ops with GPU knowledge sprinkled on top. That is to say Linux sysadmin, networking, fleet management, scaling, incident troubleshooting, etc.
- Tepix 1mo agoLatent space is calling the whole subject "inference engineering" in their episode https://www.latent.space/p/inference-eng https://www.latent.space/p/inference-eng
- om8 1mo agoWhy int4? There are a lot of superior 4 bit formats like nf4 from bitsandbytes.
- Tepix 1mo agoWell, INT4 is good enough apparently and it's very fast.
- HDBaseT 1mo agoI think Cloudflare not providing ZDR on their inference is the biggest public indicator that Cloudlare glows. We let all traffic get MITM'd, now we're letting our AI conversation get tracked. Cloudflare reeks like a US Honeypot.
- deleted 1mo ago[deleted]
- stingraycharles 1mo agoThis sounds like FUD unless backed by evidence. Cloudflare’s inference absolutely does support ZDR, as long as you use unified billing (ie not using BYOK).
- zx76 1mo agoSeems like the relevant page is: https://developers.cloudflare.com/workers-ai/platform/data-usage/ https://developers.cloudflare.com/workers-ai/platform/data-u... and their privacy policy also applies: https://www.cloudflare.com/privacypolicy/ https://www.cloudflare.com/privacypolicy/ "Your inputs (e.g., text prompts, image submissions, audio files, etc.), outputs (e.g., generated text/images, translations, etc.), embeddings, and training data constitute Customer Content. For Workers AI: * You own, and are responsible for, all of your Customer Content. * Cloudflare does not make your Customer Content available to any other Cloudflare customer. * Cloudflare does not use your Customer Content to (1) train any AI models made available on Workers AI or (2) improve any Cloudflare or third-party services, and would not do so unless we received your explicit consent. * Your Customer Content for Workers AI may be stored by Cloudflare if you specifically use a storage service (e.g., R2, KV, DO, Vectorize, etc.) in conjunction with Workers AI." OpenRouter has a page of different providers and what OpenRouter understands their position on this to be: https://openrouter.ai/docs/guides/privacy/provider-logging#data-retention--logging https://openrouter.ai/docs/guides/privacy/provider-logging#d... For CF they write: "Cloudflare • Prompts are retained for unknown period • Does not train" So in the sense of training models on your company's codebase or maybe running analytics on prompt content for the purposes of improving their AI product suite they won't use your data. Though presumably for purposes of security/abuse etc. there will be some level of retention as per their privacy policy. I guess it comes down to how much you trust providers on openrouter who claim to offer absolute ZDR versus Cloudflare and how they would use retained data. Obviously if someone thinks CF is a honeypot designed to sidestep the rise of LetsEncrypt/widespread HTTPS then they wouldn't trust a ZDR claim by them in any case. Would you then trust some of these frontier labs respective claims of ZDR when they are pushing unbelievably hard to win? I don't have strong opinions for this - I have a pretty conservative approach by default and exclusively use local inference on in-office hardware for anything close to or related to customer data. I do some coding on 3rd party services. I imagine if Mullvad offered a ZDR set of open model endpoints with similar efforts at building trust like their VPN it might be popular.
- vietvu 1mo agoThanks for the transpiration but this is too shallow when talking about LLM serving.
- bigbaguette 1mo agoCould you suggest good resources about LLM serving? Blogs, articles...
- vietvu 1mo agoI cannot think of any that so concentrated. Best place to me is r/localllama on reddit. Unsloth website and twitter posts is a good place too (usually in localllama too).
- Tepix 1mo agoThe latest Latent.Space episode https://www.latent.space/p/inference-eng https://www.latent.space/p/inference-eng
- magenta4 1mo ago[dead]
- mordae 1mo agoYeah, I've noticed them recently on OR and whitelisted. Then I backed-off pretty quickly after seeing the cache hit rates. It was also rather revealing to see how some provider hit rates differ when you are using them directly vs via OR.
- adoltech 1mo agoWorth knowing a cache hit rate can be structurally zero and look identical to a broken one.
- mordae 1mo agoOf course. On the other hand if you see 8 % gap for similar workloads, averaged across tens of sessions, with the same underlying model, it becomes a pretty clear signal. And I do exclude first request per provider per session from the statistic.
- runtime_lens 1mo ago[flagged]
- WhereIsTheTruth 1mo agoThe Cloud killed HW trust, SW is next One vCPU means nothing, which chip? which SIMD? what RAM speed? what storage? what latency? Quantization is the next layer of lies Society is moving towards offloading intelligence to clouid overlords, since nothing runs on your computer anymore, you can't trust anything And since datacenters occupy a physical space, they are building a monopoly defacto Unless transparency becomes mandatory, we are headed towards the biggest self sabotage mankind has ever witnessed
- egeozcan 1mo ago> One vCPU means nothing, which chip? which SIMD? what RAM speed? what storage? what latency? Don't forget: When will all of this change?
- lukan 1mo ago"Society is moving towards offloading intelligence to clouid overlords, since nothing runs on your computer anymore, you can't trust anything" You like sounding dramatic? There is a RAM explosion and big models already work on laptops. Some people will offload everything, some stay in control.
- muend 1mo ago[flagged]