throwdbaaway

8 ms·

1.
by throwdbaaway 20d ago
Hold on.. the routed experts are in FP8 now? Previously they were in BF16. Nice, this shall cut my download time by half!
2.
by throwdbaaway 21d ago
Sounds like that's what z.ai did to get GLM-5.3-Flash running on Huawei chips.
3.
by throwdbaaway 21d ago
Exactly. Coding for inference is solved. CUDA is no longer a moat.
4.
by throwdbaaway 22d ago
> $20k workstation, best case: $15k M5 Ultra 512GB, 36-month amortization, ~$440/mo. Runs a GLM-5.3-class model at ~30 tok/s. Saturated 24/7 it produces roughly 58M output tokens/month. For agentic coding, ~90% of the
5.
by throwdbaaway 25d ago
As for KV cache quantization, Q8_0 from llama.cpp / ik_llama.cpp should also work better than FP8 from vllm (see https://github.com/vllm-project/vllm/issues/33480#issuecomme... ).
6.
by throwdbaaway 25d ago
> Both the NVFP4 and AWQ W4A16 failed to properly close their tool calls ... If I understand correctly, this failure mode is just not possible with llama.cpp / ik_llama.cpp, which enforces token generation to follow the grammar once
7.
by throwdbaaway 1mo ago
Reproduced on the CUDA stack right? Let's say DeepSeek is being forced to use the CANN stack, and the new pricing reflects the cost when 100% of inference is done with Huawei chips. Then, I suppose we can infer that: * CANN stack is 1.
8.
by throwdbaaway 2mo ago
Objectively speaking, the 2 bit quant from antirez has very low accuracy. Meanwhile, his 4 bit quant does have decent accuracy, but is a bit pointless by being bigger than the full precision MXFP4 quant. Anyway, they all work fine in practi
9.
by throwdbaaway 2mo ago
They need to get a license from moonshot to provide inference for K3. Probably have to follow the pricing set by moonshot as well.
10.
by throwdbaaway 2mo ago
It works, thanks to https://github.com/ikawrakow/ik_llama.cpp/pull/1911 , which got merged in early June. However, there might still be some issue with the chat template.
11.
by throwdbaaway 2mo ago
I suspect this is why DeepSeek had to introduce the 2x peak hours pricing. The price would be too low otherwise.
12.
by throwdbaaway 2mo ago
Yeah antirez made a lot of big claims in that paragraph. Sounds like a case of AI psychosis.
13.
by throwdbaaway 2mo ago
If you max out the ram, TG with q3 should be at least 10 t/s. And with dsa, it can still stay close to that number as the context grows.
14.
by throwdbaaway 2mo ago
That's exactly what I said. They do care when FLOPs are involved. Restoring an old session with 900k tokens will require a lot of FLOPs to reprocess the 900k token. Meanwhile, they don't really care if you use hundreds of millions
15.
by throwdbaaway 2mo ago
Different sessions. With https://github.com/fairydreaming/llama.cpp/tree/dsv4 , 1M context with DSV4 Flash takes less than 6GB of VRAM. I can't run DSV4 Pro, but it should take less than 9GB of VRAM for 1
16.
by throwdbaaway 2mo ago
Well I wouldn't call it a low bar, since some of the edits were quite complex. And 1M context in less than 6GB of VRAM is truly impressive, but somehow this gets way less attention than the crappy turbo quant from Google.
17.
by throwdbaaway 2mo ago
While we are all speculating, Boris kindly provided some guidance in https://news.ycombinator.com/item?id=47880089 > The challenge is: when you let a session idle for >1 hour, when you come back to it and send a promp
18.
by throwdbaaway 2mo ago
Indeed they are all lossy. Not sure how much they contribute to the quality loss in long context though. I got a 700k session with DSV4 Pro (official API), and the model was still coherent and didn't make any tool call error.
19.
by throwdbaaway 2mo ago
The current top comment in https://lobste.rs/s/ua1gxl/glm_5_2_coming_ai_margin_collapse correctly zoomed into cached input tokens, but landed on the opposite conclusion: > That is, for your $100/month fee,
20.
by throwdbaaway 2mo ago
Seems like a pretty pointless post that still centers around output tokens. In agentic coding, cached input tokens is 90% of the API "cost". It doesn't require GPU compute, and DeepSeek has shown that it can be done 50~100x c
21.
by throwdbaaway 3mo ago
And somehow they claimed that it is "lossless".
22.
by throwdbaaway 3mo ago
On ZFS with zstd compression, I am getting 1.34x compressratio for the BF16 weights (across multiple models). Here's the du output for GLM-5.2: $ du -s -BG /cube/models/zai-org/GLM-5.2/ 1099G /
23.
by throwdbaaway 4mo ago
And their disk-based caching is amazing. I got a long 700k context session spanning more than a week, with pauses in between that was longer than a day, and some rewinds mixed in as well. Stats from pi: ↑400k ↓438k R432M 71.9%/1.0M Hal
24.
by throwdbaaway 4mo ago
Hah, that's because the prompt itself was only about 30 tokens. We need a much bigger prompt to properly test PP.
25.
by throwdbaaway 5mo ago
Huh that's not what I gathered from the tweet at all. If I am going to write a five why's analysis, the immediate cause is the LLM wrongly decided to delete a volume, while the root cause is the bad design to co-locate staging and
26.
by throwdbaaway 5mo ago
If I understand correctly, both the staging database and the production database share the same volume. Thus, production data was gone as well after deleting the volume. 1st hint - the API call only contains one volume: curl -X POST h
27.
by throwdbaaway 5mo ago
Should be about 10~20 GiB per session. Save/restore is exactly what DeepSeek does using its 3FS distributed filesystem: https://github.com/deepseek-ai/3fs#3-kvcache With this much cheaper setup backed by disks, th
28.
by throwdbaaway 5mo ago
Based on the release schedule of 3.5 previously, my optimistic take is that they distill the small models from the 397B, and it is much faster to distill a sparse A3B model. Hopefully the other variants will be released in the coming days.
29.
by throwdbaaway 5mo ago
His Vibe Coding book is invaluable as a textbook example of slop.
30.
by throwdbaaway 5mo ago
https://github.com/anthropics/claude-code/issues/46829#issue... - Have you checked with your colleague? (and his AI, of course)

More ›