spmurrayzzz

5 ms·

1.
by spmurrayzzz 27d ago
My expectation is always that sm120 is the forgotten orphan in the mix and that I'll have to monkey-patch my way to getting things running (and its not always due to lack of tcgen05 or other "real blackwell" features, its oft
2.
by spmurrayzzz 1mo ago
MoE's in the abstract often get presented as if theres explicit layers of experts for any given domain of knowledge, like your coding tasks are being routed to coding experts, but it's really not that at all. THe original MoE pape
3.
by spmurrayzzz 1mo ago
Yes, these are coding tasks in the embedded systems domain (I mentioned Rust and C).
4.
by spmurrayzzz 1mo ago
Right I take your point, but that's not the comparison in this case though. These are paired results on the same 250 tasks, so whether a 4 point gap is meaningful depends on the task-level deltas, not just plugging p=0.04 into the bino
5.
by spmurrayzzz 1mo ago
4% is within the margin of error anyways for pass@1, so I think pass@k > 1 is gonna be the better indicator of any movement (still need to calibrate the optimal k to re-test). 10 seems too tolerant even though that tends to be the next t
6.
by spmurrayzzz 1mo ago
> They do not beat opus on real-world usage We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores on
7.
by spmurrayzzz 1mo ago
The biggest gain you'll get is faster memory, provided you have enough capacity to load all the weight into vram. The DGX sparks and Apple silicon memory bandwidth (and also memory access latency) drag down the decode speed quite a bit
8.
by spmurrayzzz 1mo ago
Slop language aside, the abstract insight is directionally correct I think. CPUs were already becoming much more important during training for test-time scaling, but there you were still bottlenecked by GPU compute since the gradient update
9.
by spmurrayzzz 1mo ago
Given the current throughput figures on OpenRouter (~180 tk/s), its likely a much smaller param count on the order of something like Luna. I think the better, more timely comparison (re: your point on Chinese labs) would be to DeepSeek
10.
by spmurrayzzz 2mo ago
They've open sourced some of the training and inference code: https://github.com/soofi-project Their work is based on the nemotron arch (so far).
11.
by spmurrayzzz 2mo ago
When orgs/bencmarks claim 1% deviation, in most cases that means measuring perplexity loss on datasets like wikitext or c4. Even if the loss is calculated via KLD or similar, its not a good proxy for whats actually degradaing at the ta
12.
by spmurrayzzz 2mo ago
I think its good advice to test both on your own evals for sure, but the MoE parameters are already natively FP4 in ds4. Dropping to 2bpw isn't as big of a loss as it seems (and as corroborated by antirez's work). Its also only 13
13.
by spmurrayzzz 3mo ago
I had a similar reaction to the "I work at an AI company" and finding out it was Dropbox. And I agree with you, they are not in any way an AI company that would be relevant for someone making claims about frontier intelligence. I&
14.
by spmurrayzzz 3mo ago
I'm not entirely up to date with the latest batch, but I've reviewed some of the rollouts in the past and my sense is that the models are surprisingly good at getting correct custom kernels in the happy path, but still weak at sus
15.
by spmurrayzzz 3mo ago
Near-term acquihires are certainly a likely bet I think. But given model progress on related benchmarks like kernelbench [1], I do think a set of more commoditized solutions is also inevitable. The caveat though is that each new gen of hard
16.
by spmurrayzzz 3mo ago
I used it, but only briefly to evaluate it. It had some overlap with a tool I built myself, was curious if any of the extra features would be useful. Ultimately I found the data model and UI to be both cumbersome and unintuitive. Langfuse e
17.
by spmurrayzzz 3mo ago
One of my favorite code comments of all time is still in the src: "# TODO: implement a proper validator to compare against ground truth. For now we just check for exact string match on each line of stdout." [1] This was one of my
18.
by spmurrayzzz 3mo ago
I don't think its smoke and mirrors, though I do have plenty of gripes with how the labs market this product landscape generally speaking. The newest biggest model can still matter even if you do not run every prompt through it. You&#x
19.
by spmurrayzzz 3mo ago
> You can't on the one hand say "customers are beginning to understand they can spend less" and on the other hand suggest that this is good for forecasts of revenue. Sure you can. Just because there is a non-zero amount of
20.
by spmurrayzzz 3mo ago
There is a piece of this I agree with. That you do not need to be a deep technical expert to notice that a company is burning cash by overcommitting to capex, or relying on heroic revenue projections that may or may not come to pass. But th
21.
by spmurrayzzz 3mo ago
There's examples both in his writing and also in his appearances on podcasts, interviews, etc. I'll cherry pick a couple: “When these new models ‘reason,’ they break a user’s input and break into component parts, then run inferenc
22.
by spmurrayzzz 3mo ago
He has also consistently demonstrated, at least to me, that he doesn't really understand how inference works from a technical perspective, which weakens much of his core thesis for why there should be a collapse. I do value having some
23.
by spmurrayzzz 5mo ago
This depends a bit on your cost sensitivity and what model families you want support for, but Baseten and Fireworks have been my goto. Currently Baseten has ~610ms TTFT and ~82 tk/s for Kimi K2.6, which is roughly 2x the throughput of
24.
by spmurrayzzz 7mo ago
First as an aside, remember that this entire thread is about using local compute. What you're alluding to is some fantasy infinite budget where you have limitless commodity compute. That's not at all the context of this thread. Bu
25.
by spmurrayzzz 7mo ago
> That’s kind of a moot point. I don't believe it's moot, but I understand your point. The fact that models are memory bandwidth bound does not at all mean that other overhead is insignificant. Your practical delivered throughp
26.
by spmurrayzzz 7mo ago
No I'm saying there are quite a few more bottlenecks than that (I/O being a big one). Even in the more efficient training frameworks, there's per-op dispatch overhead in python itself. All the boxing/unboxing of python o
27.
by spmurrayzzz 7mo ago
For inference, even with continuous batching, getting 100% MFUs is basically impossible to do in practice. Even the frontier labs struggle with this in highly efficient infiniband clusters. Its slightly better with training workloads just d
28.
by spmurrayzzz 8mo ago
Bits per weight, its an average precision across all the weights. When you quantize these models, they don't just used a fixed precision size across all model layers/weights. There's a mix and it varies per quant method. This
29.
by spmurrayzzz 8mo ago
I've tested this myself often (as an aside: I'm in said community, I run 2x RTX Pro 6000 locally, 4x 3090 before that), and I think what you said re: "willing to wait" is probably the difference maker for me. I can run M
30.
by spmurrayzzz 8mo ago
When I've measured this myself, I've never seen a medium-to-long task horizon that would have expert locality such that you wouldn't be hitting the SSD constantly to swap layers (not to say it doesn't exist, just that in

More ›