zackangelo

8 ms·

1.
by zackangelo 9d ago
I just finished overhauling our speculative decoding implementation for Mixlayer, so maybe I can help. I think the piece of information that might make this click for you is the model outputs the probability distribution for all intermediat
2.
by zackangelo 13d ago
Something a lot of model providers don't talk about: any time an engine uses speculative decoding the throughput will depend on how much your output token distribution matches what the draft model was trained on. The DFlash2 draft mode
3.
by zackangelo 13d ago
just added 8 more H200s to the cluster, if you (or anyone else) runs into issues please feel free to drop me a message: zack at mixlayer.com
4.
by zackangelo 13d ago
apologies we just got a sudden burst of new users and traffic, it's scaling up now.
5.
by zackangelo 13d ago
We're serving it around 150-200tok/s (uses our new speculative decoding implementation on a DFlash2 draft model). https://mixlayer.com , LAUNCH-Q38-27B gets you $5 in credits if you want to kick the tires.
6.
by zackangelo 27d ago
What was wrong with your R1S? I had an R1T Launch Edition for a little over a year and it was hands down one of the best cars I’ve ever owned. The only issue I had was with the fob proximity sensor. The only reason I sold it was it was a bi
7.
by zackangelo 28d ago
DFlash is lossless so this would be a bug in the implementation if it is indeed a regression against the target model.
8.
by zackangelo 3mo ago
Blackwell supports nvfp4 natively.
9.
by zackangelo 3mo ago
what was the concurrency limitation? that node should be able to support a lot more
10.
by zackangelo 3mo ago
I don't believe safetensors has a native int4 dtype, so they packed 4 int4s into a bf16 in this checkpoint.
11.
by zackangelo 4mo ago
If you're in SF and weighing this decision, it's easy to get tilted in the buy direction because the rental stock is so horrific. Landlords have very little incentive to update properties or provide basic amenities that people tak
12.
by zackangelo 4mo ago
With the 3.5 release, the Plus model was just a rebrand of the open weight 397B. But I suspect that will change going forward. They haven’t released the weights for 3.6 but they did make it available through a few US providers.
13.
by zackangelo 4mo ago
absolutely not, take Kimi K2.6 for a spin
14.
How do agents see your website?(what-do-agents-see.runtype.app)
4 points by zackangelo 4mo ago|0 comments
15.
by zackangelo 5mo ago
Isn't Kimi K2.6 natively INT4?
16.
by zackangelo 5mo ago
I don’t think this is true across Blizzard. Overwatch is the best it’s ever been.
17.
by zackangelo 5mo ago
I give them a try about twice a year. I write a lot of Rust which should be squarely in their wheelhouse. This last time I was pleasantly surprised to find they mostly fixed their SSH remote editing support. But then it started truncating r
18.
by zackangelo 5mo ago
They are but the IDE needs to be integrated with them. Qwen specifically calls out FIM (“fill in the middle”) support on the model card and you can see it getting confused and posting the control tokens in the example here.
19.
by zackangelo 5mo ago
17b per token. So when you’re generating a single stream of text (“decoding”) 17b parameters are active. If you’re decoding multiple streams, it will be 17b per stream (some tokens will use the same expert, so there is some overlap). When t
20.
by zackangelo 8mo ago
This uses Nvidia’s CUDA snapshot API under the hood, but you have to pair it with a host side snapshot as well. Modal uses gVisor for this, which is notoriously high overhead. Does anyone know of a more efficient alternative if you’re runni
21.
by zackangelo 9mo ago
You’re right I misunderstood. I’m not sure if it would be of much utility because this would presumably be for tensor parallel workloads. In that case you want the ranks in your cluster to be uniform or else everything will be forced to run
22.
by zackangelo 9mo ago
Sparks are built for this and actually have Connect-X 7 NICs built in! You just need to get the SFPs for them. This means you can natively cluster them at 200Gbps.
23.
by zackangelo 9mo ago
No you use tensor parallelism in both cases. The way it typically works in an attention block is: smaller portions of the Q, K and V linear layers are assigned to each node and are processed independently. Attention, rope norm etc is run on
24.
by zackangelo 10mo ago
What 1T parameter base model have you seen from any of those labs?
25.
by zackangelo 11mo ago
Wouldn't you be able to test nccl if you had 2 of these?
26.
by zackangelo 11mo ago
Just a bit of feedback: > Instead of one brittle giant, we orchestrate a Mixture of Experts… “mixture of experts” is a specific term of art that describes an architectural detail of a type of transformer model. It’s definitely not using
27.
by zackangelo 1y ago
Because it depends on how much better “best” is. If it’s only incrementally better than open source models that have other advantages, why would you bother? OpenAI’s moat will only come from the products they built on top. Theoretically the
28.
by zackangelo 1y ago
Not quite a frontier model but definitely built by a frontier lab: Grok 2 was recently open sourced and I believe it uses a fairly standard MHA architecture with MoE.
29.
by zackangelo 1y ago
I feel a bit silly for not noticing this before. Over the last year or so I've often wondered when ssh added protocol-level support for session resume. I'd open my laptop on a new network and everything would be ready to go. But o
30.
by zackangelo 1y ago
Curious what issues you were having. The kernel should compile natively if you pass nvcc the correct arch flags, although it probably won't take advantage of any new hardware features.

More ›