chillee

6 ms·

1.
by chillee 6mo ago
I'd also note that you can more or less write the same code in PyTorch with torch.vmap
2.
by chillee 8mo ago
this is almost certainly not being done on cerebras
3.
by chillee 10mo ago
I think unlike Gluon/CuTe/ThunderKittens (which distinguish themselves from Triton by being lower level giving you more control, thus being less performance portable and harder to write), Helion distinguishes itself from Triton by
4.
by chillee 10mo ago
What's the point of Triton compared to Gluon? What's the point of PyTorch compared to Triton? One of the main values of Triton is that it significantly expanded the scope of folks who can write kernels - I think Helion could expan
5.
by chillee 10mo ago
If you think of Triton as a "baseline", most other DSLs are lower-level than Triton, whereas this is higher-level.
6.
by chillee 1y ago
Their gross profits are very high even though they're not making operating profit.
7.
by chillee 1y ago
Clearly not true anymore given OpenAI and Anthropic's revenue growth.
8.
by chillee 1y ago
The 32 parallel sequences is also arbitrary and significantly changes your conclusions. For example, if they run with 256 parallel sequences then that would result in a 8x cheaper factor in your calculations for both prefill and decode. The
9.
by chillee 1y ago
No. In some sense, the article comes to the right conclusion haha. But it's probably >100x off on its central premise about output tokens costing more than input.
10.
by chillee 1y ago
This article's math is wrong on many fundamental levels. One of the most obvious ones is that prefill is nowhere near bandwidth bound. If you compute out the MFU the author gets it's 1.44 million input tokens per second * 37 bil
11.
by chillee 1y ago
I mean, vllm and sglang are both "pure python" essentially as well. But yeah, in ML you rarely require C++ to get good performance for most of the systems people are writing.
12.
by chillee 2y ago
A couple things: 1. The academy has had a significant increase of young voters in the past 10 years or so. Generally speaking, young voters are more likely to take animation as a "serious" medium. 2. These interviews were always s
13.
by chillee 2y ago
https://www.theinformation.com/articles/to-reduce-ai-costs-g... Which takes it from > Broadcom generates a 70% profit margin from its work on TPUs, said a person with direct knowledge of the internal analysis. SemiA
14.
by chillee 2y ago
One of the big things this article misses is that Google pays Broadcom a significant amount for the actual chip design, also around a 70% margin. Google certainly has infra/cost advantages, but it's nowhere near 10x.
15.
by chillee 2y ago
For latency-bound inference (i.e. one request) you don't need tensor-cores since all your operations are just matrix vector multiplications.
16.
by chillee 2y ago
The big issue with Strassen isn't performance - it's numerical stability.
17.
by chillee 2y ago
Well, that'll help with a lot :) But dealing with wave quantization requires dimensions that aren't neceessarily a multiple of 2, and often are a multiple of the number of SMs on a GPU (i.e. 132 on an H100)
18.
by chillee 2y ago
It's very much in the "worse is better" camp.
19.
by chillee 2y ago
It's always fun when people use autodiff in packages like PyTorch for completely unrelated usecases :)
20.
by chillee 2y ago
> it wouldn’t use functions like these and the generated comparable code would be on-pare performance wise Perhaps if XLA generated all functions from scratch, this would be more compelling. But XLA relies very heavily on pattern-matchin
21.
by chillee 2y ago
To be clear, this performance is quite bad (presumably because you didn't manage to get compilation working). You're getting 35 tokens/s for a 405B model, which comes out to about 85 Teraflops. 8 MI300x GPUs comes out to 10.4
22.
by chillee 2y ago
> And was trying to make a broader point about the lack of transparency (in performance, lower-level impl) in PyTorch when running on NVIDIA vs. non-NVIDIA hardware. I don't quite understand this argument. Lack of transparency from
23.
by chillee 2y ago
I believe it should run on V100 as well (although definitely not tested as well), and an user reported that they got it running on T4 too.
24.
by chillee 2y ago
Ah sorry, should have put that in the blog post. This leverages Triton heavily, so it'll only work on machines that have Triton backends (at least, we've tested on Nvidia and AMD GPUs)
25.
by chillee 2y ago
In my (very amateurish) testing, I think the performance seemed pretty comparable (for non-dilated natten). I need to do some proper benchmarking though!
26.
by chillee 2y ago
These benchmarks are on Ampere, where FA3 has no performance benefits over FA2. On Hopper, FlexAttention is currently about 80% of FlashAttention3's performance (about 500 TFLOPs peak)
27.
by chillee 2y ago
https://x.com/chhillee/status/1821253769147118004?s=46 Perhaps this tweet thread would be better.
28.
by chillee 2y ago
I would agree with this. For example, how would you represent causal attention in the standard equation?
29.
by chillee 2y ago
Hi, one of the authors of this blog post (Horace He), along with Driss Guessous, Yanbo Liang, and Joy Dong. We’re quite happy with this abstraction - happy to answer any questions about it!
30.
by chillee 2y ago
Where are you seeing these benchmarks?

More ›