formalsystem

8 ms·

1.
by formalsystem 1y ago
I work on PyTorch and there are many things that make me suspicious about these results. My TL;DR is unless we get a zip file of all the kernels with how they're benchmarked results like this are almost impossible to verify 1. I don&#x
2.
Torch.load flipping default to weights_only=True(dev-discuss.pytorch.org)
2 points by formalsystem 2y ago|0 comments
3.
by formalsystem 2y ago
The project is very much focused on maxing out tensor cores and since older GPUs don’t have them it’s not where the project shines best
4.
by formalsystem 2y ago
Please ignore my previous comments - I double checked with the model developers and here's the correction. Vanilla PTQ means no fancy quantization algorithm like SpinQuant, AWQ, etc. was applied. It just applied the same quantization s
5.
by formalsystem 2y ago
You can estimate context length impact by doing back of the envelope calculations on KV cache size: 2 * layers * attention heads * head_dim * byte_per_element * batch_size * sequence_length Some pretty charts here https://github.
6.
by formalsystem 2y ago
The issue here is memory in PyTorch is byte addressable and that's a limitation we can't solve without making a lot more changes to PyTorch. But in your specific case, if you'd like to pack more data into `values` you can use
7.
by formalsystem 2y ago
Not that I know of for this study, at least for the specific scope torchao we want to make it easier for researchers to create new quantization algorithms in python and have those algorithms run fast and you can see a lot of those algorithm
8.
by formalsystem 2y ago
The naming is unfortunate but in this blog QLoRA is referring to Quantization-Aware Training with LoRA adaptor
9.
by formalsystem 2y ago
My wife calls it torch AAAW
10.
by formalsystem 2y ago
So this should be referring to w8a8 (weights and activations in 8 bit) So this is gonna be 8 bit weights, 8 bit activations, group size of 256, symmetric quantization. Not sure how to map this to the GGUF variants because they don't me
11.
by formalsystem 2y ago
It's particularly useful in memory bound workflows like batch size = 1 LLM inference where you're bottlenecked by how quickly you can send weights to your GPU. This is why at least in torchao we strongly recommend people try out i
12.
by formalsystem 2y ago
Hi I'm Mark I work on torchao which was used for the quantization aware training and ARM kernels in this blog. If you have any questions about quantization or performance more generally feel free to let me know!
13.
by formalsystem 2y ago
It's a great question! Int4 is an easy one to understand. PyTorch supports int8 but not int4 so what you can do is "pack" 2 int4 values into a single int8 value. You still get speedups even without hardware support because yo
14.
by formalsystem 2y ago
we have experimental support for float4 training with the mx formats https://github.com/pytorch/ao/tree/main/torchao/prototype/mx... But that's waiting for Blackwell to be released so we g
15.
by formalsystem 2y ago
yeah indeed choice of language might not be ideal, it seems like 2x language is clearest to folks? I can make some quick edits to the article
16.
by formalsystem 2y ago
It's both! For this blog we decided to discuss our best end user facing numbers to keep things simple. We briefly hint at our contributor guide here https://github.com/pytorch/ao/issues/391 which does a
17.
by formalsystem 2y ago
There's different tradeoffs, spinning up a separate repo is what we call "out of core" vs having everything in PyTorch "in core" Basically PyTorch is a large library where CI takes a long time to run which means mer
18.
by formalsystem 2y ago
Mostly comes down to what's fastest to develop, it's faster to write a few custom kernels than it is to develop a new compiler backend Granted after more upfront effort compilers are just such a significant UX boost that indeed yo
19.
by formalsystem 2y ago
There's a bunch of overhead associated with PTQ - but TL;DR is that much of that overhead goes away when you're using `torch.compile()` and `torchao.autoquant()` Essentially the latency overhead comes from quantizing and dequantiz
20.
by formalsystem 2y ago
Most of our performance relies on leveraging torch.compile which generates Triton kernels which run fast on CPU and GPU but not MPS since Triton does not support generating Metal kernels. So you lose the nice story of writing low bit code i
21.
by formalsystem 2y ago
Hi! I'm Mark from the PyTorch team at Meta and work on torchao. If you have any questions about the library or really anything at all about performance, don't hesitate to ask!
22.
The Meta Hacker Cup AI Track(codeforces.com)
1 points by formalsystem 2y ago|0 comments
23.
by formalsystem 3y ago
nvtop or nvidia-smi gives you a good macro overview but I personally have found that utilization (EDIT: As reported by nvidia-smi) is actually a poor proxy for how fast your workload can be outside of just ensuring that a GPU is indeed bein
24.
by formalsystem 3y ago
This was a lot of fun to read, really enjoyed the journey from game design to product design to ML. One thing I was hoping to ask was how come you felt the need to procedurally generate levels? I've often heard debates around whether c
25.
Working Class Deep Learner(marksaroufim.substack.com)
1 points by formalsystem 5y ago|0 comments
26.
I refuse to let Amazon define Rust(twitter.com)
1193 points by formalsystem 5y ago|338 comments
27.
Dell is cancelling Alienware gaming PC shipments to several US states(pcgamer.com)
23 points by formalsystem 5y ago|4 comments
28.
Wild West of MLOps(marksaroufim.substack.com)
3 points by formalsystem 5y ago|0 comments
29.
Wild West of MLOps(marksaroufim.substack.com)
3 points by formalsystem 5y ago|0 comments
30.
Dump the MBA – Gaming as Management Training(marksaroufim.substack.com)
2 points by formalsystem 5y ago|0 comments

More ›