ffast-math

4 ms·

1.
Twitter's surprisingly easy path to profitability(dblalock.substack.com)
2 points by ffast-math 4y ago|0 comments
2.
Farewell, CUDA OOM: Automatic Gradient Accumulation(mosaicml.com)
4 points by ffast-math 4y ago|0 comments
3.
by ffast-math 4y ago
I named it this in 2017 and was only worried about name collisions with other GitHub repos and ML algorithms. Also it's a backronym for Based On Lookup Tables + sounds at least somewhat evocative of going fast, so it was the best name
4.
by ffast-math 4y ago
Nope. I'd love to though.
5.
by ffast-math 4y ago
This master's thesis sort of does it for individual layers, but it doesn't have any fine-tuning yet so it completely wrecks the accuracy: https://github.com/joennlae/halutmatmul . If someone worked on contribu
6.
by ffast-math 4y ago
No ML frameworks implement it yet, though I'd be happy to work with people from the PyTorch/TF/JAX/CUDNN/CUTLASS/etc. teams (or volunteers) if anyone wants to make this happen. Also, while you can get 200x co
7.
by ffast-math 4y ago
IMO it would be super cool and I hope someone does it. There are a lot of interesting tradeoffs around which techniques to use for which matrix sizes and under which assumptions about read vs write ratios, what you have a training set for,
8.
by ffast-math 4y ago
email. <my first name>@mosaicml.com
9.
by ffast-math 4y ago
Thanks for posting it! It should be possible to get large speedups on CPUs, but the trick will be gradually approximating each of the layers in the model (see my reply to sibling comment). It's not conceptually difficult, but will requ
10.
by ffast-math 4y ago
Yes. It's another research project to make this happen, but I think it would be fairly straightforward. The issue is that you can't backprop through the assignment step, so you get no gradient with respect to the input. This manda
11.
by ffast-math 4y ago
We found sparse, truncated PCA to be the most competitive baseline. We beat it by a lot (see the paper [1]), but the other big drawback is that trading off the rank vs sparsity was an ugly hyperparameter tuning problem. By ugly, I mean that
12.
by ffast-math 4y ago
Exactly. You can run it on sparse inputs. It's just that our implementation doesn't exploit the sparsity, so we don't claim that it will work better.
13.
by ffast-math 4y ago
Definitely. On CPUs, you could make this 2x faster pretty easily with just another execution port for vpshufb / vtbl and a 4bit lo and hi unpack instruction. Though the real speedup would be allowing dense matmul ASICs to operate on 16
14.
by ffast-math 4y ago
There's definitely a tradeoff between speed and accuracy. We characterize this for various problems in the paper ( https://arxiv.org/pdf/2106.10860.pdf ), but tl;dr is that it speeds things up more at a given level
15.
by ffast-math 4y ago
Author here. Ask me anything--happy to answer questions. Also, if you like this kind of work, you might like what I've been building for the past year: Composer [1]. It speeds up neural net training by a lot (e.g., 7x faster for ResNet
16.
by ffast-math 5y ago
I think we only claim to be able to preprocess a matrix at "up to" 100GB/s/core. The overall matrix product will take longer and depend on the matrix shapes. To simplify Section 1.1, we help when: 1) You need to perfor
17.
by ffast-math 5y ago
We have a generalization guarantee in Section 4.5. It's not especially tight though; in practice, the errors from different codebooks tend to be mostly independent, and you get nice Gaussian-like concentration. I would look at the empi
18.
by ffast-math 5y ago
So this misses a few aspects of why the method works: - You can't actually get a speedup from the proposed approach. You'd need a lookup table of size b^2 to multiply two b-bit numbers, which will be much slower than just doing th
19.
by ffast-math 5y ago
I'm actually not quite sure what you mean by breaking down into two-dimensional operations. We use operations on pairs of vectors, but nothing is assumed to be two-dimensional, and I don't think we suggest imagining anything as a
20.
by ffast-math 5y ago
You could still optimize the prototypes, so fine-tuning with this in place would be possible (see, e.g., [1]). But we don't yet have data on how well this would work using our exact method, how early in training you could do the op rep
21.
by ffast-math 5y ago
Yes, basically correct. A couple notes/clarifications for other readers: - The rows a of A are "random," but in the sense of being drawn from some distribution for which we have a training set--not in the sense of, e.g., &quo
22.
by ffast-math 5y ago
We compared to several frequent directions variants, Fast Johnson–Lindenstrauss, some other hashing-based methods, and a bunch of other approximate matrix multiplication approaches. We had to omit some of them from the results section thoug
23.
by ffast-math 5y ago
There's been some work on doing adds instead of multiplies (e.g., https://arxiv.org/abs/2012.03458 ). And I think float8 will roughly be doing this under the hood. Personally, I'm not sure whether this is the
24.
by ffast-math 5y ago
Great observations. I see this paper as the first in a three-part series. The second part is specializing it for convolution (which has additional structure to exploit), and the third is hooking these approximate ops into deep neural nets t
25.
by ffast-math 5y ago
Primary author here. Happy to answer questions! Also, feel free to email me at the address in the paper if you're interested in talking about it in more detail. E.g., I've already heard from some hardware folks looking at expandin
26.
by ffast-math 7y ago
I'm a big fan of ann-benchmarks and will be the first to tell you that the research community needs way more benchmarks like this. But I do want to add a couple caveats about it for people looking into this area: 1) Most of these datas
27.
by ffast-math 7y ago
> "As each vector takes 2 KB space, the minimum storage space for 100 million vectors is about 200 GB" Why are you not quantizing the vectors when you insert them? Bolt [1] and Quicker-ADC [2] make 10-100x compression basically
28.
by ffast-math 7y ago
My experience attempting to optimize information intake: -Many books are no better than podcasts or articles, and only leave you with a handful of takeaways. -But the best books are fantastic and well worth the time/money.[1] -Audioboo