t-vi

6 ms·

1.
by t-vi 4mo ago
Not sure about the link, the official landscape URL seems to be this (same data probably): https://landscape.pytorch.org/
2.
by t-vi 4mo ago
We do try to check the pulse of projects there with a checklist, but I guess it started in a more uncoordinated way. If you file an issue here, I think it would work to update things: https://github.com/pytorch-fdn/ecos
3.
by t-vi 5mo ago
They have some pretty cool people, though, no reason not to think they'll catch up soon enough.
4.
The fiery, deadly crashes involving the Tesla Cybertruck(theguardian.com)
13 points by t-vi 6mo ago|4 comments
5.
by t-vi 10mo ago
- I don't think it hurts to learn PyTorch (and having learned JAX is good, too). I don't know if JAX + triton is as impossible as you make it out, but it seems that PyTorch integration is quite good for many things. - For pallas,
6.
by t-vi 10mo ago
Note that the NVIDIA container uses CUDA+cuBLAS 13.0.2 which cites "Improved performance on NVIDIA DGX Spark for FP16/BF16 and FP8 GEMMs", which seems to be your use-case. In general, I would suspect that it mostly comes to v
7.
by t-vi 11mo ago
It seems to me that in 2016 people did (have to) play a lot more tricks with the backpropagation than today. Back then it was common to meddle with gradients in between the gradient propagation. For example, Alex Graves's (great! with
8.
AMD and OpenAI Announce Strategic Partnership to Deploy 6 Gigawatts of AMD GPUs(amd.com)
4 points by t-vi 1y ago|1 comments
9.
The Buchstabenmuseum Berlin is closing(buchstabenmuseum.de)
204 points by t-vi 1y ago|69 comments
10.
by t-vi 1y ago
Note that PyTorch's kernels are somewhat generic in shape. It has always been relatively easy to get speedups by specializing the shape, e.g. Apache TVM had that (back before it was "Apache" even).
11.
by t-vi 2y ago
If you like JIT wrappers and Python interpreters: In Thunder[1], a PyTorch to Python JIT compiler for optimizing DL models, we are maintaining a bytecode interpreter covering 3.10-3.12 (and 3.13 soon) for our jit. That allows to run Python
12.
by t-vi 2y ago
When I did a similar thing (but with less LLM) I liked https://github.com/coqui-ai/TTS but back then I needed to cut out the conversion step from tensor to a list of numbers to make it work really nicely.
13.
by t-vi 3y ago
The subtraction is because "is an example of constructing the “Inner Product” distance" per the text above it. That ymmone might not be one could be because they only need that up to a constant and so don't care, but it'
14.
by t-vi 3y ago
> Is avoiding CF potentially just a matter of sheer scale ? My intuition would be that you get more orthogonal directions to the gradient (of previous samples) if you have larger model.
15.
by t-vi 3y ago
After the first epoch, the average time since the present data item was last used for during training is small at the beginning of an epoch grows during the epoch. I'd expect that to positively relate to loss on the present iteration.
16.
by t-vi 3y ago
My former neighbors run https://justanotherfoundry.com/ and I like their work and bought some.
17.
by t-vi 3y ago
To be honest, I never get what people want with all that business and wonder if it is because the abstraction ("ordered derivatives") implied is not ideal. If we follow the ordinary chain rule (for a single coordinate if you want)
18.
by t-vi 3y ago
The PEP says that a naïve implementation would cause a 4% performance hit and have a list of various ways of speeding up cases where immortality makes sense / is guaranteed that they say can bring this to parity. Maybe they implemented
19.
by t-vi 3y ago
This special casing just had not been implemented yet. But as it is an interesting optimization, more so with multi-interpreter or no-GIL Python, the developers will actually introduce immortal objects in Python 3.12 to avoid counting refer
20.
by t-vi 3y ago
Not saying that the vector space bit isn't neat, but it's called functional analysis because you can take limits of various forms and define (semi-) continuity, have completions of spaces, and all that has nice properties. So to
21.
by t-vi 3y ago
It knows nothing of the modern stuff (because MacKay died too early), but skipping the first parts of David MacKay: Information Theory, Inference, and Learning Algorithms you get a very accessible course in (200x) Bayesian Inference that sh
22.
by t-vi 3y ago
The logic error seems to be that you have the second for loop where you only want to destructure the splitted line into a tuple. Now writing that decomposition as a for loop over a one-element list (as in the corrected code) works, but seem
23.
by t-vi 3y ago
I use Debian's system Python 3.10 install for most of my stuff and it works really well for me. Some things I install via pip but the key libraries (e.g. PyTorch) from source.
24.
by t-vi 3y ago
There is A. Karpathy's recipe for training NNs but it is not a walkthrough with an example: https://karpathy.github.io/2019/04/25/recipe/ but the general idea of "get something that can overfit
25.
by t-vi 3y ago
The adapter and LoRa have a drastically fewer parameters, so one might expect that forward + backward is roughly 2x the cost of forward. Then (as far as I know), in contrast to generation, training is done on the entire output of the transf
26.
by t-vi 3y ago
Out of curiosity, how do you handle things where the output shape is input dependent (as opposed to only dependent on input shapes)? This is from `torch.sum(tensor, dim)` where dim might be nonconstant to `torch.nonzero(x)` and of course ad
27.
by t-vi 3y ago
On the other hand, it would fix cases where there are somewhat rare hangs that are hard to reproduce, for example in multithreaded applications embedding Python. A while ago, I hit a deadlock in LibreOffice's Python extension system. I
28.
by t-vi 3y ago
Both: A long while ago, I wrote a little tutorial[0] on quantizing a speech commands network to the Raspberry. I used that to control lights directly and also for wake word detection. More recently, I found that I can just use more classic
29.
by t-vi 3y ago
Personally, I plugged a Jabra conference speaker to a Raspberry and if it hears something interesting, it sends to my local GPU computer for decoding (with whisper) + answer-getting + response sent back to the Raspberry as audio (with a mod
30.
by t-vi 3y ago
Not to forget CPython's own faster-cpython project which aims at JIT compiling. [0] Also JAX[1], PyTorch[2] come with JIT compilation specifically aimed at GPU kernels "fusing" multiple higher-level operation And NumPy/S

More ›