jafioti

7 ms·

1.
Producing The Perfect Token(blog.luminal.com)
1 points by jafioti 5mo ago|0 comments
2.
by jafioti 6mo ago
i did a quick scroll and was happy to see a long-ish article on XLA and TPUs. then i realized it was literally just "using vmap for parallel loops is better than fori", but in massively wordy claudisms.
3.
by jafioti 8mo ago
That's a bit trite tbh. We all know of these techniques, but actually implementing them on GPUs in a low-overhead manner that maintains the model's fidelity is challenging. It's much more than just breaking out the old CS boo
4.
by jafioti 8mo ago
Thats reasonably accurate, we're fusing both pre-defined operations as well as codegenned operations. Block-level operations live inside the search space, as do kernel, warp and thread level operations. Since it's a unified search
5.
Compiling models to megakernels(blog.luminal.com)
35 points by jafioti 8mo ago|19 comments
6.
by jafioti 1y ago
could be...im not opposed to looking into this to see if there's no possible trajectory from naive to strassen's without leaving logical equivalency. all the optimizations for matmul so far have been straightforward trajectories f
7.
by jafioti 1y ago
mcts / rl isn't really a heuristic. but yes heuristics can be used temporarily to keep the search space small, and removed over time as the search algorithm improves.
8.
by jafioti 1y ago
np-hard is still solveable with constraints. look at go.
9.
by jafioti 1y ago
you suppose correctly ;)
10.
by jafioti 1y ago
yep, parallelized profiling across many devices is definitely something i want to add.
11.
by jafioti 1y ago
hopefully! i dont know the exact trick they used, but the idea is to design the search space such that that trick is discoverable.
12.
by jafioti 1y ago
yup! we build a search space by iteratively applying rewrite rules in every possible order (using e-graphs to do this efficiently). the rewrites alter stuff like looping / tiling structures, as well as algebraic rewrites like softmax t
13.
by jafioti 1y ago
e-graphs are awesome! none of this would be possible without them.
14.
by jafioti 1y ago
we're working on techniques like mcts and RL (e.g. AlphaGo) to manage the search space, but you'd be suprised how far you can get if you carefully design the search space to prevent explosions.
15.
by jafioti 1y ago
ah i see the confusion. we do common subexpression elimination of the terms in the search space (which allows single application of rewrites to apply to many repeat patterns) but the search can choose to re-use patterns of terms when we ext
16.
by jafioti 1y ago
very similar to superoptimisation, but most superoptimisers try to tackle turing-complete code. by just doing a very limited space of computation (linear algebra with 12 primitive ops) the search remains tractable. the search space is desig
17.
by jafioti 1y ago
we're just optimizing linear algebra, which is mostly made up of patterns of simple ops. for instance, matmul is just broadcasted multiply -> sum reduce. the search does common subexpression elimination by default. if two patterns a
18.
by jafioti 1y ago
basically autotuning on steroids. instead of searching single dimensions of optimization (tile sizing, etc.) we search through full algebraic rewrites (like rewriting softmax to online softmax) and various loop / tiling structures in
19.
by jafioti 1y ago
a lot of the ideas luminal is built on are here: https://arxiv.org/abs/2304.04332
20.
by jafioti 1y ago
yep! currently we're emitting cuda / metal but once the search is better, i want to directly emit ptx / low-level asm on other hardwares.
21.
by jafioti 1y ago
a lot of the search is still being optimized so we don't match super hand-optimized kernels like llama.cpp has, so we def don't match their tps yet, but i want to make a perf tracking page to see improvements over time and prevent
22.
by jafioti 1y ago
we do support tensor cores, but the ops are only part of the search space, so there's virtually no overhead for them. the frontend and main ir is only 12 ops, and we can add hardware-specific ops in to the search space and only add in
23.
Show HN: Luminal – Open-source, search-based GPU compiler(github.com)
153 points by jafioti 1y ago|60 comments
24.
Bitter Lesson GPU Kernel Search in Luminal(twitter.com)
1 points by jafioti 1y ago|0 comments
25.
Luminal: Fast ML in Rust through graph compilation(github.com)
7 points by jafioti 3y ago|1 comments
26.
by jafioti 3y ago
Hi everyone, I've been working on luminal for a while now. Luminal is a deep learning library that uses composable compilers to achieve high performance. Current ML libraries tend to be large and complex because they try to map high le
27.
by jafioti 5y ago
As to what models are used, we use a combination of GPT-3 and in-house models.
28.
by jafioti 5y ago
I provided the demo with some information about common cases, but if you ask more complex questions, it should be able to follow along, even without preprogrammed responses. The bot is able to both use information given to it to answer ques
29.
Show HN: GPT-3 Powered Customer Service(sidekickai.co)
1 points by jafioti 5y ago|3 comments