7 ms·
this is cool but like, are we just vibe coding NAND burners at this point? these decode times don't really tell the whole story, because prefill becomes the bo
by brrrrrm 1mo ago
this is cool but like, are we just vibe coding NAND burners at this point? these decode times don't really tell the whole story, because prefill becomes the bottleneck.
half an hour to process 10k tokens on an M5 seems... not great
- kennywinker 1mo agoNot great for coding, or realtime agent interactions. But for background processing tasks overnight? Seems like it’d work pretty well
- selcuka 1mo agoI'm pretty sure one can rent a GPU for a few minutes with the electricity cost of leaving an M5 overnight.
- hdgvhicv 1mo agoDomestic electricity is free nowadays, certainly for most of the year, as solar plus battery covers your usage for a tiny percentage of the cost of your house.
- wccrawford 1mo agoOnly if you don't count the cost of the equipment and installation.
- hdgvhicv 1mo agoOr the cost of the house. Given the cost of a building is far more than the cost of generating enough power for that building it doesn’t really matter
- kennywinker 1mo agoSure. One could. But then one wouldn’t be in control of every step of the process.
- fsuts 1mo agoThis is how progress happens, someone gets to 3t/s, the next person gets to6/s and eventually we get to 100t/s. People like this person are laying the foundations.
- IsTom 1mo agoThere is a limit how much you can squeeze out of given hardware. Betting on it being closer to 100t/s than 6t/s is only that, a bet.
- rhdunn 1mo agoOn my 4090 setup I'm getting 86t/s on a 12B Q6_K quantized model running entirely in VRAM. The current GPUs are optimized for processing huge numbers of triangles per second. There are three things at play here: 1. the organization of the data being sent to the GPU to optimize throughput; 2. the speed at which the GPU can read that data from its VRAM; 3. how many triangles it can process in parallel by using individual compute units. I suspect that given parallel improvements for neural networks, we'll see similar improvements: 1. optimizing the structure of the weights in the model for efficient access by the CPU/GPU/NPU/TPU; 2. efficient access of data strides (matrix rows) in the memory, e.g. being able to read multiple 2x2 matrix values in one clock cycle, or stepwise pairs of values (a(i,j), b(j,k)) needed for matrix multiplication; 3. parallel compute for matrix and tensor multiplication and other operations needed by neural networks.
- IsTom 1mo agoIf you've got X GB of weights in slow access memory (be it RAM vs VRAM or SSD vs RAM) and Y GB of fast memory then no matter what, if you want to use them you'll need to transfer X-Y GB and will be bound by memory throughput. You can try to reduce number of activated weights, but how much can be gained that way is speculative so far.
- leonickson 1mo agoagree, prefill is the weak spot right now. it goes through the same per-token path as decode, which is dumb for long prompts. The fix is on the list: during prefill we can batch the expert reads for the whole prompt per layer instead of per token, that amortizes the IO a lot. until that lands, honest answer is this is good for chat-length stuff, not for feeding it a 10k token document.
- piyh 1mo agoSSD NAND reads are nearly infinite. Still makes me uncomfortable, but writing is what kills. There's a reason SSDs are rated by TBW, not TBR.