Y
HN Search
Hacker News Search
new
|
comments
|
top
|
jobs
hn_c
searching Neon…
1.
▲
2.
▲
3.
▲
4.
▲
5.
▲
6.
▲
6 ms
·
1.
▲
by
hn_c
2mo ago
> For decode, (...) it is difficult to batch multiple output tokens together. I think it's the other way around? The GPU has to stream gigabytes of active layer weights to compute the next token, so having a batch of next-token pred