anuarsh

6 ms·

1.
Show HN: Fine-tune Llama3-8B on 8GB GPU without quantization(github.com)
3 points by anuarsh 11mo ago|0 comments
2.
by anuarsh 1y ago
Good question, need to research this one
3.
by anuarsh 1y ago
There's one more exciting thing about Qwen3-next (except, efficient MoE architecture and fast linear attention) - MTP (Multi token prediction). It is the additional layer that allows generating more tokens without the need to go throu
4.
by anuarsh 1y ago
I haven't tested on Apple machines yet, but gpt-oss and qwen3-next should work I assume. Llama3 versions use cuda specific loading logic for speed boost, so it won't work for sure
5.
by anuarsh 1y ago
Thanks! I don't have much experience with diffusion models, but technically any multi-layer model could benefit from loading weights one by one
6.
by anuarsh 1y ago
1tok/2s is the best I got on my PC, thanks to MoE architecture of qwen3-next-80B. gpt-oss-20B is slower because I load all single layer experts to GPU and unpack weights (4bit -> bf16) each time. While with qwen3-next I load only ac
7.
by anuarsh 1y ago
CPU is much slower than GPU. You can actually use both by offloading some layers to CPU as o.offload_layers_to_cpu(layers_num=12). It is faster to load from RAM than from SSD.
8.
Show HN: Run Qwen3-Next-80B on 8GB GPU at 1tok/2s throughput(github.com)
123 points by anuarsh 1y ago|17 comments
9.
Show HN: Run gpt-oss-20b on 8GB GPUs(github.com)
6 points by anuarsh 1y ago|0 comments
10.
by anuarsh 1y ago
We are talking about 100k context here. 20k would be much faster, but you won't need KVCache offloading for it
11.
by anuarsh 1y ago
Absolutely, there are tons of cases where interactive experience is not required, but ability to process large context to get insights.
12.
by anuarsh 1y ago
Hi everyone, any comments or questions are appreciated
13.
Show HN: oLLM – LLM Inference for large-context tasks on consumer GPUs(github.com)
3 points by anuarsh 1y ago|7 comments