ggerganov

6 ms·

1.
by ggerganov 1mo ago
Didn't expect this to pop up here - please ignore for now. We are preparing official semantic versioning of llama.cpp and it's almost ready, but not quite. More info about the versioning process is here: https://github.
2.
by ggerganov 3mo ago
Here are the prefill speeds: Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32109 MiB | model | size | params | backend | fa | test | t&#x
3.
by ggerganov 3mo ago
As a baseline, I run all models in Q8 [0] because I want to be confident that when I observe a problem, the root cause is not due to the quantization. However, in this specific case, I use Q8 on the mac and Q4 on the RTX machine because the
4.
by ggerganov 3mo ago
I haven't spent a dime on cloud inference, so cannot make a direct comparison like you. But I can 100% attest to the fact that Qwen3.6-27B is a very capable local model for coding tasks. Over the last month and a half I've been us
5.
by ggerganov 5mo ago
llama-batched-bench -hf ggml-org/Qwen3.6-27B-GGUF -npp 512,1024,2048,4096,8192,16384,32768 -ntg 128 -npl 1 -c 36000 M2 Ultra, Q8_0 | PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s |
6.
by ggerganov 6mo ago
Better keep the KV cache in full precision
7.
by ggerganov 11mo ago
Yes, I provided detailed numbers here: https://github.com/ggml-org/llama.cpp/discussions/16578
8.
by ggerganov 11mo ago
FYI you should have used llama.cpp to do the benchmarks. It performs almost 20x faster than ollama for the gpt-oss-120b model. Here are some samples results on my spark: ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GB10, c
9.
by ggerganov 1y ago
They should add "custom endpoint" support instead [0]. [0] https://github.com/microsoft/vscode/issues/249605
10.
by ggerganov 1y ago
Gemma 3n (the model used by this app) would run on any Apple Silicon device (even with 8GB RAM).
11.
by ggerganov 2y ago
The llama.cpp tools and examples download the models by default to a OS-specific cache folder [0]. We try to follow the HF standard (as discussed in the linked thread), though the layout of the llama.cpp cache is not the same atm. Not sure
12.
by ggerganov 2y ago
Yes, exactly. You can set --ctx-size to a smaller value if you know that you will not hit the limit of 32k - this will save you VRAM. To control how much global context to keep in the ring buffer (i.e. the context that is being reused to en
13.
by ggerganov 2y ago
The primary tricks for reducing the latency are around context reuse, meaning that the computed KV cache of tokens from previous requests is reused for new requests and thus computation is saved. To get high-quality completions, you need to
14.
by ggerganov 2y ago
Appreciate the feedback! Currently, there isn't a user-friendly way to disable the stats from showing apart from modifying the "'show_info': 0" value directly in the plugin implementation. These things will be impro
15.
by ggerganov 2y ago
There are 4 stopping criteria atm: - Generation time exceeded (configurable in the plugin config) - Number of tokens exceeded (not the case since you increased it) - Indentation - stops generating if the next line has shorter indent than th
16.
by ggerganov 2y ago
Yes, I think it is surprising that it works. I think a fairly large amount, though can't give a good number. I have been using Github Copilot from the very early days and with the release of Qwen Coder last year have fully switched to
17.
by ggerganov 2y ago
Hi HN, happy to see this here! I highly recommend to take a look at the technical details of the server implementation that enables large context usage with this plugin - I think it is interesting and has some cool ideas [0]. Also, the same
18.
by ggerganov 3y ago
So far is going great! Good community, having fun. Many ideas to explore :-)
19.
by ggerganov 3y ago
> Thanks to the amazing work of @ggerganov on llama.cpp which made this possible. If there is anything that you wish to exist in an ideal local AI app, I'd love to hear about it. The app looks great! Likewise, if you have any reques
20.
by ggerganov 3y ago
I've found lowering the temperature and disabling the repetition penalty can help [0]. My explanation is that the repetition penalty penalizes the end of sentences and sort of forces the generation to go on instead of stopping. [0] ht
21.
by ggerganov 3y ago
Yes, I was planning to do this back then, but other stuff came up. There are many different ways in which this simple example can be improved: - better detection of when speech ends (currently basic adaptive threshold) - use small LLM for q
22.
by ggerganov 3y ago
Heh, funny to see this popup here :) The performance on Apple Silicon should be much better today compared to what is shown in the video as whisper.cpp now runs fully on the GPU and there have been significant improvements in llama.cpp gene
23.
by ggerganov 3y ago
Yes, I assumed that checking the weights for presence and amount of outliers is not something that is usually done and effects on this can be overlooked. If my assumption is wrong and researchers do usually look at such metrics, then my que
24.
by ggerganov 3y ago
> I don't recall the details exactly, but I don't think it ever did very much. How would you have known if the trick actually reduces the outliers in the weights? Even if the transformer quality does not improve overall, having
25.
by ggerganov 3y ago
It was designed in #915 (read just the OP and the linked PRs at the end) and the implementation pretty much follows it closely, at least for the Metal backend. The CUDA and OpenCL backends are currently slightly coupled in ggml as they star
26.
by ggerganov 3y ago
My POV is that llama.cpp is primarily a playground for adding new features to the core ggml library and in the long run an interface for efficient LLM inference. The purpose of the examples in the repo is to demonstrate ways of how to use t
27.
by ggerganov 3y ago
I'm planning to write code and have fun!
28.
by ggerganov 3y ago
The weights are preprocessed into integer quants combined with scaling factors in various configurations (4, 5, 8-bits and recently more exotic 2, 3 and 6-bit quants). At runtime, we use efficient SIMD implementations to perform the matrix
29.
by ggerganov 3y ago
ggml started with focus on CPU inference, but lately we have been augmenting it with GPU support. Although still in development, it already has partial CUDA, OpenCL and Metal backend support
30.
by ggerganov 3y ago
For videos without subtitles one could chain Whisper to auto-generate transcripts, though that would require downloading the audio and processing it

More ›