ngxson

6 ms·

1.
The False Sense of Productivity(blog.ngxson.com)
1 points by ngxson 3mo ago|0 comments
2.
by ngxson 3mo ago
I do use both DSv4 the "normal" and the flash variant, non-locally. It works well, not exceptionally. And while it's cheap, I'd say that the difference between $1 per month vs $5 per month is not a big concern to me. IMO
3.
by ngxson 3mo ago
The dsv4 flash is 158B params in total. It is possible to run locally but will require all my system RAM. Also, a lot of my day-to-day tasks perform the same on both small and bigger models: summarize a web page, draft a response, translati
4.
by ngxson 3mo ago
My 2c: I think the "cloud vs local" debate is (maybe) a false dichotomy. In my experience, I use a hybrid approach and I've seen a huge productivity boost from it. The cloud-based models are fine for big and complex tasks, bu
5.
by ngxson 1y ago
Ahhh no sorry I forgot that the actual code controlling this is inside llama-model.cpp ; sorry for the misinfo, the -ngl only set to max by default if you're using Metal backend (See the code in side llama_model_default_params())
6.
by ngxson 1y ago
Ohhh nice to know! I was pretty sure that someone already tried to fix the chat template haha, but because we also allow users to freely create their quants via the GGUF-my-repo space, I have to fix the quants produces from that source
7.
by ngxson 1y ago
We also support SmolVLM series which delivers light-speed response thanks to its mini size! This is perfect for real-time home video surveillance system. That's one of the ideas for my next hobby project! llama-server -hf ggml-or
8.
by ngxson 1y ago
Ah no I mean we can omit the whole "-ngl N" argument for now, as it is internally set to -1 by default in CPP code (instead of being 0 traditionally), and -1 meaning offload everything to GPU I have no idea how to specify custom l
9.
by ngxson 1y ago
And btw, -ngl is automatically set to max value now, you don't need to -ngl 99 anymore! Edit: sorry this is only true on Metal. For CUDA or other GPU backends, you still need to manually specify -ngl
10.
by ngxson 1y ago
Thanks Daniel! Kudos for your great work on quantization, I use the Mistral Small IQ2_M from unsloth during development and it works very well!!
11.
by ngxson 1y ago
For brew users, you can specify --HEAD when installing the package. This way, brew will automatically build the latest master branch. Btw, the brew version will be updated in the next few hours, so after that you will be able to simply &quo
12.
by ngxson 1y ago
Two things: 1. Because the support in llama.cpp is horizontal integrated within ggml ecosystem, we can optimize it to run even faster than ollama. For example, pixtral/mistral small 3.1 model has some 2D-RoPE trick that use less memory
13.
by ngxson 2y ago
Hi I'm Xuan-Son, Small correct, I'm not just asking it to convert ARM NEON to SIMD, but for the function handling q6_K_q8_K, I asked it to reinvent a new approach (without giving it any prior examples). The reason I did that was b
14.
Software U2F with Fingerprint (On Linux)(github.com)
2 points by ngxson 3y ago|1 comments
15.
by ngxson 3y ago
This project aims to support U2F / FIDO2 using fingerprint reader on Linux (via libfprint). The goal is to have the same user experience with 2FA using Windows Hello. This project is based on https://github.com/danstine