sleepyeldrazi

14 ms·

1.
by sleepyeldrazi 1mo ago
2 things, 1st: Alibaba's official endpoint pricing. they don't want to undercut too much as there is profit to be made to be close to it but not too low 2nd, and maybe more importantly: KV is not as efficient (vram usage-wise) as
2.
by sleepyeldrazi 1mo ago
yeah, I was confused during the whole thing, i get 70 t/s on a 3090, which evens out around 50 t/s at 128k+ , have been running 3.6 and now 3.8 (both iq4_nl at 256k q4 kv) on the 3090 for months. I am confused as to what we '
3.
by sleepyeldrazi 1mo ago
MTP is lossless in the sense that running a model with and without (at temp=0, meaning no randomness) will produce identical results. It's true that with enough samples across domains and runs with MTP it should even out around concre
4.
by sleepyeldrazi 1mo ago
Ran a quick test so that we both have accurate numbers, without MTP* at 10k ctx 27B hovers around 42 ts in llama.cpp, 35B around 135 ts. So not the 8x I assumed, just over 3x, but thats still a big difference. For the sake of testing I turn
5.
by sleepyeldrazi 1mo ago
Speed is (for the most part) active-parameter based, so a 30B-A3B model is roughly 10x the speed of a dense 30B (realistically closer to 8x) in the case when both fit. That's the proposition of MoE and why everyone is trying to make ma
6.
by sleepyeldrazi 1mo ago
loading the model would be similar vram footprint, correct, however the size of KV is based on 'active' params, not total params. So while at 1k ctx both will be in the same ballpark vram footprint-wise, at 100k the story will be
7.
by sleepyeldrazi 1mo ago
Not by the looks of it, but it got me thinking, currently in the middle of Level1Techs coverage on the model and switchyard and he mentions "how easy it is to customize it". Fully admitting that I haven't yet read the docs, m
8.
by sleepyeldrazi 3mo ago
I got it off kleinanzeigen, its a ebay-like site (but mostly 'pick it up yourself' instead of delivery). Looking at it right now, i do see multiple sales for 850-900. I did spot the 750 one after frequenting the site for a week or
9.
by sleepyeldrazi 3mo ago
I can't speak for the US, but in Germany (where hardware is usually more expensive, not less), I got my 3090 3 months ago for 750 euro and have been running the iq4_nl 27B using q4 kv (which after recent patches in llama.cpp is in my x
10.
by sleepyeldrazi 3mo ago
I need to ask, since I have desperately wanted to make Gemma 4 12B work, but im not sure if its the quant (i usually up it to q8, which is a lot higher than iq4_nl that i use for 3.6 27B) or the model itself, but it just starts confusing it
11.
by sleepyeldrazi 3mo ago
I've been running it almost since launch on a 3090 (24gb vram), you really don't need that much. Second hand those are really cheap and i get 50-70 t/s (with MTP at 2), full ctx. IQ4_NL (unsloth) on this model seems suspiciou
12.
DeepSeek V4 Flash optimized framework and model variants for DGX Spark(github.com)
3 points by sleepyeldrazi 3mo ago|1 comments
13.
by sleepyeldrazi 3mo ago
Inspired by [ https://github.com/antirez/ds4 ](ds4), [ https://github.com/CerebrasResearch/reap ](REAP) and [ https://huggingface.co/0xSero/DeepSeek-V4-Flash-162B ](OxSero's D
14.
by sleepyeldrazi 3mo ago
That's why I like qwen3.6 27B, it has 0 ego, it knows that it doesn't have complete world knowledge, so when it sees a web_search tool it searches all the time. Even qwen3.5 9B is mostly search-eager (but given the size, it&#x27
15.
by sleepyeldrazi 3mo ago
Opus also has a deeply ingrained personality that always de-rails sneakily into what it's taught, not what the user intends. This is good if the user doesn't know the details of the work they need performed and a huge time waste w
16.
by sleepyeldrazi 4mo ago
Have you tested Qwen3.6 35B? Putting aside the capability claims for that model (which I support, but are not my point here), that 35B has smaller active parameter count than the gemma 4 26B, potentially making both prefill and decode faste
17.
Ask HN: Local model experiences with 'high-reasoning distill' finetunes
2 points by sleepyeldrazi 4mo ago|0 comments
18.
by sleepyeldrazi 4mo ago
Finetuning takes little resources, the base model training is the slow and expensive part. Architecturally 3.5 models are identical to their 3.6 counterparts, that is why there is a consensus that those are probably finetunes and not re-tra
19.
by sleepyeldrazi 4mo ago
The best thing I have come up with is just make a bunch of prompts / tasks that I personally care about and need a model to know how to do. As an example, when qwen3.6 27B dropped, I ran it, kimi, claude and glm 5/5.1 on a bunch o
20.
by sleepyeldrazi 4mo ago
I don't think I can handle another small model release by qwen, I'm still trying to find the limits of 3.6 27B and they are already threatening us with a new one? But jokes aside, I love the fast iteration, these are most probably
21.
by sleepyeldrazi 4mo ago
I feel like if I had the infrastructure and saw that there is a huge interest in the model, i'd just undercut alibaba's prices a little harder to grab all the consumers. I am sure that the providers have done the math and found th
22.
by sleepyeldrazi 4mo ago
If you want a good dense model, use qwen3.6 27B instead, speed will be up, and if you don't take my word for it being smarter, take openrouter's prices of it against the bigger, slower and less memory-efficient gemma do the talkin
23.
by sleepyeldrazi 4mo ago
It is actually very exciting that they are also working on 3.5, I will keep this toy project up in the meantime, trying it out and testing things around it helps me learn a bunch. As for the treating them as a block idea, that was my initia
24.
by sleepyeldrazi 4mo ago
Think of this as another way of achieving that. This theoretically has a higher ceiling of how much it can predict at a time. And more importantly is a lot more memory efficient during actual inference.
25.
by sleepyeldrazi 4mo ago
If anyone is interested in watching my 0.8B experiments: https://orthrus.kokoham.com/ . The current code is here: https://git.kokoham.com/sleepy/qwen_orthrus . The hard part was that the original Orthr
26.
by sleepyeldrazi 4mo ago
My plan is to validate it first using qwen3.5 0.8B if it even works (as it has the same architecture as qwen3.6 27b, just scaled down a bit) on my 3090. If it does, I'll make a git about the process if anyone wants to use my approach,
27.
by sleepyeldrazi 4mo ago
Scratch that, I don't have that kind of money, and 3.5's architecture is a little more divergent from 3's, so it will be a bit less trivial. It does look possible, just not on a student's paycheck.
28.
by sleepyeldrazi 4mo ago
From a quick and shallow view of the paper, it looks very feasible (with a little tinkering ) to be adapted to qwen3.6 27B. The process looks somewhat similar to training a LoRA, or in a way distilling your own model so that a mini model le
29.
by sleepyeldrazi 4mo ago
I love this community, I started building a simple website for this exactly a couple of hours ago and you made an even more advanced version already. Hats off to you sir. If i ever decide to actually publish the site, is it alright if I men
30.
by sleepyeldrazi 5mo ago
I specifically tested on tasks I designed because I know every modern model, not only local ones, are bechmaxxed. The common benchmarks most labs use are (very likely) in their datasets to a degree (I'm assuming unintentionally, but is

More ›