8 ms·
A few words on DS4
- bjconlan 4mo agoThis is great! I feel the same way about the deepseek v4 architecture for commodity hardware. Also have enjoyed playing with https://huggingface.co/HuggingFaceTB/nanowhale-100m-base https://huggingface.co/HuggingFaceTB/nanowhale-100m-base (but early days for me understanding this space)
- kamranjon 4mo agoVery cool! I had no idea that HF was doing this - I really love their small model experiments.
- simonw 4mo agoI got this running on a 128GB M5 the other day - pretty painless, model runs in about 80GB of RAM and it seemed to be very capable at writing code and tool execution.
- perfmode 4mo agoHow’s the token throughput / response time?
- simonw 4mo agoHealthy! prefill: 30.91 t/s, generation: 29.58 t/s From https://gist.github.com/simonw/31127f9025845c4c9b10c3e0d861217d https://gist.github.com/simonw/31127f9025845c4c9b10c3e0d8612...
- xienze 4mo agoI don't want to be a jerk but 31t/s prefill is basically unusable in an agentic situation. A mere 10k in context and you're sitting there for 5+ minutes before the first token is generated.
- aiscoming 4mo agoif it's just the coding agent system prompt and tools, you can cache that
- xienze 4mo agoYeah the problem is that's just the start of the context. There's, you know, all the tool call results and file reads and stuff.
- deleted 4mo ago[deleted]
- fgfarben 4mo agoThat prefill number isn't right. M4 Max hits 200-300: https://github.com/antirez/ds4/blob/main/speed-bench/m4_max_ts.svg https://github.com/antirez/ds4/blob/main/speed-bench/m4_max_...
- hadlock 4mo agoM5 studio is gonna sell like hot cakes
- throwdbaaway 4mo agoHah, that's because the prompt itself was only about 30 tokens. We need a much bigger prompt to properly test PP.
- embedding-shape 4mo agoComparison with a RTX Pro 6000, with DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf: prefill: 121.76 t/s, generation: 47.85 t/s Main target seems to be Apple's Metal, so makes sense. Might be fun to see how fast one could make it go though :) The model seems really good too, even though it's in IQ2.
- rtpg 4mo agowhat are token speeds like for frontier models, if that gives a rough idea of how much "slower" slow is?
- antirez 4mo agoPrefill is 400 t/s in that hardware. Just if the prompt is very short you can't see the real speed and it will default to single token context processing.
- simonw 4mo agoHah, that's my fault for just using "Generate an SVG of a pelican riding a bicycle" as my test prompt!
- incidentist 4mo agoSomeone is working on a fork that is optimized for M5, might be worth a look: https://github.com/Swival/ds4-m5 https://github.com/Swival/ds4-m5
- chatmasta 4mo agoSo you’re saying I should buy the M5? :) I’ve been resisting, thinking I’ll never use it… it’ll be better in a year… I’ll wait for the Studio (do we still think that’s coming in June?)… etc.
- simonw 4mo agoI expect this to be my main machine for the next 3-4 years (which is how I justified the 128GB one). It's a beast of a machine - I love that I can run an 80GB model and still have 48GB left for everything else. Can't say that it wouldn't be a better idea to spend that cash on tokens from the frontier hosted models though. I'm an LLM nerd so running local models is worth it from a research perspective.
- simpaticoder 4mo agoAn M5 Max MBP with 128G of RAM costs ~$5k. An Nvidia RTX 5090 with 32G RAM is $4-5k, and RTX PRO 6000 with 96GB RAM $10k. Do you have any data on which is the best price/performance for local inference? Do you know what the big OpenAI/Anthropic/Google datacenters are running?
- aiscoming 4mo ago[dead]
- driese 4mo agoAs always: it depends on your needs. Here's a very basic heuristics rundown: - More RAM: bigger models, more intelligence. - More FLOPs: higher pre-fill (reading large files and long prompts before answering, the so-called "time to first token"). - More RAM bandwidth: higher token generation (speed of output). So basically Macs (high RAM, okay bandwidth, lowish FLOPs) can run pretty intelligent models at an okay output speed but will take a long time to reply if you give them a lot of context (like code bases). Consumer GPUs have great speed and pre-fill time, but low RAM, so you need multiple if you want to run large intelligent models. Big boy GPUs like the RTX 6000 have everything (which is why they are so expensive). There are some more nuances like the difference of Metal vs. CUDA, caching, parallelization etc., but the things above should hold true generally.
- kamranjon 4mo agoJust want to mention that I've been pulling down and using DwarfStar locally and it's incredible. I actually have it running on my personal macbook m4 max with 128gb of ram and I am running the server to share it through tailscale with my work laptop and just have pi running there. The long context reasoning is something I haven't even seen in frontier models - I was running at 124k tokens earlier and it was still just buzzing along with no issues or fatigue. I am amazed at how well it works, I'm using it right now for some pretty complex frontend work, and it is much much faster than, for example running a dense 27b or 31b model (like qwen or gemma) for me (The benefits of MoE) - but the long context capabilities have been what have been absolutely flooring me. Super excited about this project and hope Antirez can keep himself from burning out - i've been following the repo pretty closely and there are a ton of PR's flooding in and it seems like he's had to do a lot of filtering out of slop code.
- le-mark 4mo agoIs DS4 dwarf star 4 or deep seek 4?
- 0xbadcafebee 4mo agoI don't see an explanation of why they would make a model-specific inference engine vs just using llamacpp. There are already lots of people working on the llamacpp integration. This is a lot of effort spent on a single model which is likely to become obsolete when a different model comes out that does better. In some discussions, people are now making PRs against both the llamacpp branches and ds4... so it's taking a rare commodity (people investing development time in this model) and fragmenting it
- zozbot234 4mo agoAuthor has mentioned many times that the llama.cpp maintainers don't want code that's prevalently written by AI with no human revision. If anyone wants to try and get the support upstreamed into that project, they're quite free to do that: the code is MIT licensed.
- kristianp 4mo agoAlso Antirez has been able to use GPT to iterate on the code and performance. He/they (others contributed to DS4) has a set of result files to ensure that correctness is maintained, and benchmarks to verify performance, and the LLM is able to iterate within that framework. Having a small, focussed codebase helps here. Antirez explained the dev process when he posted a pure C implementation of the Flux 2 Klein image gen model, at https://news.ycombinator.com/item?id=46670279 https://news.ycombinator.com/item?id=46670279
- flakiness 4mo agoI believe the assumption is: The code is cheap. The collaboration (eg. upstreaming) is expensive. Is it true? We'll see, in a few years.
- fgfarben 4mo agoAt a certain point the level of abstraction / genericization necessary for a big flexible project (like llama.cpp or Linux) blows things up into a huge number of files. Something newer and smaller can move faster.
- 4mo ago
- minimaxir 4mo agoA relevant recent tweet from antirez: https://x.com/antirez/status/2054854124848415211 https://x.com/antirez/status/2054854124848415211 > Gentle reminder on how, in the recent DS4 fiesta, not just me but every other contributor found GPT 5.5 able to help immensely and Opus completely useless. I've noticed the same for lower level squeezing-as-much-performance-as-possible code work.
- throwaway041207 4mo agoAssuming we are talking about Code/Codex are you on API billing or subscription? I have essentially unlimited API billing at my disposal and I haven't noticed any degradation of quality across Opus versions.
- chatmasta 4mo agoSame here, the enterprise version of Claude has been great. Luckily I’m not the one paying for it. We also have CoPilot and when GPT-5.4 came out, and was 1x request cost, I was very impressed but haven’t had much time to compare the two. I also don’t have time to do much personal coding outside of work, so I haven’t subscribed to a personal one yet. But I intend to go for Codex just to balance the Claude at work and also because of the hostile moves from Anthropic toward their consumer business.
- sanxiyn 4mo agoThere is a benchmark for performance work, and I think it is not being optimized by model vendors. The latest result from GSO is that both Opus 4.6 and 4.7 slightly outperforms GPT 5.5. This also matches my experience. https://gso-bench.github.io/ https://gso-bench.github.io/
- vitorsr 4mo agoTasks are taken from commit histories in public Git repositories which defeats the purpose.
- deleted 4mo ago[deleted]
- codedokode 4mo agoI thought DeepSeek was closed-weights and proprietary? I wonder how it compares against Western open-weight models. The hugging face page contains the comparison only with proprietary models for some reason.
- zozbot234 4mo agoNemotron would be a comparable Western open model AIUI.
- itishappy 4mo agoDeepSeek has always been open-weight, and the DeepSeek HuggingFace page does not contain any comparisons. Where did you form these opinions?
- codedokode 4mo agoIt contains comparisons: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash
- itishappy 4mo agoJust the first one then... Apologies. Where did I form my opinions?
- deleted 4mo ago[deleted]
- sbinnee 4mo agoIt is a big thing for sure to have a competitive local agentic model. I've replaced gemini 3 flash preview with DeepSeek v4 flash for all of my personal use cases. Starting from chat app, language learning, and even hobby coding. For coding, I couldn't get decent results no matter which sota latest models I used before. It's not close to Opus or Codex models. It's a flash model and makes mistakes here and there (I just saw `from opentele while import trace`, new Python syntax!) But I found its tool calling is reliable than other oss models I tried. I assume that it attributes to interleaved thinking. Its reasoning effort is adjusted automatically by queries. I enjoy reading these reasoning traces from open models because you can't see them from proprietary models. I would love to try DS4 so bad. Well, I don't have a machine for it. I will just stick to openrouter. I wish I can run a competitive oss model on 32GB machine in 3 years.
- zozbot234 4mo ago> I wish I can run a competitive oss model on 32GB machine in 3 years. You could try DS4 on that machine anyway and see how gracefully it degrades (assuming that it runs and doesn't just OOM immediately). Experimenting with 36GB/48GB/64GB would also be nice; they might be able to gain some compute throughput back by batching multiple sessions together (though obviously at the expense of speed for any single session).
- thegeomaster 4mo ago> `from opentele while import trace` FYI, this to me points to an inference bug, bad sampling, or a non-native quant. OpenRouter is known to route requests to absolutely terrible, borked implementations. A model like DeepSeek V4 Flash shouldn't be making syntax errors like this.
- kristianp 4mo ago> I wish I can run a competitive oss model on 32GB machine in 3 years. It's so hard to predict what size the open-weight models will be, even in 6 months time. Will a 96GB machine turn out to be a complete waste of money? Who knows.
- 4mo ago
- brcmthrowaway 4mo agoThis guy is falling deep into Yegge-tier psychosis.
- linkregister 4mo agoEmpirically, DS4 is hosting the DeepSeek v4 Flash model with good performance on home hardware. I'm curious how you came to this conclusion.
- dakolli 4mo ago"Empirically", have you tested this yourself?
- linkregister 4mo agoIt's trivial to find reviews and benchmarks of DS4 online. Also, there are benchmarks in the article. Here's one of the top hits: https://forums.developer.nvidia.com/t/fully-custom-cuda-native-deepseek-4-flash-optimized-for-1x-spark-antirez-ds4/369791 https://forums.developer.nvidia.com/t/fully-custom-cuda-nati... Bizarre comment; sounds like "How do you know Porsches are fast? Did you drive one?"
- calmingsolitude 4mo agoParent is simply pointing out the incorrect usage of "empirically", which should typically only be mentioned when you've tested it yourself.
- linkregister 4mo agoI'm having trouble finding dictionaries or other references that add the qualifier that it needs to be self-tested and not relying on the research of others. Can you point me to one?
- dakolli 4mo ago
- deleted 4mo ago[deleted]
- FuckButtons 4mo agoIt’s shocking how close this feels to claude, obviously it's much slower, but I don’t know that it’s significantly dumber. Interestingly the imatrix quantization seems to be better than whatever quant the zdr inference backends on open router are using. It was self aware enough yesterday to realize that it’s own server process was itself without me telling it, which is not something I’ve ever observed a local model doing before.
- stavros 4mo agoIn my (obviously anecdotal) testing, DeepseekV4 Pro was better than Sonnet at coding. However, it is much slower, but also many times cheaper, especially with the promotion right now.
- DeathArrow 4mo agoDo they have a coding plan or you only pay per API call?
- trollbridge 4mo agoIt’s just per token, but burning up 100 million+ tokens is a $3 transaction with their pricing right now
- DeathArrow 4mo agoDo you use the official API or another provider?
- stavros 4mo agoI use the official API, OpenRouter somehow didn't use caching and one short session with Qwen cost me $5.
- trollbridge 4mo agoJust directly. Paid for it with PayPal. It’s quite simple to set up and use.
- somewhatrandom9 4mo agoWith "intelligence" (or whatever you want to call it) and speed both seeming to ramp up quickly with local models I wonder what the growth rate and ceiling(?) might be in this space. Will this kind of iq and performance work with just e.g: 16GB RAM in a couple years? Is there a new kind of Moore's law to be defined here?
- lwansbrough 4mo agoThe people working at the leading edge of this stuff seem to believe that there is a need for parallel models that solve different problems. A crow exhibits some degree of intelligence in what is a very small brain compared to humans. There is overlap in the problem solving skills of the dumbest humans and the smartest crows. So the question is: what is that? Yann LeCun seems to think it’s what we now call world models. World models predict behaviour as opposed to predicting structured data (like language.) If your model can predict how some world works (how you define world largely depends on the size of your training data), then in theory it is able to reason about cause and effect. If you can combine cause and effect reasoning with language, you might get something truly intelligent. That’s where things seem to be going. Once we have a prototype of that system, there will be many questions about how much data you really need. We’ve seen how even shrinking LLMs with 1-bit quantization can lead to models that exhibit a fairly strong understanding of language. I don’t think it’s unreasonable to expect to see some very intelligent low (relatively) memory AI systems in the next couple years.
- hadlock 4mo ago640gb ought to be enough for anybody
- famouswaffles 4mo agoSqueezing a model like this complete with 'big model smell' into 16GB...Honestly it's not even possible or feasibly possible today. It'll require some kind of: - breakthrough in architecture or - breakthrough in hardware or - some breakthrough quantisization technique The problem is that all the parameters need to be in memory, even the ones that aren't active (say for Mixture Of Expert Models) because switching parametrs in and out of ram is far too slow.
- karmakaze 4mo agoGreat to find this narrow focused thing: > We support the following backends: Metal is our primary target. Starting from MacBooks with 96GB of RAM. NVIDIA CUDA with special care for the DGX Spark. AMD ROCm is only supported in the rocm branch. It is kept separate from main since I (antirez) don't have direct hardware access, so the community rebases the branch as needed. > This project would not exist without llama.cpp and GGML, make sure to read the acknowledgements section, a big thank you to Georgi Gerganov and all the other contributors. Edit: aww, doesn't seem to support offloading to system RAM[0] (yet) [0] https://github.com/antirez/ds4/issues/108 https://github.com/antirez/ds4/issues/108 Guess I'll have to keep watching the llama.cpp issue[1] [1] https://github.com/ggml-org/llama.cpp/issues/22319 https://github.com/ggml-org/llama.cpp/issues/22319
- keyle 4mo agoIf only you could still buy Mac's with that much RAM
- shric 4mo agoYou can buy 128GB M5 MacBook Pros? Configured one just now, delivers in 2 weeks
- keyle 4mo agoInteresting there were news last week or so of apple removing Mac minis options.
- littlecranky67 4mo agoThey removed the baseline 8GB RAM/256GBstorage model. My bet is with increased RAM prices the markup on the lower end is not enough to still make a profit
- a1o 4mo agobaseline was 16GB RAM
- gcr 4mo agoDwarfStar4 is a small LLM inference runtime that can run DeepSeek 4. The blog post implies that it currently requires 96GB of VRAM. For others who are lacking context :-)
- foresto 4mo agoThanks. Outside of LLM circles, DS4 is usually a video game controller.
- artyom 4mo agoWell, I was sitting here expecting the Redis creator have an opinion on still-unannounced Dark Souls 4.
- low_tech_love 4mo agoHaha the same here!!
- jofzar 4mo agoI am actually kind of disappointed it wasn't a deep dive on the dual shock 4
- oezi 4mo agoOr a car from Citroen
- pavlov 4mo agoTechnically DS is an independent sibling of Citroën within Stellantis, a sprawling car conglomerate that owns a dog’s dinner of car brands in Europe and USA.
- orthoxerox 4mo agoIt's still the Lexus to Citroen's Toyota.
- 4mo ago
- easythrees 4mo agoI thought for a moment there was a Dark Souls 4
- JavierFlores09 4mo agoGlad I wasn't the only one, my second thought was Dual Shock controller but that wasn't it either lol
- NDlurker 4mo agoI was thinking dual shock 4
- txhwind 4mo agoFucking abbreviations. Who knows it's DeepSeek, Dark Souls or DualShock? All possible on HN.
- the__alchemist 4mo agoCould be Death Stranding too
- blitzar 4mo agoThe prequel to the prequel of Deep Space 9
- tuveson 4mo agoI thought Future put out a new album.
- zmmmmm 4mo agoI'm very curious where we will saturate the curve on "enough" intelligence for coding. At some point, you can let a less smart model hammer at a problem for longer and get to the same result, and as long as you are not involved it comes to the same thing. I feel like DeepSeek V4 Pro is nearly there. Maybe Flash is too. Once we hit that point, I am curious how much of Anthropic's current business model falls apart? So far it's always been clear that you just pay for the most intelligent model you can get because it is worth it. It now seems clear to me that there is limited runway on that concept. It is just a question of how long that runway is. I honestly wonder how much of their frantic push to broaden out into enterprise / productivity is because they see this writing on the wall already.
- jofzar 4mo ago> I'm very curious where we will saturate the curve on "enough" intelligence for coding. At some point, you can let a less smart model hammer at a problem for longer and get to the same result, and as long as you are not involved it comes to the same thing. I feel like DeepSeek V4 Pro is nearly there. Maybe Flash is too. It's always going to be cost; developer time vs developer cost vs AI cost vs developer productivity. With 4.6 it's looking like we are at the upper limit of appetite for cost (for "regular" Business) so the other levers will probably need to change.
- loeg 4mo ago> At some point, you can let a less smart model hammer at a problem for longer and get to the same result, and as long as you are not involved it comes to the same thing. Is that true? I find the smarter models can just be effective when smaller models can't. It isn't a matter of just waiting longer.
- davnicwil 4mo agoit's almost certainly not true yet but at some point there might be an equilibrium reached of speed Vs quality (and let's not forget, cost) where it's true for most of what you do. Perhaps you'd still turn to hosted models for the hardest tasks, but most tasks go local. It does seem like that would make demand go down significantly. Of course that's all predicated on model advances plateauing, or at least getting increasingly more expensive for incremental improvements, such that local open source models can catch up on that speed/quality/cost curve. But there is a fair amount of evidence that's happening. The models are still getting noticably better, but relative improvement does seem to be slowing, and cost is seemingly only going up.
- kgeist 4mo agoDid someone compare DeepSeek 4 Flash to Qwen3.6-27B on real tasks (quality + speed)? According to the benchmarks at artificialanalysis.ai, Qwen3.6-27B is better at agentic tasks, and DS4 is only 2 points better at coding (both with max reasoning effort, full weights). At the same time, DS4 requires 5 times more VRAM even at 2 bits. Last time I explored this topic, large MoE models at 2-3 bits usually performed worse (quality-wise) than dense ~30B models at 4-8 bits, despite being much heavier to run. Sure, MoE models have more knowledge, but extreme quantization may negate the benefits. And generally for coding tasks, you don't need a model that has memorized all the irrelevant trivia like, I don't know, the list of all villages in country X. DS4 also seems to run much slower on Mac Studio Ultra, which appears to be more or less in the same price range as RTX 5090. RTX 5090 gives me 50-60 tok/sec and 260k context with Unsloth's 5-bit quantization (only some layers are 5-bit too) and an 8-bit KV cache; prefill is instant too. It works flawlessly in OpenCode. If you already have a spare high-end Mac, I can see the benefit, but I'm not sure it's a good configuration overall. Unless Qwen3.6 is more benchmaxxed than DS4 :)
- muyuu 4mo agofor unified memory, the dense models are way too slow and for local GPU-based setups, large MoE are too large but they're fine on unified memory systems essentially, hardware is the main reason you may choose one or the other locally i have a Strix Halo system so I will be trying this Dwarf Star 4 thingie eventually when i have some free time
- Riany 4mo agoI think local models need to be good enough that privacy, latency, and control become worth the tradeoff, instead of beat the best cloud models
- vrighter 4mo agoDamn it I was expecting something interesting about the ps4 controller. Not some more junk about AI. Such a rugpull
- ilaksh 4mo agoI want something like this but not only for my own computer but also for client projects or stuff I might run in cloud GPUs. Because the core idea of having a strong model that is efficient and doesn't require a cluster still applies to a lot of business cases. I am hoping something like this can work in batch mode. Right now I feel like a 4bit Qwen 3.6 27B with MTP is one of the best for agentic tool calling for some smart voice agents in an H200. I wonder if DS4 Flash being using 80b at 2 bit with 13b active and MTP could be even faster and smarter and allow more concurrent sequences? This special 2bit quantization seems like a big deal.
- sourcecodeplz 4mo agoThis project is a week old and already super popular. Guess people really were tired of lmstudio or tunning llama.cpp with settings.
- zargon 4mo agollama.cpp (and consequently LM Studio) don't support DeepSeek V4. If you want to run V4, this is your only option right now unless you have hardware that can run vLLM.
- ljosifov 4mo agoLove this, even if can't use it atm (not got the h/w - only 96gb on M2 Max). I get it the general comp/public will find it unusable or worse. Reminds me of how home computers were - mere toys - before they became personal computers (PC). On my h/w the only passable combo for me atm is pi agent + llama.cpp + nemotron cascade-2 model: to 1M context, hybrid arch doesn't crash & burn 1/N^2 with context depths of 10K-50K-100K used by code agents. Was on a plane without Internet the other day. Brought a smile to my face that I could run pi agent (with llama.cpp serving), and it was just about usable at 40-30 tok/s. Afaik the usual API speeds are double that, 60-80 tok/s. Sensors showing using 60W when running inference. So battery probably would not last more than >3h. Model only 30B in size leaves plenty of space for KV-caches, and other programs - even at generous 8-bit quant. Only 3B active params at one time (with MoE A3B) is about the most that ageing M2 Max can carry it seems.
- zozbot234 4mo agoIt should work with 96GB, especially on a limited context. But the M2 Max is a bit slower, yes.
- embedding-shape 4mo ago> even if can't use it atm (not got the h/w - only 96gb on M2 Max). Not sure if it works different on macOS, but with CUDA + DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf I can fit it within 96GB of VRAM, together with context, so theoretically I feel like you should too, unless macOS uses GB of RAM/VRAM for the OS/display by default.
- ljosifov 4mo agoOn 96gb I can give up to about 88GB to the GPU with sysctl iogpu.wired_limit_mb=88000, without suffering any ill-effects. When pushed higher I tend to notice e.g. graphic driver errors, youtube web page not working, other semi-random glitches. So the ~80 GB of DS4-flash quants I could just about fit. Leaving some extra for the KV caches. Will try, I'm curious how's the DS4 degradation with context depth growth, how fast does tok/s drop. E.g. 2-bit lowest quant MiniMax-M2.6 runs, but starts low tok/s and degrades fast with context depth. The biggest models I can comfortably run are about 1/2 the DS4F size - like gpt-oss-120b. Lately was toying with Ling-2.6-flash. Got the agents to adapt existing metal kernels in llama.cpp, and it did run (model https://huggingface.co/ljupco/Ling-2.6-flash-GGUF https://huggingface.co/ljupco/Ling-2.6-flash-GGUF, branch https://github.com/ljubomirj/llama.cpp/tree/LJ-Ling-2.6-flash-r2 https://github.com/ljubomirj/llama.cpp/tree/LJ-Ling-2.6-flas...). It's 104B-A7B4, and for the M2 Max 7.4B active is about the most it can take while still producing 40 tok/s. And the hybrid arch allows for graceful degradation, still close to 30 tok/s at 64K context depth. Too bad L2.6F while the best have, is not that much better in agentic benchmarks compared to my current incumbent local llm (nemotron-cascade-2). Got inspired by DS4 to start a l26f branch (WIP https://github.com/ljubomirj/l26f https://github.com/ljubomirj/l26f). :-) Try squeeze the most from L2.6F. There should be low hanging fruit in good integration of the agent and the inferencing engine. On input - considering the huge difference cached v.s. non-cached tokens. On output - considering that the NN gives us the complete logits set for all 200K+ tokens vocabulary.
- petercooper 4mo agoI've been using the Q4 version on my Mac Studio over my local network and it's been good. Indeed, I had the first ever experience where I was playing with it alongside my various other agents and forgot it was a local model as it was doing such a good job. I do wonder, though, if another agent is really needed. I've been driving it with Pi (Claude Code's system prompt is far too heavy given the prefill speeds) and it's been great. OpenCode is another good option. Is there anything else to gain from another similar tool specific to Deepseek 4?
- zozbot234 4mo agoDS4 is an inference engine, not a harness. It provides an inference API server and you point your coding harness to it.
- antirez 4mo agoYou misunderstood the OP. I hinted, in my blog, at my interest to also putting an agent harness inside.
- antirez 4mo agoThere is no need for another agent, functionally. But if you follow the idea of DS4 itself: the API agents use forces to do odd things, like translating the DSML stanzas to JSON, with all the canonicalization / KV cache checkpointing problems resulting from that. Is it really the case? What about also providing a sane alternative? Also I'm not sure why people don't try to write more stuff in that area in C/Go/Rust to have more control / speed / less dependences. Also there is a lot more to imagine, TUI side. The problem is that most projects all copy what they already saw. For instance I just did this in 20 minutes: https://x.com/antirez/status/2055190821373116619 https://x.com/antirez/status/2055190821373116619 Now that code is cheap, ideas have more value. Are we sure that today it is still the case to think in terms: "Is another XYZ needed"? It could be the case that only just to explore new ideas, it is worth it. I I don't like the Javascript / Node ecosystem for my code, so if I have to explore a new TUI or agent workflow, if I do it with the tools I'm more happy to use, the result, the iterations, are different.
- karel-3d 4mo agoOh a local DeepSeek? Nice > Starting from MacBooks with 96GB of RAM. ... oh. And I thought I bought a lot with 48 GB.
- zozbot234 4mo ago96GB is what the author claims will work in a foolproof way for easy production use. But nothing stops you from trying to run it on 48GB, it ought to gracefully fall back on accessing model layers from the disk.
- whazor 4mo agoSome of my colleagues believe that current frontier AIs are too heavily subsidized and it will come to an end. They think frontier coding AI's might get unavailable for one reason or another. But these kind of projects show that with 6000$ Macbook we are getting closer to a local frontier model. More importantly, it shows the genie will not go back into the bottle.
- NitpickLawyer 4mo ago> This project supports steering with single-vector activation directions; [...] This is also useful for cybersecurity researchers who want to reduce a model's willingness to provide dual-use or offensive security guidance. Wink wink, nudge nudge. I have a feeling most cybersec researchers would only be interested in negative values of "reduce" :D
- albertzeyer 4mo agoMore information about DwarfStar 4 (DS4) in the readme: https://github.com/antirez/ds4 https://github.com/antirez/ds4 The code seems based on llama.cpp and GGML. I don't fully understand why it is a standalone project. The readme discusses this: DwarfStar 4 is a small native inference engine specific for DeepSeek V4 Flash. It is intentionally narrow: ... I think the only bigger difference in DeepSeek V4 vs other models is maybe the type of self-attention. And that leads to: KV cache is actually a first-class disk citizen. But I still feel like those changes could have been implemented as part of some of the other local engines. I also assume more models will come out, not just from DeepSeek but also from others, and they might share similar self-attention approaches, that would benefit from a similar KV cache implementation.
- skiwithuge 4mo agobecause llama.cpp doesn't accept fully pr made by ai agents even if they are guided by the author https://github.com/ggml-org/llama.cpp/blob/master/AGENTS.md https://github.com/ggml-org/llama.cpp/blob/master/AGENTS.md
- embedding-shape 4mo agoWhich makes sense, the amount of PRs llama.cpp receives from authors who have no clue what they're doing and can't even answer simple questions about "what they did" is staggering, must be very exhausting to have to figure out "is it worth replying to this author?" for every single PR.
- antirez 4mo agoCheck the readme better. The code overlap with ggml is very small, but a few kernel and ideas and the quants code were taken. Still the project connection with llama.cpp and ggml is huge and also present in the license because it's not a matter of code but of a whole ecosystem built, engineering lessons on how to do things and many other stuff. Also the readme explains exactly why a vertical inference system for a single model is the goal of the project.
- ttoinou 4mo agoWhen I ran DS4 Q2 the other day (without the new update Q2 imatrix) it was behaving quite poorly after a few agentic turns with opencode, it couldn't modify the files it was telling me the work was ready and didn't use any tool to update files
- antirez 4mo agoThe bugs were on the API tool call handing. The model worked well. I would retest with updated code and gguf and I and many others never saw it missing anything obvious. Reliable tool call and reasoning. The project is a few days old so certain agents / API combinations definitely had DSML related issues.
- ttoinou 4mo agoI retested and it’s much better now. Wow !
- wg0 4mo agoDeepSeekV4 Pro is really really competent model and what makes it extremely good is the price point it is offered at. I have been toying with a 2.5D engine in C on on top of raylib and using DeepSeek as companion in between. It's thinking transcripts in OpenaCode are transparent and mind boggling to look at things it would consider in its thought process. Very long to read but none of them useless or meaningless. Always happened that I discovered an assumption that I didn't think about or was wrong but DeepSeek flags it in its thought process and then in final output it would "align" to my flawed request and I'll tell it wait, I saw you thought so and so too and that's correct I made a mistake let's consider that aspect too.
- etoulas 4mo ago> The last week was funny and also tiring, I worked 14 hours per day on average. My normal average is 4/6 since early Redis times, but the first few months of Redis were like that. Is he taking about 4/6 h of coding? If he meant total working time I’d say this is a very balanced lifestyle!