14 ms·

A 10 year old Xeon is all you need

740 points by cafkafk 4mo ago

bflesch 4mo ago
Might consider going for even older CPUs which don't have the Intel ME ring -3 thing which is full of backdoors
- bflesch 4mo ago
  I appreciate the downvotes without any reasoning. It's a fact that newer Intel CPUs have Intel ME which was not in older CPUs and significantly increases attack surface if you are not living in a five eyes state.
  - s20n 4mo ago
    I agree with the first part. I think this article by FSF about Intel's ME summarizes the issue https://static.fsf.org/nosvn/blogs/Intel_ME_Carikli_article_PRINT_2.pdf https://static.fsf.org/nosvn/blogs/Intel_ME_Carikli_article_... As for the second part, I am not sure about how living in a five eyes state would mitigate it. What do you mean by that?
    - bflesch 4mo ago
      As five eyes citizen you have at least some rights on paper and you can appeal to your government, but if you are foreigner these guys can go gloves off without any fear of retribution. Try analyzing Epstein files and posting about it, they'll give you a proper penetration test of all your devices to see what you found out about their ex employee. Nowadays even EU citizens migrating away from US cloud providers are a "national security issue".
      - smilespray 4mo ago
        Isn't the whole five eyes argument moot because member states spy on citizens from the other countries and trade intel with each other?
        bflesch 4mo ago
        No need for that charade if you are a foreigner, even from NATO ally.
  - tryauuum 4mo ago
    How old are we talking?
    - bflesch 4mo ago
      IIRC it is pre-2008.
  - adrian_b 4mo ago
    In a server, you have to worry about the ME only if you also have an Intel Ethernet interface, which is connected to a potentially hostile network. If that is not true, the ME cannot be controlled remotely. The existence of the ME is much more worrisome in laptops, where the ME can be accessed remotely through WiFi. There, to be certain that there is no way for the ME to be accessed remotely you would have to disconnect or cut the internal antennas and use a USB dongle for WiFi.
shovas 4mo ago
I have run llama.cpp on an i7-2600 with a 1050. It's too slow for everyday usage but it's not too slow to make it obvious AI is going to be everywhere and in everything. It's too easy to run.
cafkafk 4mo ago
Hi HN. I wrote this post after getting frustrated by the lack of ways to run the new Gemma 4 Drafter models, and mainstream tools not prioritizing this, and hiding all the performance levers. I ended up getting a modern 26B MoE model (Gemma 4) running at reading speed on an old recycled server with a single Xeon E5-2620 v4 and 128GB of DDR3 RAM (and no GPU). It took a lot of work, but it actually worked out somehow. I've also linked the quants at the end, but they're not gonna run unless you use the ik_llama-cpp fork I mention, see other posts for more details. I'm not an ML engineer, so I'm by no means an expert, and the server is busy acting as a Nix cache, but if you have any question, I can try to answer, but best effort.
- fragmede 4mo ago
  (purple on black is really hard to read) You say it runs "at reading speed". Have you benchmarked it?
  - cafkafk 4mo ago
    > (purple on black is really hard to read) Noted, and agree (it looks like it has also already been clicked, which I dislike). I honestly I need to redo the themes. > You say it runs "at reading speed". Have you benchmarked it? At some point a few weeks ago, yes I think so, but I didn't write it down for some reason... so I'll have to find a time when it's not busy and do it again without a noisy system. Right now the system is noisy, but that said doing it like this: llama-cli --model gemma-4-26B-A4B-it-Q8_0.gguf --model-draft gemma-4-26B-A4B-t-assistant-GGUF/wikitext-2-raw_ik-llama-mtp_drafter-conservative/gemma-4-26B-A4B-it-assistant-Q8_0.gguf --spec-type mtp --draft-max 3 --draft-p-min 0.0 --color -sm graph -smgs -sas -mea 256 --split-mode-f32 --temp 0.7 --cpu-moe -t 8 --flash-attn on --mla-use 3 --merge-up-gate-experts --special --mlock --run-time-repack --spec-autotune --no-kv-offload --parallel 8 --jinja -p "Why is the sky blue?" -n 128 Gives: llama_print_timings: load time = 83911.65 ms llama_print_timings: sample time = 26.99 ms / 128 runs ( 0.21 ms per token, 4742.15 tokens per second) llama_print_timings: prompt eval time = 343.41 ms / 7 tokens ( 49.06 ms per token, 20.38 tokens per second) llama_print_timings: eval time = 10639.36 ms / 127 runs ( 83.77 ms per token, 11.94 tokens per second) llama_print_timings: total time = 11114.98 ms / 134 tokens So 11.94 tokens per second while it's also playing binary cache and CI builder. When I do it properly, I'll add it to the blog as well!
Eonexus 4mo ago
I wonder what the tokens per second actually are. Yes, it does say "reading speed" but that varies for everyone, no?
- cafkafk 4mo ago
  That is a very fair point! I just ran a not very scientific benchmark with the system under load, and posted the raw logs in a sibling comment above, but the short answer is that it's hitting 11.94 tokens per second for generation - while it's also being a binary cache and CI build server. Totally just vibes based, I think it goes up to 20+ tps when it's not under load (and that's me trying to be conservative). For context, reading speed at 250 wpm would be around 5 to 6 tokens per second.
  - Eonexus 4mo ago
    Huh, that's actually not bad at all! Sure, it's not at the speed of a GPU, but still, 20 tps is cromulent for a CPU.
potus_kushner 4mo ago
@cafkafk got a recommendation for a good model that fits into 64GB and leaves a couple GB free for other tasks ?
- cafkafk 4mo ago
  Honestly, at this point you're probably looking at a smaller model, for the Gemma series I'd go with Gemma 4 E4B with drafters, but that's just a hunch from using it on my laptop (where I do have a RTX 4060 M and 96gb ram). So you'd change the invocation slightly here, but a lot of things you can potentially reuse. That said, the Gemma 4 E4B models have so far in my experience been... not great when it comes to long context, but they are very passable for basic tasks, and even seem surprisingly okay at tool calls.
  - potus_kushner 4mo ago
    i tried the Q4_K_M model form unsloth with your Q4_K_M drafter, but the required memory to load everything is 72GB. odd. otoh i could load Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled.IQ4_XS.gguf and it requires just ~18 GB: ~/ik_llama.cpp[main]$ build/bin/llama-cli --model ~/models/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled.IQ4_XS.gguf --spec-type mtp --draft-max 3 --draft-p-min 0.0 --spec-autotune -cnv --color --jinja --special -smgs -sas -mea 256 --temp 0.7 -t 6 --parallel 6 --cpu-moe --merge-up-gate-experts --flash-attn on --mla-use 3 --mlock --run-time-repack --no-kv-offload . works pretty fast, at about 15 t/s: llama_print_timings: sample time = 45.28 ms / 404 runs ( 0.11 ms per token, 8921.67 tokens per second) llama_print_timings: prompt eval time = 949.42 ms / 51 tokens ( 18.62 ms per token, 53.72 tokens per second) llama_print_timings: eval time = 24067.08 ms / 400 runs ( 60.17 ms per token, 16.62 tokens per second) llama_print_timings: total time = 242192.55 ms / 451 tokens so i wonder why the params used by the quantified qwen model use way less memory than the ones of gemma.
  - sleepyeldrazi 4mo ago
    Have you tested Qwen3.6 35B? Putting aside the capability claims for that model (which I support, but are not my point here), that 35B has smaller active parameter count than the gemma 4 26B, potentially making both prefill and decode faster out of the box, and has MTP heads built in the model and well supported (you may need to make sure you download a quant that didn't strip them off, as some do to preserve space). I would be curious to see your numbers there too. And if you do test this, please go for a clean one and not a fine-tuned one.
christkv 4mo ago
Makes you wonder if its possible to squeeze more tps out of a strix halo system using the 16 zen5 cores as well as the gpu.
- cafkafk 4mo ago
  If you get the inference engine to route the heavy matrix math to the GPU and the speculative drafting to the CPU without choking on latency it's probably gonna be very fast. Would love to see the benchmarks if someone actually pulls something like that off.
- Havoc 4mo ago
  In general you’re mem bandwidth constrained so cpu vs gpu often ends up similar on APUs
  - fulafel 4mo ago
    There are ways to trade off compute power for memory bandwidth (like MTP and other speculative decoding approaches). The CPU and GPU would need to be able to share the same cache for this to work. In the Strix Halo case the GPU has a private cache on the GPU die I think, which is the snag.
asimovDev 4mo ago
I have an ancient DDR3 Xeon that doesn't support any AVX (dual x5690 and 96GB 1333 MHz RAM). You reckon it would even build / run at all?
- tgtweak 4mo ago
  It may work - depending on your ram speeds it might not even be that much slower.
- cafkafk 4mo ago
  Loading will take some minutes, but at 96 you can squeeze the model in and have some headroom around like ~10 GB, although depending on the Xeon, you may have to downgrade to E4B instead. Should still work thou.
- qwertox 4mo ago
  CPU (2012) Model name: Intel(R) Xeon(R) CPU E3-1265L V2 @ 2.50GHz Mainboard Product Name: P8Z77 WS GPU 05:00.0 VGA compatible controller: NVIDIA Corporation AD106 [GeForce RTX 4060 Ti 16GB] (rev a1) 05:00.1 Audio device: NVIDIA Corporation AD106M High Definition Audio Controller (rev a1) Memory: 32GB This works.
- burnt-resistor 4mo ago
  I run Win 11 Enterprise on an el cheapo spare parts Xeon E3-1275 V2 + 32 GiB DDR3-2133 + Gigabyte GA-B75M-D3H rev. 1.2 (TPM support)
gigatexal 4mo ago
What kind of tokens per second did the op get I saw nothing of this written.
- urbandw311er 4mo ago
  11.94 tokens/sec (from another answer above)
- deleted 4mo ago
  [deleted]
vhaudiquet 4mo ago
The E5 2620-v4 only supports DDR4.
- bobmcnamara 4mo ago
  Probably in an x99 motherboard
  - mwpmaybe 4mo ago
    The memory controller is integrated into the CPU, so the motherboard chipset is irrelevant. There are some OEM-only v3/v4 parts with dual memory controllers, but the E5-2620 v4 is not one of them.
    - bobmcnamara 4mo ago
      Ooh weird!
NSUserDefaults 4mo ago
How about the iMac Pro? Would that work? I was able to put 128gb in it (not as easy as the regular iMac but possible).
- wazoox 4mo ago
  I've been running various models on a Mac Pro 2013 (8 cores, 32 GB RAM) at about 8 to 10 t/s for months. It's not fast, but it's more than enough for many actual tasks, in particular background tasks. An iMac pro will do just as well I suppose.
  - fooker 4mo ago
    What are the tasks that do well with 8-10 t/s ?
    - wazoox 4mo ago
      The sort of task you don't expect to end immediately. If extracting data from a bunch of PDFs takes 1 hour or the whole night, that doesn't make much difference to me. It's not fast enough for auto completion and slightly too slow for chat (but bearable IMO).
      - fooker 4mo ago
        Running a local llm at 10 t/s overnight to extract data from a few PDFs will burn more in electricity than paying cents for the hosted kimi models. You can (sometimes) break even if you have a workstation GPU.
        wazoox 4mo ago
        Sometimes data privacy is paramount.
  - neverartful 4mo ago
    I have and use a Mac Pro 2013 too. Mine is 8 cores with 64 GB RAM. I haven't used mine for any LLM workloads, but it does just fine for most stuff. My biggest concern with it is the OS. I'm still running macOS (the latest supported version) but it's getting continually further out-of-date security wise all the time.
nurettin 4mo ago
I also run a Qwen 3.6 moe A4B on old hardware. I set it up with numactl --membind=1 so it is constrained to one of the memory sticks which speeds up token generation a little.
hparadiz 4mo ago
I'm now staring at a 10 year old 4U with 256 GB of DDR4 and thinking hmmmmm
hypfer 4mo ago
> The argument for speculative decoding is stronger on CPU than on GPU. Uh. Uuuh. No? ___ Also > While a GPU has a massive pool of ultra-fast High-Bandwidth Memory (HBM), a CPU relies on small, lightning-fast “caches” (L1, L2, L3) built directly onto the processor chip. What purpose does the quoting of "caches" serve there? Is this AI writing written by that model running on that host?
phaser 4mo ago
What intrigues me the most about AI progress, is not AGI or the model du jour by $AI_UNICORN, but rather what can be run locally. I remember having an amusing, but rather useless model in a beefy gaming PC that I had 6 years ago; and now, something that’s a hundred times better on my M5 laptop. Should the market react to the memory shortage, the progress of the Apple silicon continue at the same pace, and what we’ll be able to run locally in 6 years will be very exciting. or frightening. Also I don’t know what this means for the valuation of the AI companies. I remember asking about this very idea to one of their employees at an event and instead of answering he bailed out to grab a cocktail.
- skdb476 4mo ago
  Its a convenience thing. You can run a whole lot of stuff locally from wikipedia to social media/email/video servers whatever. Most people with a full time job and 2 kids dont do it cause who has time and energy to patch and maintain the ever growing complexity of this stuff. These systems will keep growing complex. That also means more bugs. Age old tradeoff between freedom and convenience.
  - SadErn 4mo ago
    [dead]
  - phaser 4mo ago
    You can run mediawiki at home but you won't have wikipedia. You can run a video server but you won't have all the movies that Netfix has. A local model is actually the real thing.
    - skdb476 4mo ago
      you can have the whole wiki loaded with full search available locally. check out kiwix.
      - phaser 4mo ago
        Thanks I didn't know about kiwix, but, let's consider the fact that a wiki, or netflix movies are cheap or free, while AI is actually quite expensive at least for now, and i'm not sure if it's because of real costs or to justify the valuation. So there is a bigger incentive to run locally something that's gonna get you $20 or $100 worth of bills to OpenAI than to mirror something that is actually free. Example: In the past there was a whole market for sound cards, if you wanted your computer to have any "multimedia" capabilities you needed to get a sound blaster but now everybody assumes a computer will produce sound, and it's basically for free as all chips have it. Now sound interfaces are still a thing but only for audiophiles who are esoteric enough like me to believe that it's worth to have that extra hi-fi quality. What I think it could happen, is that eventually AI will be part of all the chips, just like soundcards. And there will be people who will buy specialized AI from companies that perhaps are not OpenAI or Anthropic but second-generation sleepers who watched the carnage in the market and decided to enter when it was reasonable. This could be Apple, or Nvidia or something new. They're just waiting for the others to do the research and introduce the taste for it to the masses, just like sound blaster made us fall in love with high fidelity sound in our computers.
car 4mo ago
Similar recent posting with optimizations for older Xeon: High-Performance AI on a Budget: Optimizing llama.cpp for Qwen3.5 Inference on a Dual-GPU HP Z440 https://news.ycombinator.com/item?id=47320244 https://news.ycombinator.com/item?id=47320244
cykros 4mo ago
Does this mean my 15 year old Phenom is too old? But it has 16 gb of DDR3 RAM! Admittedly web browsers and it don't get along that well. Literally the only thing that drags though on my Slackware 15 system, and even then usually only when it gets to around 15 or so open tabs.
SXX 4mo ago
Now we need someone try run Kimi K2.6 on old Xeon and DDR3. After all these platforms do support up to 768GB RAM.
- Havoc 4mo ago
  It’ll work but yield a token per minute. With ancient servers the throughput is the limiting aspect not mem size
- segmondy 4mo ago
  You can run these on a turing machine. At what point is it not worth it? At some point the energy to generate each token matters. We often seen token per second. I think a missing metric is tokens per kilowatt. That is what really matters.
  - SXX 4mo ago
    This is just like running Crysis via software rendering on CPU / llvmpipe. It dont have to be practical in order to be fun to try.
egorfine 4mo ago
This and the previous one are insanely good articles. Thank you!
haunter 4mo ago
And this is one of those CPUs which had dual slot motherboards so you can have double the fun (and power bill) https://pcpartpicker.com/products/motherboard/#s=20028,20029&m=8,18,27,46 https://pcpartpicker.com/products/motherboard/#s=20028,20029...
anon-3988 4mo ago
I tried to run gemma 4 on this CPU and it did not go well https://www.techpowerup.com/cpu-specs/ryzen-7-4800u.c2281 https://www.techpowerup.com/cpu-specs/ryzen-7-4800u.c2281 It is way too slow
throwaway2027 4mo ago
Glad to see other people realizing this. I've been running Gemma 26B-A4B Q4 on a 2012 Xeon with 16GB to 24GB of RAM in a container. It's getting around 8 to 12 tokens per second. Obviously it's not comparable to huge contexts and running it on a GPU and the image decoder in llama.cpp is super slow compared to a GPU but for some small automation tasks and general trivia questions it's decent. The speed is just enough to not have to wait for it to finish so you can read along. Here's my setup. You may want to figure out what the best optimizations are for your specific CPU like AVX2 because mine didn't have most of them. I did try MTP briefly but I wasn't getting performance improvements. You could play around with the batch sizes for cache or context or go even lower for Q2 and don't overcommit on threads either, but I would suggest either defaults or trying out llama-bench. This isn't by any means the best I assume but it worked decently for me and I sometimes swap out Gemma for Qwen. You could also lower q8_0 to q4_0 for more context but it could hurt quality some say, altough I have noticed it too on some models. # Building cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS -DGGML_OPENMP=ON # Running export OPENBLAS_NUM_THREADS=4 export OMP_NUM_THREADS=4 OPENBLAS_NUM_THREADS=4 OMP_NUM_THREADS=4 \ llama.cpp/build/bin/llama-server -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL --temp 1.0 --top-p 0.95 --top-k 64 --min-p 0.00 --jinja --host 0.0.0.0 --port 8080 --cache-type-k q8_0 --cache-type-v q8_0 --threads 4 --threads-batch 4 --ctx-size 8192 -n 8192 --batch-size 2048 --ubatch-size 512 --no-mmap --mlock --chat-template-kwargs '{"enable_thinking":false}' --no-mmproj -np 1 -fa 1
- duffyjp 4mo ago
  I'm setting up a Frankenstein system at the moment. It's a Chinese DDR3 X99 motherboard with a 12 core Xeon v3, 32gb 1866MT/s ram, and a 1080 Ti. I'm shoehorning it back in the Optiplex that donated the ram, so it's not ready to go at the moment, but when I had it running on top of the motherboard box as a test I ran the (9B?) gemma4:e4b-it-q4_K_M since it can fit entirely in the 11gb vram. It flew, more than 50tk/s. A model that small isn't useful for coding, but there could be uses. I'd love to figure out a Wake-on-Use and use it as my personal ChatGPT. I'm not sure how that would work... Maybe proxy the LLM thru a Pi with a script to Wake-on-LAN the PC? It'll be a fun weekend project someday. My always-on LLM is the dense Gemma4:31b that's not quite half in GPU on a 12gb 2060. It's really slow, but the quality is great and my use case is an automated queue so I'm not sitting there watching the output. I have another 2060 but unfortunately the PC won't POST with both installed for some reason.
shevy-java 4mo ago
The webpage's layout is just horrible. Scrolling is also non-default - and thus rather annoying; I had to stop after two scroll events. Why do people think they need so much fancy effects or non-standard behaviour, if their alleged goal is to get information across to other people?
jansommer 4mo ago
The E5-2620 v4 is great. Have been using it for 10 years now. Wanted to upgrade until I saw current prices. I have 64 GB ddr4. Paired it with rx 9060 xt 16 GB and games run as fast as ever. Perhaps the cpu is a slight bottleneck in DOOM The Dark Ages, but i'm at 60 fps, so no problem. Light llm on the gpu is a nobrainer, and it's cool to see that things can be tuned to run ok on the cpu. I bought 2667 v4 a month ago for 30$. I'd expect it to give a decent performance boost but I just haven't had the need for it yet, but pushing into llm like in the article I'd probably upgrade because 2667 can handle slightly faster ram.
- throwaway2037 4mo ago
  > The E5-2620 v4 is great. Have been using it for 10 years now. 10 years? Damn, that is a long time. I always assumed that heat-induced damage will kill a CPU after a certain amount of time (5-7 years). Am I wrong here? I assume yes. Or are CPUs must stronger/tougher than the bad old days?
  - jansommer 4mo ago
    A quick search on Xeon production yields that it goes through a rather rigorous testing. I wouldn't be surprised that server cpu's in a desktop pc works longer. I can't overclock it either, and that probably helps with its lifespan as well. But yeah, the fact that it actually powers on when i click the button and isn't a limiting factor after 10 years is quite something.
    - mrmlz 4mo ago
      Back from my old overclocking days - its heat that kills life. And if you keep that under control (what ages is the heatpaste, replace it ever so often) i very much doubt you'll have any life issues from the cpu itself. Bearings in fans, caps etc. are also stuff that you need to keep an eye on. I just replaced a i5-660 thats been powered on since 2010 24/7, heatpaste was fucked so it crashed during heavy loads :)
    - throwaway2037 4mo ago
deng 4mo ago
Nice post and technically impressive work. I agree we need to understand the build pipeline and be able to do things locally. However, depending on your electricity cost, it might not make sense financially. These old servers are not energy efficient at all (I'm guessing that old Xeon server will easily pull 200W on load), and that model is currently at 0.1$/0.3$ per 1M tokens (with 76 tps and 262k context) in Openrouter (also, these servers are LOUD). EDIT: I stand corrected, 200W is apparently way too high of an estimate. I used to run a bunch of old Xeon servers and they slurped watts like crazy, but I can't remember which ones exactly those were.
- jansommer 4mo ago
  It should be closer to 85W on load. And it's incredibly silent on even a low end cooler. I rarely get above 50° Celcius.
  - deng 4mo ago
    OK, then you're in luck. I had a bunch of old 1U rack servers and even in the next room it was too annoying to run them (they had a bunch of 40mm fans which always ran at full speed, because in a server room, no one can hear you scream).
    - jansommer 4mo ago
      Could it just be really bad cooling? Looking at 9800X3D, it seems like it's running in a similar range wrt TDP unless you really push the 9800X3D. I'm comparing with desktop cpu's because that's what my workload is. cpu governor is set to performance (no schedutil). No audible change in fan speed during heavy compilation or gaming (very silent humming), and i don't have any fans beside cheap intake, cpu and exhaust fans (1 each) + an excessive amount of dust.
      - deng 4mo ago
        These servers had no fan control whatsoever, they always ran full blast. That's not untypical for rack servers, because as written: they are designed for server rooms, and you're supposed to wear ear protection there anyway... Yes, I could've modified them, but I ditched them because running them simply made no sense (especially the high idle power consumption was ridiculous).
rvba 4mo ago
As someone doing this for fun on a windows 11 machine (96gb ram, 5090 24gb) I wonder if I need any flags to keep the model in memory and avoid swapping to ssd? I use LM studio and qwen3.5 35B - but never figured out if it is swapping or not. Om am unrelated note, does anyone know a model that can help with this use case: https://news.ycombinator.com/item?id=48301635 https://news.ycombinator.com/item?id=48301635
- smw 4mo ago
  The article talks about using --mlock
ezconnect 4mo ago
When you use page up and page down key when reading that blog the first line on the screen is obscured by the floating bar or what ever it is. It is not even needed for reading.
Hasan121212 4mo ago
I think one overlooked advantage of older Xeon systems is their availability. Many people can experiment with local AI deployments at a fraction of the cost of building a brand-new setup.
FartyMcFarter 4mo ago
I may have missed this in the article, but: What was the net effect of the optimisations? How much faster did it get?
robotswantdata 4mo ago
Granite or sapphire rapids are very under rated for MoE inference loads. But you need a GPU for the KV cache. Plus many boards also support CXL for RAM expansion over PCI 5! Source: building a hybrid inference business for regulated industry workloads.
coldcity_again 4mo ago
This is great work. I'd love if anyone knows how I might fare with an old Dell R710 with 2 x Xeon 5600 (12 cores total) and 96Gb of DDR3.
- rythie 4mo ago
  I don’t think it would work as well as there is no AVX or AVX2 on those older CPUs unfortunately.
  - coldcity_again 4mo ago
    Thanks very much. I'd forgotten that these were Westmere generation! Experimenting anyway; at least the RAID controller is behaving, and Ubuntu 26.04 LTS has gone on cleanly.
deleted 4mo ago
[deleted]
alimbada 4mo ago
What's the best way to apply this to slightly more modern hardware - i.e. 5800XT 32GB DDR4, 9060XT 16GB?
cmiles8 4mo ago
We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.
- herval 4mo ago
  this is sorta like saying that being able to run your blog on your laptop will completely implode the cloud business
  - Kinrany 4mo ago
    More like implode proprietary blog hosting platforms and replace them with commodity VMs that can be used for blog hosting, among other things
  - grumpymuppet 4mo ago
    It's a little different because cloud and blogs didn't actively get in the way of your home compute. To wit, the various cost spikes for hardware. People -- WANT -- this technology on their home devices and (apparently?) the providers of this tech don't seem to be running a profit so they probably don't want the maintenance tail on their side either. I think it's a bit different. Inevitable that this becomes a household-run thing? Not likely.
  - asimovDev 4mo ago
    Wouldn't arcade cabinets vs home video game consoles be a more apt comparison?
  - malmz 4mo ago
    Running an LLM locally is theoretically viable. Running your blog on your laptop is never viable (unless you hook it up like a server). One just requires compute while the other a stable network.
    - Scoundreller 4mo ago
lreeves 4mo ago
Doesn't accepting 100% of the MTP draft tokens mean you should just be using the smaller model? Usually the acceptance rate in Qwen36 at least is around 60-70% and the "wrong" tokens are still filled in entirely by the base model, but when you just accept 100% of the draft tokens it seems kind of self defeating unless I'm wrong. Also I feel like everyone leaves off prompt processing/prefill speeds in these articles. If you are using a very small prompt and asking for mostly generated tokens, sure but I'd love to know the time-to-response of asking for an analysis of an image or a few hundred lines of code.
- dvdkon 4mo ago
  As far as I know, speculative decoding still verifies that the proposed tokens are what the "big" model would generate, it just uses the guesses to make that process faster. Setting the probability threshold too low then shouldn't affect correctness, just speed (time will be wasted verifying bad guesses).
  - lreeves 4mo ago
    But won't setting it to accept 100% of the proposed tokens will skip the verification?
- naasking 4mo ago
  It depends on the type of MTP. If you're using two models, draft + full, then arguably yes, the larger model isn't providing much benefit if you really are seeing 100% acceptance rates. There are other forms of speculative decoding that work within the larger model by itself though, eg. Qwen has additional speculative decoding attention heads, so there is no secondary drafting model.
- ac29 4mo ago
  None of those settings set the speculative decoder to accept 100% of drafted token. I assume you are looking at --draft-p-min 0.0, if so, you are misunderstanding what it does.
montroser 4mo ago
Result is ~12 tokens per second, as reported by OP down in these comments here. An impressive effort, and better than I would have thought possible on this hardware -- but still pretty far short of what one needs for an satisfactory interactive session.
- andix 4mo ago
  Especially if you consider those smaller models are really cheap and fast on platforms like openrouter. Often by the factor 100-500 cheaper than SOTA models, and 2-5x in TPS.
- causal 4mo ago
  Yeah took way too long to find that result. Being able to run on slow RAM isn't surprising considering you can run a model off an SSD.
- greenavocado 4mo ago
  I was about to ask that
- gowld 4mo ago
  Right. You can also perform RSA encryption on pencil and paper with a scientific calculator. It works, but it's not useful throughput for serious work
- kingnothing 4mo ago
  It's not terrible for interactive... https://mikeveerman.github.io/tokenspeed/?rate=12&mode=text https://mikeveerman.github.io/tokenspeed/?rate=12&mode=text And it should be just fine for plenty of background use cases.
cbdevidal 4mo ago
Old hardware is surprisingly effective. I've been considering a side hustle selling offline AI to local businesses who are privacy-sensitive. Medical, legal, places like that. At the low end, I'd use old Xeons with gobs of DDR3, install some V100s, run a smaller agent for general chat inquiries, and a frontier model for the deeper stuff, with a router that passes between them depending on the complexity. The frontier model would perform very slowly, but if it's a deep task the user can submit it in a batch in the evening e.g. "Correlate all of these cases and look for patterns" then receive the output with morning coffee. Of course, AI helped me work out a plan for this. Haha
- nicogentile 4mo ago
  [flagged]
RobotToaster 4mo ago
Apparently Itanium works quite well for LLMs https://medium.com/@tglozar/running-llama-inference-on-intel-itanium-part-1-be62ff3f5c2f https://medium.com/@tglozar/running-llama-inference-on-intel... Which makes sense I suppose.
Liftyee 4mo ago
Very intriguing. This might be the use for my e5-2430 V2 X2 server that's been lying around. DDR3 is (relatively) cheap now too. Could fit 192GB of RAM in it and play around for much cheaper than a new GPU.
ForOldHack 4mo ago
Well, lets get started. I have 4 of those machines, and they are Two dual processor. They all had 32GB of ram, so now I have two with 64GB, and two with zero. They all hand stock K5000s, now how two have two cards. I stripped the uni processors ram and video cards, and put those into the dual procs. They have 256Gb SSDs, and two 1TB disk drives. One machine has 8Gb of VRam across two cards. Dual processors are 8Cx2 and 32 Threads. They can easily play 16 videos at once. For AI, I have not found a model that I can get above 3 tokens a second. Not a one.
- nixgeek 3mo ago
  That’s because all of that hardware probably dates back to when Steve Jobs roamed the planet.
SirMaster 4mo ago
Either they have a E5-2620 V2 from 13 years ago, or they have DDR4, not DDR3. The V3 and V4 only support DDR4.
- lolcw 4mo ago
  No they don’t. Models ending in 6 have DDR3 controller. For example E5-2696 v3/v4 E5-2686 v3/v4 E5-2666 v3/v4 also 2673 v3/v4 2678 v3/v4 as well as E5 2629 v3 E5 2649 v3 E5 2669 v3
sperandeo 4mo ago
ive been doing the same thing. i refactored a old newtek stream machine . its my new favorite thing to do! adding old PCs to my "starcraft" fleet xD
bombcar 4mo ago
Is this John Siracusa? It sounds like it could be something he’d say… (He has a fully maxed out “last Intel” Mac Pro and laments the lack of replacement).
kristjansson 4mo ago
Noting for reference that Gemma4 MTP work is in progress[0] on llama.cpp; similar work for Qwen3.6 landed recently and has been great thus far. [0]: https://github.com/ggml-org/llama.cpp/pull/23398 https://github.com/ggml-org/llama.cpp/pull/23398
mv4 4mo ago
I have an old 192GB DDR4 Dell Precision with dual Intel Xeon Gold 6130 that I've considered spinning up. What's giving me pause is 250W at idle.
- mtoner23 4mo ago
  Surely that number can go lower with some tweaks
  - mv4 4mo ago
    I am sure it can. It will still generate a lot of heat when under load. Are you telling me I should go for it? :) I do have a dual DGX Spark cluster running MiniMax M2.7 already so I am all for on-prem. But will be interesting how this old machine will perform!
qingcharles 4mo ago
Would there be any advantage of running this as dual Xeon? The CPUs are $5 and a dual mobo is $50...
- bee_rider 4mo ago
  More memory bandwidth presumably. Not sure how well the ecosystem handles thread pinning though.
ryandrake 4mo ago
I've got an old HP Z-620 workstation with dual E5-2697 v2 CPUs (24 cores total, 48 threads @ 2.7GHz) and 128GB of DDR3 RAM. The docs say it supports up to 192GB, but I wasn't able to get it to POST with all the RAM slots full. It's still a "homelab" beast and does great with development and GIS/Mapping applications. I was not able to figure out how to run AI workloads on it with decent performance, however, so I finally broke down and got a dedicated GPU for it. It's pretty great what can still be done with older hardware.
- FpUser 4mo ago
  I self host on old HP Z-840 with 2x3.6 GHz Xeons 24 total cores, and 512 GB RAM. Cost me peanuts used and works like a charm for many years already
- bobmcnamara 4mo ago
  I'm in the same situation of having an older workstation nearly maxed out with RAM and neither wanting to pay for the equivalent RAM on a new system neither go down in GBs.
6_7 4mo ago
[dead]
andai 4mo ago
I want to share something strange. I found a typo or two in the post and this absolutely delighted me, because it implies a human wrote the words. (Or was at least heavily involved in the editing.) Guess I am a species-ist after all ;)
- bicepjai 4mo ago
  I hope LLMs don’t get trained with this reply and start adding typos for making it look like it came from a human :)
  - andai 4mo ago
    I felt like I had lost something valuable when I switched to mostly AI based programming, because I used to make so many mistakes that the computer would often do truly magical things I did not even realize were possible. e.g. one time I tried making a collaborative drawing application but I messed up the logic, and the brush strokes would just get temporarily mirrored between the client and server, so you'd see it getting drawn over and over again in a loop. The drawing wasn't stored anywhere, it existed only in the network packets between client and server. Accidental GNU. http://www.gnuterrypratchett.com/ http://www.gnuterrypratchett.com/ So I started working on a tool that adds random errors back into my programs. To reintroduce the possibility of such happy little accidents.
  - gowld 4mo ago
    AIs already make typos, not directly intentionally. Since they are token-based, and tokens are lexemes, they can misconjugate works or make grammatical errors.
danbruc 4mo ago
Did some try to estimates what it would take to bake interference for a capable large language model into silicon so that one can pipeline inputs through it and produce outputs at one token per clock cycle?
- knorker 4mo ago
  I'd expect it to require too much RAM bandwidth to be feasible. RAM is really slow at silicon speeds. Very little is reachable in one clock cycle, unless the clock cycle is abysmally slow.
  - danbruc 4mo ago
    No RAM. Instead of having a general purpose multiplier that multiplies an input with a weight stored in RAM, just have a multiplier that hardcodes the weight. In some sense replace each weight with a specialized multiplier and wire them together with accumulators and activation functions in between. And some registers for pipelining. If one goes for four bit quantization, one could have sixteen optimized multipliers, one for each possible weight, and the one just selects and connects them according to the model weights and structure. Example. If you have a neuron with 16 inputs each 8 bit wide and with a 4 bit weight per input, you will have 16 specialized multipliers each scaling its input by the corresponding weight and then the 16 scaled inputs feed into an adder tree and finally an activation function.
    - thomashabets2 4mo ago
      That sounds like wiring the RAM information into order of magnitude same number of transistors. A modern CPU has (quick googling) 184B transistors. If they were bits then that's 23GB. But presumably a model bit needs more than one transistor to represent how it acts as a neuron with its interactions. Then there's the current speedup in inference from restricting which subset of the model is used, which is not a "swap in" that would work with hard wired neurons. But I dunno. Maybe. I'm just guessing.
Aurornis 4mo ago
llama.cpp includes a benchmarking tool called llama-bench https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md https://github.com/ggml-org/llama.cpp/blob/master/tools/llam... ik_llama includes llama-sweep-bench https://github.com/ikawrakow/ik_llama.cpp/blob/main/examples/sweep-bench/README.md https://github.com/ikawrakow/ik_llama.cpp/blob/main/examples... When comparing hardware, the output of these tools is very helpful to let others put it into context. The post says the output is "reading speed" but knowing the prefill and token generation speeds would be a lot more helpful.
tomega2134 4mo ago
I wish this were somehow tagged with AI, so I would know that it's not about say, general computing or cost-efficiency (e.g. using an old xeon machine from ebay instead of new, in these cost-conscious times.) As it is, the title is click-bait for me, as 1) it says I need at least a Xeon somehow and 2) as it doesn't say what I actually need it for.
maxothex 4mo ago
[flagged]
api 4mo ago
Have to point out one boring thing though: this will use a lot more electricity than newer things. So it'll work, but it'll run up your electric bill.
bitwize 4mo ago
Successfully ran Gemma4-26B-A4B on my 8yo first-gen Ryzen with a GeForce GTX 1070. It actually ran acceptably well; I was surprised. I even did some coding with it, but the wheels fell abruptly off when it tried several times to use a constant I told it doesn't exist. I only have 32 GiB of RAM in this old bucket, and these results are not worth the RAM consumption, so I put it aside. Maybe if I finish that build with more memory...
b65e8bee43c2ed0 4mo ago
so how many tokens/s do you get, pp and tg? did I miss it in the article?
npn 4mo ago
I bought one AMD MI50 32GB back then when they were sold rather cheap (around $150-$170). it can easily generate over 70 tokens per second for gemma 4 26B moe model (q4). I have no doubt that we will have another wave of cheap retired server gpus just like before. And that is the time when everyone will have their own models at their home. Or we can just buy the newest medusa halo mini pc. they will be pretty decent, too, albeit pricey.
hualapais 4mo ago
Went this route after hemming and hawing over a Mac Studio Pro for some time. Eventually bought and configured a headless HP Z620 with 192 GB of ECC RAM and dual Xeon E5-2680 v2 processors, an Optane AIC, two P102-100s with 10 GB VRAM each, and a minimal bootable SDD running Debian 12.6 with an older, locked version of CUDA that supports the Pascal cards. Run it remotely from the basement via AMT/meshcommander. Just fire up llama.cpp and its front end and connect over the local network. Currently playing with Talkie, Qwen 3.6 27b, and medgemma, but have had good luck with GGUF performance in general after selecting an appropriate quant. Total cost was under $500, but I bought the server via eBay last year; things may be different now. Details aside, the hope is that ternary LLMs blossom in the coming months and this old hardware can eventually host some very dense models full of factual information, perhaps even larger than the GPU RAM and spilling over to the Optane for IO. Speed would be less important than general factual knowledge. The plan would be to configure then mothball the machine in a Faraday trashcan in the basement, retaining it as a possible "rebuild civilization" oracle should the world fall apart. Of course, power would be an issue in such a scenario, but for how cheap this hardware is and how often AI seems to be practically useful in its latest iterations, why not...
fortran77 4mo ago
My current desktop machine is a 24-core Xeon-3345 with 256GB of RAM and an Nvidia 5090. It still feels extremely fast, even though it's about 8 year old technology with a newer video card.
1970-01-01 4mo ago
Hah. My Xeon turns 20 this year. No issues.
dzonga 4mo ago
for solo operators that run saas (targeting business customers) & if you do a lot of data processing - old servers are the best bang for the buck. remember if you serve real customers as a bootstrapped business - you can afford the whole serve down for maintenance. no need for 99.999%. better than hetzner.
rbanffy 4mo ago
The other day I was considering the adoption of a POWER7+ box. Sadly, Linux hasn't supported POWER7 in quite some time. The machine looked pretty nice, with 4 CPUs with 8 cores each, a total of 128 threads and 512 GB of RAM. I'm not sure it'd run AIX without a license though, which is unfortunate - it's a gorgeous box.
Floppyrom 4mo ago
Famous last words...
rldjbpin 4mo ago
the surge of articles on using decommissioned datacentre hw to run LLMs lately, is more of a symptom of the times than their viability. back when intel had a monopoly on cpu and would refuse to give consumers more than four cores, the old xeon route was popular for a different reason. memory is the bottleneck here (capacity, or rather speed). before you run out to set up your own, try to rather squeeze out the most of your existing hardware. if you are a lucky owner of a lot of cheap memory, you are already in luck. otherwise LM studio allows you to split memory between your gpu and system memory. avoid MoE models or even consider tensor parallelism between the onboard gpu and dedicated one before going for more hardware. there is little to no benefit for using a specific quantization for your models, so go crazy and test out whatever can easily run for you.

740 points by cafkafk 4mo ago

bflesch 4mo ago

Might consider going for even older CPUs which don't have the Intel ME ring -3 thing which is full of backdoors

bflesch 4mo ago
I appreciate the downvotes without any reasoning. It's a fact that newer Intel CPUs have Intel ME which was not in older CPUs and significantly increases attack surface if you are not living in a five eyes state.
- s20n 4mo ago
  I agree with the first part. I think this article by FSF about Intel's ME summarizes the issue https://static.fsf.org/nosvn/blogs/Intel_ME_Carikli_article_PRINT_2.pdf https://static.fsf.org/nosvn/blogs/Intel_ME_Carikli_article_... As for the second part, I am not sure about how living in a five eyes state would mitigate it. What do you mean by that?
  - bflesch 4mo ago
    As five eyes citizen you have at least some rights on paper and you can appeal to your government, but if you are foreigner these guys can go gloves off without any fear of retribution. Try analyzing Epstein files and posting about it, they'll give you a proper penetration test of all your devices to see what you found out about their ex employee. Nowadays even EU citizens migrating away from US cloud providers are a "national security issue".
    - smilespray 4mo ago
      Isn't the whole five eyes argument moot because member states spy on citizens from the other countries and trade intel with each other?
      - bflesch 4mo ago
        No need for that charade if you are a foreigner, even from NATO ally.
- tryauuum 4mo ago
  How old are we talking?
  - bflesch 4mo ago
    IIRC it is pre-2008.
- adrian_b 4mo ago
  In a server, you have to worry about the ME only if you also have an Intel Ethernet interface, which is connected to a potentially hostile network. If that is not true, the ME cannot be controlled remotely. The existence of the ME is much more worrisome in laptops, where the ME can be accessed remotely through WiFi. There, to be certain that there is no way for the ME to be accessed remotely you would have to disconnect or cut the internal antennas and use a USB dongle for WiFi.

shovas 4mo ago

I have run llama.cpp on an i7-2600 with a 1050. It's too slow for everyday usage but it's not too slow to make it obvious AI is going to be everywhere and in everything. It's too easy to run.

cafkafk 4mo ago

Hi HN. I wrote this post after getting frustrated by the lack of ways to run the new Gemma 4 Drafter models, and mainstream tools not prioritizing this, and hiding all the performance levers. I ended up getting a modern 26B MoE model (Gemma 4) running at reading speed on an old recycled server with a single Xeon E5-2620 v4 and 128GB of DDR3 RAM (and no GPU). It took a lot of work, but it actually worked out somehow. I've also linked the quants at the end, but they're not gonna run unless you use the ik_llama-cpp fork I mention, see other posts for more details. I'm not an ML engineer, so I'm by no means an expert, and the server is busy acting as a Nix cache, but if you have any question, I can try to answer, but best effort.

fragmede 4mo ago
(purple on black is really hard to read) You say it runs "at reading speed". Have you benchmarked it?
- cafkafk 4mo ago
  > (purple on black is really hard to read) Noted, and agree (it looks like it has also already been clicked, which I dislike). I honestly I need to redo the themes. > You say it runs "at reading speed". Have you benchmarked it? At some point a few weeks ago, yes I think so, but I didn't write it down for some reason... so I'll have to find a time when it's not busy and do it again without a noisy system. Right now the system is noisy, but that said doing it like this: llama-cli --model gemma-4-26B-A4B-it-Q8_0.gguf --model-draft gemma-4-26B-A4B-t-assistant-GGUF/wikitext-2-raw_ik-llama-mtp_drafter-conservative/gemma-4-26B-A4B-it-assistant-Q8_0.gguf --spec-type mtp --draft-max 3 --draft-p-min 0.0 --color -sm graph -smgs -sas -mea 256 --split-mode-f32 --temp 0.7 --cpu-moe -t 8 --flash-attn on --mla-use 3 --merge-up-gate-experts --special --mlock --run-time-repack --spec-autotune --no-kv-offload --parallel 8 --jinja -p "Why is the sky blue?" -n 128 Gives: llama_print_timings: load time = 83911.65 ms llama_print_timings: sample time = 26.99 ms / 128 runs ( 0.21 ms per token, 4742.15 tokens per second) llama_print_timings: prompt eval time = 343.41 ms / 7 tokens ( 49.06 ms per token, 20.38 tokens per second) llama_print_timings: eval time = 10639.36 ms / 127 runs ( 83.77 ms per token, 11.94 tokens per second) llama_print_timings: total time = 11114.98 ms / 134 tokens So 11.94 tokens per second while it's also playing binary cache and CI builder. When I do it properly, I'll add it to the blog as well!

Eonexus 4mo ago

I wonder what the tokens per second actually are. Yes, it does say "reading speed" but that varies for everyone, no?

cafkafk 4mo ago
That is a very fair point! I just ran a not very scientific benchmark with the system under load, and posted the raw logs in a sibling comment above, but the short answer is that it's hitting 11.94 tokens per second for generation - while it's also being a binary cache and CI build server. Totally just vibes based, I think it goes up to 20+ tps when it's not under load (and that's me trying to be conservative). For context, reading speed at 250 wpm would be around 5 to 6 tokens per second.
- Eonexus 4mo ago
  Huh, that's actually not bad at all! Sure, it's not at the speed of a GPU, but still, 20 tps is cromulent for a CPU.

potus_kushner 4mo ago

@cafkafk got a recommendation for a good model that fits into 64GB and leaves a couple GB free for other tasks ?

cafkafk 4mo ago
Honestly, at this point you're probably looking at a smaller model, for the Gemma series I'd go with Gemma 4 E4B with drafters, but that's just a hunch from using it on my laptop (where I do have a RTX 4060 M and 96gb ram). So you'd change the invocation slightly here, but a lot of things you can potentially reuse. That said, the Gemma 4 E4B models have so far in my experience been... not great when it comes to long context, but they are very passable for basic tasks, and even seem surprisingly okay at tool calls.
- potus_kushner 4mo ago
  i tried the Q4_K_M model form unsloth with your Q4_K_M drafter, but the required memory to load everything is 72GB. odd. otoh i could load Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled.IQ4_XS.gguf and it requires just ~18 GB: ~/ik_llama.cpp[main]$ build/bin/llama-cli --model ~/models/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled.IQ4_XS.gguf --spec-type mtp --draft-max 3 --draft-p-min 0.0 --spec-autotune -cnv --color --jinja --special -smgs -sas -mea 256 --temp 0.7 -t 6 --parallel 6 --cpu-moe --merge-up-gate-experts --flash-attn on --mla-use 3 --mlock --run-time-repack --no-kv-offload . works pretty fast, at about 15 t/s: llama_print_timings: sample time = 45.28 ms / 404 runs ( 0.11 ms per token, 8921.67 tokens per second) llama_print_timings: prompt eval time = 949.42 ms / 51 tokens ( 18.62 ms per token, 53.72 tokens per second) llama_print_timings: eval time = 24067.08 ms / 400 runs ( 60.17 ms per token, 16.62 tokens per second) llama_print_timings: total time = 242192.55 ms / 451 tokens so i wonder why the params used by the quantified qwen model use way less memory than the ones of gemma.
- sleepyeldrazi 4mo ago
  Have you tested Qwen3.6 35B? Putting aside the capability claims for that model (which I support, but are not my point here), that 35B has smaller active parameter count than the gemma 4 26B, potentially making both prefill and decode faster out of the box, and has MTP heads built in the model and well supported (you may need to make sure you download a quant that didn't strip them off, as some do to preserve space). I would be curious to see your numbers there too. And if you do test this, please go for a clean one and not a fine-tuned one.

christkv 4mo ago

Makes you wonder if its possible to squeeze more tps out of a strix halo system using the 16 zen5 cores as well as the gpu.

cafkafk 4mo ago
If you get the inference engine to route the heavy matrix math to the GPU and the speculative drafting to the CPU without choking on latency it's probably gonna be very fast. Would love to see the benchmarks if someone actually pulls something like that off.
Havoc 4mo ago
In general you’re mem bandwidth constrained so cpu vs gpu often ends up similar on APUs
- fulafel 4mo ago
  There are ways to trade off compute power for memory bandwidth (like MTP and other speculative decoding approaches). The CPU and GPU would need to be able to share the same cache for this to work. In the Strix Halo case the GPU has a private cache on the GPU die I think, which is the snag.

asimovDev 4mo ago

I have an ancient DDR3 Xeon that doesn't support any AVX (dual x5690 and 96GB 1333 MHz RAM). You reckon it would even build / run at all?

tgtweak 4mo ago
It may work - depending on your ram speeds it might not even be that much slower.
cafkafk 4mo ago
Loading will take some minutes, but at 96 you can squeeze the model in and have some headroom around like ~10 GB, although depending on the Xeon, you may have to downgrade to E4B instead. Should still work thou.
qwertox 4mo ago
CPU (2012) Model name: Intel(R) Xeon(R) CPU E3-1265L V2 @ 2.50GHz Mainboard Product Name: P8Z77 WS GPU 05:00.0 VGA compatible controller: NVIDIA Corporation AD106 [GeForce RTX 4060 Ti 16GB] (rev a1) 05:00.1 Audio device: NVIDIA Corporation AD106M High Definition Audio Controller (rev a1) Memory: 32GB This works.
burnt-resistor 4mo ago
I run Win 11 Enterprise on an el cheapo spare parts Xeon E3-1275 V2 + 32 GiB DDR3-2133 + Gigabyte GA-B75M-D3H rev. 1.2 (TPM support)

gigatexal 4mo ago

What kind of tokens per second did the op get I saw nothing of this written.

urbandw311er 4mo ago
11.94 tokens/sec (from another answer above)
deleted 4mo ago
[deleted]

vhaudiquet 4mo ago

The E5 2620-v4 only supports DDR4.

bobmcnamara 4mo ago
Probably in an x99 motherboard
- mwpmaybe 4mo ago
  The memory controller is integrated into the CPU, so the motherboard chipset is irrelevant. There are some OEM-only v3/v4 parts with dual memory controllers, but the E5-2620 v4 is not one of them.
  - bobmcnamara 4mo ago
    Ooh weird!

NSUserDefaults 4mo ago

How about the iMac Pro? Would that work? I was able to put 128gb in it (not as easy as the regular iMac but possible).

wazoox 4mo ago
I've been running various models on a Mac Pro 2013 (8 cores, 32 GB RAM) at about 8 to 10 t/s for months. It's not fast, but it's more than enough for many actual tasks, in particular background tasks. An iMac pro will do just as well I suppose.
- fooker 4mo ago
  What are the tasks that do well with 8-10 t/s ?
  - wazoox 4mo ago
    The sort of task you don't expect to end immediately. If extracting data from a bunch of PDFs takes 1 hour or the whole night, that doesn't make much difference to me. It's not fast enough for auto completion and slightly too slow for chat (but bearable IMO).
    - fooker 4mo ago
      Running a local llm at 10 t/s overnight to extract data from a few PDFs will burn more in electricity than paying cents for the hosted kimi models. You can (sometimes) break even if you have a workstation GPU.
      - wazoox 4mo ago
        Sometimes data privacy is paramount.
- neverartful 4mo ago
  I have and use a Mac Pro 2013 too. Mine is 8 cores with 64 GB RAM. I haven't used mine for any LLM workloads, but it does just fine for most stuff. My biggest concern with it is the OS. I'm still running macOS (the latest supported version) but it's getting continually further out-of-date security wise all the time.

nurettin 4mo ago

I also run a Qwen 3.6 moe A4B on old hardware. I set it up with numactl --membind=1 so it is constrained to one of the memory sticks which speeds up token generation a little.

hparadiz 4mo ago

I'm now staring at a 10 year old 4U with 256 GB of DDR4 and thinking hmmmmm

hypfer 4mo ago

> The argument for speculative decoding is stronger on CPU than on GPU. Uh. Uuuh. No? ___ Also > While a GPU has a massive pool of ultra-fast High-Bandwidth Memory (HBM), a CPU relies on small, lightning-fast “caches” (L1, L2, L3) built directly onto the processor chip. What purpose does the quoting of "caches" serve there? Is this AI writing written by that model running on that host?

phaser 4mo ago

What intrigues me the most about AI progress, is not AGI or the model du jour by $AI_UNICORN, but rather what can be run locally. I remember having an amusing, but rather useless model in a beefy gaming PC that I had 6 years ago; and now, something that’s a hundred times better on my M5 laptop. Should the market react to the memory shortage, the progress of the Apple silicon continue at the same pace, and what we’ll be able to run locally in 6 years will be very exciting. or frightening. Also I don’t know what this means for the valuation of the AI companies. I remember asking about this very idea to one of their employees at an event and instead of answering he bailed out to grab a cocktail.

skdb476 4mo ago
Its a convenience thing. You can run a whole lot of stuff locally from wikipedia to social media/email/video servers whatever. Most people with a full time job and 2 kids dont do it cause who has time and energy to patch and maintain the ever growing complexity of this stuff. These systems will keep growing complex. That also means more bugs. Age old tradeoff between freedom and convenience.
- SadErn 4mo ago
  [dead]
- phaser 4mo ago
  You can run mediawiki at home but you won't have wikipedia. You can run a video server but you won't have all the movies that Netfix has. A local model is actually the real thing.
  - skdb476 4mo ago
    you can have the whole wiki loaded with full search available locally. check out kiwix.
    - phaser 4mo ago
      Thanks I didn't know about kiwix, but, let's consider the fact that a wiki, or netflix movies are cheap or free, while AI is actually quite expensive at least for now, and i'm not sure if it's because of real costs or to justify the valuation. So there is a bigger incentive to run locally something that's gonna get you $20 or $100 worth of bills to OpenAI than to mirror something that is actually free. Example: In the past there was a whole market for sound cards, if you wanted your computer to have any "multimedia" capabilities you needed to get a sound blaster but now everybody assumes a computer will produce sound, and it's basically for free as all chips have it. Now sound interfaces are still a thing but only for audiophiles who are esoteric enough like me to believe that it's worth to have that extra hi-fi quality. What I think it could happen, is that eventually AI will be part of all the chips, just like soundcards. And there will be people who will buy specialized AI from companies that perhaps are not OpenAI or Anthropic but second-generation sleepers who watched the carnage in the market and decided to enter when it was reasonable. This could be Apple, or Nvidia or something new. They're just waiting for the others to do the research and introduce the taste for it to the masses, just like sound blaster made us fall in love with high fidelity sound in our computers.

car 4mo ago

Similar recent posting with optimizations for older Xeon: High-Performance AI on a Budget: Optimizing llama.cpp for Qwen3.5 Inference on a Dual-GPU HP Z440 https://news.ycombinator.com/item?id=47320244 https://news.ycombinator.com/item?id=47320244

cykros 4mo ago

Does this mean my 15 year old Phenom is too old? But it has 16 gb of DDR3 RAM! Admittedly web browsers and it don't get along that well. Literally the only thing that drags though on my Slackware 15 system, and even then usually only when it gets to around 15 or so open tabs.

SXX 4mo ago

Now we need someone try run Kimi K2.6 on old Xeon and DDR3. After all these platforms do support up to 768GB RAM.

Havoc 4mo ago
It’ll work but yield a token per minute. With ancient servers the throughput is the limiting aspect not mem size
segmondy 4mo ago
You can run these on a turing machine. At what point is it not worth it? At some point the energy to generate each token matters. We often seen token per second. I think a missing metric is tokens per kilowatt. That is what really matters.
- SXX 4mo ago
  This is just like running Crysis via software rendering on CPU / llvmpipe. It dont have to be practical in order to be fun to try.

egorfine 4mo ago

This and the previous one are insanely good articles. Thank you!

haunter 4mo ago

And this is one of those CPUs which had dual slot motherboards so you can have double the fun (and power bill) https://pcpartpicker.com/products/motherboard/#s=20028,20029&m=8,18,27,46 https://pcpartpicker.com/products/motherboard/#s=20028,20029...

anon-3988 4mo ago

I tried to run gemma 4 on this CPU and it did not go well https://www.techpowerup.com/cpu-specs/ryzen-7-4800u.c2281 https://www.techpowerup.com/cpu-specs/ryzen-7-4800u.c2281 It is way too slow

throwaway2027 4mo ago

Glad to see other people realizing this. I've been running Gemma 26B-A4B Q4 on a 2012 Xeon with 16GB to 24GB of RAM in a container. It's getting around 8 to 12 tokens per second. Obviously it's not comparable to huge contexts and running it on a GPU and the image decoder in llama.cpp is super slow compared to a GPU but for some small automation tasks and general trivia questions it's decent. The speed is just enough to not have to wait for it to finish so you can read along. Here's my setup. You may want to figure out what the best optimizations are for your specific CPU like AVX2 because mine didn't have most of them. I did try MTP briefly but I wasn't getting performance improvements. You could play around with the batch sizes for cache or context or go even lower for Q2 and don't overcommit on threads either, but I would suggest either defaults or trying out llama-bench. This isn't by any means the best I assume but it worked decently for me and I sometimes swap out Gemma for Qwen. You could also lower q8_0 to q4_0 for more context but it could hurt quality some say, altough I have noticed it too on some models. # Building cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS -DGGML_OPENMP=ON # Running export OPENBLAS_NUM_THREADS=4 export OMP_NUM_THREADS=4 OPENBLAS_NUM_THREADS=4 OMP_NUM_THREADS=4 \ llama.cpp/build/bin/llama-server -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL --temp 1.0 --top-p 0.95 --top-k 64 --min-p 0.00 --jinja --host 0.0.0.0 --port 8080 --cache-type-k q8_0 --cache-type-v q8_0 --threads 4 --threads-batch 4 --ctx-size 8192 -n 8192 --batch-size 2048 --ubatch-size 512 --no-mmap --mlock --chat-template-kwargs '{"enable_thinking":false}' --no-mmproj -np 1 -fa 1

duffyjp 4mo ago
I'm setting up a Frankenstein system at the moment. It's a Chinese DDR3 X99 motherboard with a 12 core Xeon v3, 32gb 1866MT/s ram, and a 1080 Ti. I'm shoehorning it back in the Optiplex that donated the ram, so it's not ready to go at the moment, but when I had it running on top of the motherboard box as a test I ran the (9B?) gemma4:e4b-it-q4_K_M since it can fit entirely in the 11gb vram. It flew, more than 50tk/s. A model that small isn't useful for coding, but there could be uses. I'd love to figure out a Wake-on-Use and use it as my personal ChatGPT. I'm not sure how that would work... Maybe proxy the LLM thru a Pi with a script to Wake-on-LAN the PC? It'll be a fun weekend project someday. My always-on LLM is the dense Gemma4:31b that's not quite half in GPU on a 12gb 2060. It's really slow, but the quality is great and my use case is an automated queue so I'm not sitting there watching the output. I have another 2060 but unfortunately the PC won't POST with both installed for some reason.

shevy-java 4mo ago

The webpage's layout is just horrible. Scrolling is also non-default - and thus rather annoying; I had to stop after two scroll events. Why do people think they need so much fancy effects or non-standard behaviour, if their alleged goal is to get information across to other people?

jansommer 4mo ago

The E5-2620 v4 is great. Have been using it for 10 years now. Wanted to upgrade until I saw current prices. I have 64 GB ddr4. Paired it with rx 9060 xt 16 GB and games run as fast as ever. Perhaps the cpu is a slight bottleneck in DOOM The Dark Ages, but i'm at 60 fps, so no problem. Light llm on the gpu is a nobrainer, and it's cool to see that things can be tuned to run ok on the cpu. I bought 2667 v4 a month ago for 30$. I'd expect it to give a decent performance boost but I just haven't had the need for it yet, but pushing into llm like in the article I'd probably upgrade because 2667 can handle slightly faster ram.

throwaway2037 4mo ago
> The E5-2620 v4 is great. Have been using it for 10 years now. 10 years? Damn, that is a long time. I always assumed that heat-induced damage will kill a CPU after a certain amount of time (5-7 years). Am I wrong here? I assume yes. Or are CPUs must stronger/tougher than the bad old days?
- jansommer 4mo ago
  A quick search on Xeon production yields that it goes through a rather rigorous testing. I wouldn't be surprised that server cpu's in a desktop pc works longer. I can't overclock it either, and that probably helps with its lifespan as well. But yeah, the fact that it actually powers on when i click the button and isn't a limiting factor after 10 years is quite something.
  - mrmlz 4mo ago
    Back from my old overclocking days - its heat that kills life. And if you keep that under control (what ages is the heatpaste, replace it ever so often) i very much doubt you'll have any life issues from the cpu itself. Bearings in fans, caps etc. are also stuff that you need to keep an eye on. I just replaced a i5-660 thats been powered on since 2010 24/7, heatpaste was fucked so it crashed during heavy loads :)
  - throwaway2037 4mo ago

deng 4mo ago

Nice post and technically impressive work. I agree we need to understand the build pipeline and be able to do things locally. However, depending on your electricity cost, it might not make sense financially. These old servers are not energy efficient at all (I'm guessing that old Xeon server will easily pull 200W on load), and that model is currently at 0.1$/0.3$ per 1M tokens (with 76 tps and 262k context) in Openrouter (also, these servers are LOUD). EDIT: I stand corrected, 200W is apparently way too high of an estimate. I used to run a bunch of old Xeon servers and they slurped watts like crazy, but I can't remember which ones exactly those were.

jansommer 4mo ago
It should be closer to 85W on load. And it's incredibly silent on even a low end cooler. I rarely get above 50° Celcius.
- deng 4mo ago
  OK, then you're in luck. I had a bunch of old 1U rack servers and even in the next room it was too annoying to run them (they had a bunch of 40mm fans which always ran at full speed, because in a server room, no one can hear you scream).
  - jansommer 4mo ago
    Could it just be really bad cooling? Looking at 9800X3D, it seems like it's running in a similar range wrt TDP unless you really push the 9800X3D. I'm comparing with desktop cpu's because that's what my workload is. cpu governor is set to performance (no schedutil). No audible change in fan speed during heavy compilation or gaming (very silent humming), and i don't have any fans beside cheap intake, cpu and exhaust fans (1 each) + an excessive amount of dust.
    - deng 4mo ago
      These servers had no fan control whatsoever, they always ran full blast. That's not untypical for rack servers, because as written: they are designed for server rooms, and you're supposed to wear ear protection there anyway... Yes, I could've modified them, but I ditched them because running them simply made no sense (especially the high idle power consumption was ridiculous).

rvba 4mo ago

As someone doing this for fun on a windows 11 machine (96gb ram, 5090 24gb) I wonder if I need any flags to keep the model in memory and avoid swapping to ssd? I use LM studio and qwen3.5 35B - but never figured out if it is swapping or not. Om am unrelated note, does anyone know a model that can help with this use case: https://news.ycombinator.com/item?id=48301635 https://news.ycombinator.com/item?id=48301635

smw 4mo ago
The article talks about using --mlock

ezconnect 4mo ago

When you use page up and page down key when reading that blog the first line on the screen is obscured by the floating bar or what ever it is. It is not even needed for reading.

Hasan121212 4mo ago

I think one overlooked advantage of older Xeon systems is their availability. Many people can experiment with local AI deployments at a fraction of the cost of building a brand-new setup.

FartyMcFarter 4mo ago

I may have missed this in the article, but: What was the net effect of the optimisations? How much faster did it get?

robotswantdata 4mo ago

Granite or sapphire rapids are very under rated for MoE inference loads. But you need a GPU for the KV cache. Plus many boards also support CXL for RAM expansion over PCI 5! Source: building a hybrid inference business for regulated industry workloads.

coldcity_again 4mo ago

This is great work. I'd love if anyone knows how I might fare with an old Dell R710 with 2 x Xeon 5600 (12 cores total) and 96Gb of DDR3.

rythie 4mo ago
I don’t think it would work as well as there is no AVX or AVX2 on those older CPUs unfortunately.
- coldcity_again 4mo ago
  Thanks very much. I'd forgotten that these were Westmere generation! Experimenting anyway; at least the RAID controller is behaving, and Ubuntu 26.04 LTS has gone on cleanly.

deleted 4mo ago

[deleted]

alimbada 4mo ago

What's the best way to apply this to slightly more modern hardware - i.e. 5800XT 32GB DDR4, 9060XT 16GB?

cmiles8 4mo ago

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use cases. That will completely implode what’s going on at the moment in tech.

herval 4mo ago
this is sorta like saying that being able to run your blog on your laptop will completely implode the cloud business
- Kinrany 4mo ago
  More like implode proprietary blog hosting platforms and replace them with commodity VMs that can be used for blog hosting, among other things
- grumpymuppet 4mo ago
  It's a little different because cloud and blogs didn't actively get in the way of your home compute. To wit, the various cost spikes for hardware. People -- WANT -- this technology on their home devices and (apparently?) the providers of this tech don't seem to be running a profit so they probably don't want the maintenance tail on their side either. I think it's a bit different. Inevitable that this becomes a household-run thing? Not likely.
- asimovDev 4mo ago
  Wouldn't arcade cabinets vs home video game consoles be a more apt comparison?
- malmz 4mo ago
  Running an LLM locally is theoretically viable. Running your blog on your laptop is never viable (unless you hook it up like a server). One just requires compute while the other a stable network.
  - Scoundreller 4mo ago

lreeves 4mo ago

Doesn't accepting 100% of the MTP draft tokens mean you should just be using the smaller model? Usually the acceptance rate in Qwen36 at least is around 60-70% and the "wrong" tokens are still filled in entirely by the base model, but when you just accept 100% of the draft tokens it seems kind of self defeating unless I'm wrong. Also I feel like everyone leaves off prompt processing/prefill speeds in these articles. If you are using a very small prompt and asking for mostly generated tokens, sure but I'd love to know the time-to-response of asking for an analysis of an image or a few hundred lines of code.

dvdkon 4mo ago
As far as I know, speculative decoding still verifies that the proposed tokens are what the "big" model would generate, it just uses the guesses to make that process faster. Setting the probability threshold too low then shouldn't affect correctness, just speed (time will be wasted verifying bad guesses).
- lreeves 4mo ago
  But won't setting it to accept 100% of the proposed tokens will skip the verification?
naasking 4mo ago
It depends on the type of MTP. If you're using two models, draft + full, then arguably yes, the larger model isn't providing much benefit if you really are seeing 100% acceptance rates. There are other forms of speculative decoding that work within the larger model by itself though, eg. Qwen has additional speculative decoding attention heads, so there is no secondary drafting model.
ac29 4mo ago
None of those settings set the speculative decoder to accept 100% of drafted token. I assume you are looking at --draft-p-min 0.0, if so, you are misunderstanding what it does.

montroser 4mo ago

Result is ~12 tokens per second, as reported by OP down in these comments here. An impressive effort, and better than I would have thought possible on this hardware -- but still pretty far short of what one needs for an satisfactory interactive session.

andix 4mo ago
Especially if you consider those smaller models are really cheap and fast on platforms like openrouter. Often by the factor 100-500 cheaper than SOTA models, and 2-5x in TPS.
causal 4mo ago
Yeah took way too long to find that result. Being able to run on slow RAM isn't surprising considering you can run a model off an SSD.
greenavocado 4mo ago
I was about to ask that
gowld 4mo ago
Right. You can also perform RSA encryption on pencil and paper with a scientific calculator. It works, but it's not useful throughput for serious work
kingnothing 4mo ago
It's not terrible for interactive... https://mikeveerman.github.io/tokenspeed/?rate=12&mode=text https://mikeveerman.github.io/tokenspeed/?rate=12&mode=text And it should be just fine for plenty of background use cases.

cbdevidal 4mo ago

Old hardware is surprisingly effective. I've been considering a side hustle selling offline AI to local businesses who are privacy-sensitive. Medical, legal, places like that. At the low end, I'd use old Xeons with gobs of DDR3, install some V100s, run a smaller agent for general chat inquiries, and a frontier model for the deeper stuff, with a router that passes between them depending on the complexity. The frontier model would perform very slowly, but if it's a deep task the user can submit it in a batch in the evening e.g. "Correlate all of these cases and look for patterns" then receive the output with morning coffee. Of course, AI helped me work out a plan for this. Haha

nicogentile 4mo ago
[flagged]

RobotToaster 4mo ago

Apparently Itanium works quite well for LLMs https://medium.com/@tglozar/running-llama-inference-on-intel-itanium-part-1-be62ff3f5c2f https://medium.com/@tglozar/running-llama-inference-on-intel... Which makes sense I suppose.

Liftyee 4mo ago

Very intriguing. This might be the use for my e5-2430 V2 X2 server that's been lying around. DDR3 is (relatively) cheap now too. Could fit 192GB of RAM in it and play around for much cheaper than a new GPU.

ForOldHack 4mo ago

Well, lets get started. I have 4 of those machines, and they are Two dual processor. They all had 32GB of ram, so now I have two with 64GB, and two with zero. They all hand stock K5000s, now how two have two cards. I stripped the uni processors ram and video cards, and put those into the dual procs. They have 256Gb SSDs, and two 1TB disk drives. One machine has 8Gb of VRam across two cards. Dual processors are 8Cx2 and 32 Threads. They can easily play 16 videos at once. For AI, I have not found a model that I can get above 3 tokens a second. Not a one.

nixgeek 3mo ago
That’s because all of that hardware probably dates back to when Steve Jobs roamed the planet.

SirMaster 4mo ago

Either they have a E5-2620 V2 from 13 years ago, or they have DDR4, not DDR3. The V3 and V4 only support DDR4.

lolcw 4mo ago
No they don’t. Models ending in 6 have DDR3 controller. For example E5-2696 v3/v4 E5-2686 v3/v4 E5-2666 v3/v4 also 2673 v3/v4 2678 v3/v4 as well as E5 2629 v3 E5 2649 v3 E5 2669 v3

sperandeo 4mo ago

ive been doing the same thing. i refactored a old newtek stream machine . its my new favorite thing to do! adding old PCs to my "starcraft" fleet xD

bombcar 4mo ago

Is this John Siracusa? It sounds like it could be something he’d say… (He has a fully maxed out “last Intel” Mac Pro and laments the lack of replacement).

kristjansson 4mo ago

Noting for reference that Gemma4 MTP work is in progress[0] on llama.cpp; similar work for Qwen3.6 landed recently and has been great thus far. [0]: https://github.com/ggml-org/llama.cpp/pull/23398 https://github.com/ggml-org/llama.cpp/pull/23398

mv4 4mo ago

I have an old 192GB DDR4 Dell Precision with dual Intel Xeon Gold 6130 that I've considered spinning up. What's giving me pause is 250W at idle.

mtoner23 4mo ago
Surely that number can go lower with some tweaks
- mv4 4mo ago
  I am sure it can. It will still generate a lot of heat when under load. Are you telling me I should go for it? :) I do have a dual DGX Spark cluster running MiniMax M2.7 already so I am all for on-prem. But will be interesting how this old machine will perform!

qingcharles 4mo ago

Would there be any advantage of running this as dual Xeon? The CPUs are $5 and a dual mobo is $50...

bee_rider 4mo ago
More memory bandwidth presumably. Not sure how well the ecosystem handles thread pinning though.

ryandrake 4mo ago

I've got an old HP Z-620 workstation with dual E5-2697 v2 CPUs (24 cores total, 48 threads @ 2.7GHz) and 128GB of DDR3 RAM. The docs say it supports up to 192GB, but I wasn't able to get it to POST with all the RAM slots full. It's still a "homelab" beast and does great with development and GIS/Mapping applications. I was not able to figure out how to run AI workloads on it with decent performance, however, so I finally broke down and got a dedicated GPU for it. It's pretty great what can still be done with older hardware.

FpUser 4mo ago
I self host on old HP Z-840 with 2x3.6 GHz Xeons 24 total cores, and 512 GB RAM. Cost me peanuts used and works like a charm for many years already
bobmcnamara 4mo ago
I'm in the same situation of having an older workstation nearly maxed out with RAM and neither wanting to pay for the equivalent RAM on a new system neither go down in GBs.

6_7 4mo ago

[dead]

andai 4mo ago

I want to share something strange. I found a typo or two in the post and this absolutely delighted me, because it implies a human wrote the words. (Or was at least heavily involved in the editing.) Guess I am a species-ist after all ;)

bicepjai 4mo ago
I hope LLMs don’t get trained with this reply and start adding typos for making it look like it came from a human :)
- andai 4mo ago
  I felt like I had lost something valuable when I switched to mostly AI based programming, because I used to make so many mistakes that the computer would often do truly magical things I did not even realize were possible. e.g. one time I tried making a collaborative drawing application but I messed up the logic, and the brush strokes would just get temporarily mirrored between the client and server, so you'd see it getting drawn over and over again in a loop. The drawing wasn't stored anywhere, it existed only in the network packets between client and server. Accidental GNU. http://www.gnuterrypratchett.com/ http://www.gnuterrypratchett.com/ So I started working on a tool that adds random errors back into my programs. To reintroduce the possibility of such happy little accidents.
- gowld 4mo ago
  AIs already make typos, not directly intentionally. Since they are token-based, and tokens are lexemes, they can misconjugate works or make grammatical errors.

danbruc 4mo ago

Did some try to estimates what it would take to bake interference for a capable large language model into silicon so that one can pipeline inputs through it and produce outputs at one token per clock cycle?

knorker 4mo ago
I'd expect it to require too much RAM bandwidth to be feasible. RAM is really slow at silicon speeds. Very little is reachable in one clock cycle, unless the clock cycle is abysmally slow.
- danbruc 4mo ago
  No RAM. Instead of having a general purpose multiplier that multiplies an input with a weight stored in RAM, just have a multiplier that hardcodes the weight. In some sense replace each weight with a specialized multiplier and wire them together with accumulators and activation functions in between. And some registers for pipelining. If one goes for four bit quantization, one could have sixteen optimized multipliers, one for each possible weight, and the one just selects and connects them according to the model weights and structure. Example. If you have a neuron with 16 inputs each 8 bit wide and with a 4 bit weight per input, you will have 16 specialized multipliers each scaling its input by the corresponding weight and then the 16 scaled inputs feed into an adder tree and finally an activation function.
  - thomashabets2 4mo ago
    That sounds like wiring the RAM information into order of magnitude same number of transistors. A modern CPU has (quick googling) 184B transistors. If they were bits then that's 23GB. But presumably a model bit needs more than one transistor to represent how it acts as a neuron with its interactions. Then there's the current speedup in inference from restricting which subset of the model is used, which is not a "swap in" that would work with hard wired neurons. But I dunno. Maybe. I'm just guessing.

Aurornis 4mo ago

llama.cpp includes a benchmarking tool called llama-bench https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md https://github.com/ggml-org/llama.cpp/blob/master/tools/llam... ik_llama includes llama-sweep-bench https://github.com/ikawrakow/ik_llama.cpp/blob/main/examples/sweep-bench/README.md https://github.com/ikawrakow/ik_llama.cpp/blob/main/examples... When comparing hardware, the output of these tools is very helpful to let others put it into context. The post says the output is "reading speed" but knowing the prefill and token generation speeds would be a lot more helpful.

tomega2134 4mo ago

I wish this were somehow tagged with AI, so I would know that it's not about say, general computing or cost-efficiency (e.g. using an old xeon machine from ebay instead of new, in these cost-conscious times.) As it is, the title is click-bait for me, as 1) it says I need at least a Xeon somehow and 2) as it doesn't say what I actually need it for.

maxothex 4mo ago

[flagged]

api 4mo ago

Have to point out one boring thing though: this will use a lot more electricity than newer things. So it'll work, but it'll run up your electric bill.

bitwize 4mo ago

Successfully ran Gemma4-26B-A4B on my 8yo first-gen Ryzen with a GeForce GTX 1070. It actually ran acceptably well; I was surprised. I even did some coding with it, but the wheels fell abruptly off when it tried several times to use a constant I told it doesn't exist. I only have 32 GiB of RAM in this old bucket, and these results are not worth the RAM consumption, so I put it aside. Maybe if I finish that build with more memory...

b65e8bee43c2ed0 4mo ago

so how many tokens/s do you get, pp and tg? did I miss it in the article?

npn 4mo ago

I bought one AMD MI50 32GB back then when they were sold rather cheap (around $150-$170). it can easily generate over 70 tokens per second for gemma 4 26B moe model (q4). I have no doubt that we will have another wave of cheap retired server gpus just like before. And that is the time when everyone will have their own models at their home. Or we can just buy the newest medusa halo mini pc. they will be pretty decent, too, albeit pricey.

hualapais 4mo ago

Went this route after hemming and hawing over a Mac Studio Pro for some time. Eventually bought and configured a headless HP Z620 with 192 GB of ECC RAM and dual Xeon E5-2680 v2 processors, an Optane AIC, two P102-100s with 10 GB VRAM each, and a minimal bootable SDD running Debian 12.6 with an older, locked version of CUDA that supports the Pascal cards. Run it remotely from the basement via AMT/meshcommander. Just fire up llama.cpp and its front end and connect over the local network. Currently playing with Talkie, Qwen 3.6 27b, and medgemma, but have had good luck with GGUF performance in general after selecting an appropriate quant. Total cost was under $500, but I bought the server via eBay last year; things may be different now. Details aside, the hope is that ternary LLMs blossom in the coming months and this old hardware can eventually host some very dense models full of factual information, perhaps even larger than the GPU RAM and spilling over to the Optane for IO. Speed would be less important than general factual knowledge. The plan would be to configure then mothball the machine in a Faraday trashcan in the basement, retaining it as a possible "rebuild civilization" oracle should the world fall apart. Of course, power would be an issue in such a scenario, but for how cheap this hardware is and how often AI seems to be practically useful in its latest iterations, why not...

fortran77 4mo ago

My current desktop machine is a 24-core Xeon-3345 with 256GB of RAM and an Nvidia 5090. It still feels extremely fast, even though it's about 8 year old technology with a newer video card.

1970-01-01 4mo ago

Hah. My Xeon turns 20 this year. No issues.

dzonga 4mo ago

for solo operators that run saas (targeting business customers) & if you do a lot of data processing - old servers are the best bang for the buck. remember if you serve real customers as a bootstrapped business - you can afford the whole serve down for maintenance. no need for 99.999%. better than hetzner.

rbanffy 4mo ago

The other day I was considering the adoption of a POWER7+ box. Sadly, Linux hasn't supported POWER7 in quite some time. The machine looked pretty nice, with 4 CPUs with 8 cores each, a total of 128 threads and 512 GB of RAM. I'm not sure it'd run AIX without a license though, which is unfortunate - it's a gorgeous box.

Floppyrom 4mo ago

Famous last words...

rldjbpin 4mo ago

the surge of articles on using decommissioned datacentre hw to run LLMs lately, is more of a symptom of the times than their viability. back when intel had a monopoly on cpu and would refuse to give consumers more than four cores, the old xeon route was popular for a different reason. memory is the bottleneck here (capacity, or rather speed). before you run out to set up your own, try to rather squeeze out the most of your existing hardware. if you are a lucky owner of a lot of cheap memory, you are already in luck. otherwise LM studio allows you to split memory between your gpu and system memory. avoid MoE models or even consider tensor parallelism between the onboard gpu and dedicated one before going for more hardware. there is little to no benefit for using a specific quantization for your models, so go crazy and test out whatever can easily run for you.