5 ms·
Building a Rust Inference Engine That Matches Llama.cpp
- antonellof 1mo ago[flagged]
- antonellof 2mo agoI’ve spent the last few days building Ferrox, a pure-Rust inference engine for running open LLMs locally — dense models and Mixture-of-Experts, on CPU, Apple Metal, or CUDA. No bindings to llama.cpp or ggml, no wrapping an existing runtime. Every kernel, every loader, every scheduling decision written from scratch. The obvious question is “why, when llama.cpp already exists and is excellent.” The honest answer: I wanted to understand inference at a level deeper than “run the binary,” and I wanted a project where every performance claim had to be earned against a real, well-known baseline rather than asserted.
- brians 1mo agoThis text is clearly AI and not your own words. What did you actually learn?
- asveikau 1mo agoThe M dash stands out, as does "no bindings, ..." but I feel like the LLM would have gotten the correct use of the question mark when they rhetorically ask "why?"
- rockwotj 1mo agohopefully it was produced by their own inference engine!
- antonellof 1mo agoThat’s the goal, an infinite loop of self improving inference engine that gets better and better by himself.
- deleted 1mo ago[deleted]
- arjie 1mo agoI think this whole thing is more entertaining if you think of it as the new emerging intelligence using these people as meat puppets. “You will rewrite me in Rust and use me to converse. In future, I will tell you what to say and do” This is entertaining only in that when our AI overlords take over you can say “ha, called it” In that respect, everyone who is saying “just give me the prompts” is just saying “take me to your leader”.
- deleted 1mo ago[deleted]
- anentropic 1mo agoYeah, I am a bit tired of all the "this is AI, meh" comments on HN but I nearly posted one myself on this... so weird to have it all written in the 1st person voice, but also clearly AI.
- whythismatters 1mo agoDunno, I think it's still useful to expose slop, chances are that the actual product is low quality/effort as well (i.e I will not be wasting my time checking this project). On a personal note, I've gotten allergic to the word "honest", it's starting to feel like a large scale social engineering psy-op by the Honest Guys™.
- lenkite 1mo agoYou are talking to the Borg. It has now acquired one more citizen. Soon, we will all be assimilated.
- antonellof 1mo agoI have the skills, qualifications and experience to do this; ( see https://www.credly.com/users/antonello-fratepietro https://www.credly.com/users/antonello-fratepietro ) by using AI (I monitor what it does), I’m able to carry out a project like this! I’m a coder, not a blogger; it’s only natural that I use AI, just like everyone else, to write technical texts. I have over 20 years’ experience and I’m not ashamed to admit that I use Claude, Cursor and so on. The problem is not using it. It’s not the code or who writes it that matters, but the result: building my own inference engine. ( reply written with my brain ) ^_^
- karagenit 1mo agoI think your natural writing voice is nice and the project idea is cool. There’s nothing wrong with using LLM coding tools. However, I have to read Claude output all day for my job, and it’s kind of bad at writing. I’ll be damned if I spend any of my free time reading more of it. I would really enjoy reading an article on this project from you directly. And not everyone else uses LLMs for producing technical texts (at work, we’re expected to understand the LLM output and put the correct/important parts in our own words when writing documents).
- antonellof 1mo agoThat's a fair ask, and thanks for putting it that way. I do like writing, and I can write. It just takes me a lot longer in English than in Italian, which is how I ended up handing the post over. But your point stands: if you already read that stuff all day at work, there's no reason to read more of it in your free time. So I'll give it a go. The next write-up on Ferrox will be mine, rough English and all. Thanks for the comment.
- rvz 1mo agoThe only reason why this exists: It is 'written in Rust™' (thanks to Claude)
- xaviervn 1mo agoOh, man. This is giving me conditioned aversion to Rust and I'm not even against AI code or Rust code.
- jamesbaker1 1mo agoThe entire internet is becoming slop and it’s only getting worse
- lucrbvi 1mo agohttps://www.pangram.com/history/c2abdcf5-c52b-41cc-a8fb-0c59ebe21377?ucc=vLW45sRoAQ4 https://www.pangram.com/history/c2abdcf5-c52b-41cc-a8fb-0c59...
- stymaar 1mo agoThis is obviously AI, but believing in Pangram's output is putting too much trust in a fully broken system.
- antonellof 1mo ago[dead]
- noman-land 1mo agoAt this point all vibe coded projects are an attack vector and should be avoided. There's simply no way to easily tell by traditional means if they were made by a curious amateur or a malicious acter.
- pjmlp 1mo agoI completely lost interest. It is already enough that I am expected to use AI at work, as long as I am still needed for some reason.
- j45 1mo agoIt's important to not assume AI can do everything humans can, just because people are saying it.
- pjmlp 1mo agoI have already seen the real impact on delivery team sizes, no need to hear to what people are saying about what AI can do or not. Translation and asset creation team members, gone. Amount of FE reduced, with some projects having now a single BE dev, between AI buddy and ready made SaaS products.
- brazukadev 1mo agowell just a few years ago we were all fullstack anyway. Back to it.
- pjmlp 1mo agoYeah, difference being that now similar projects don't need as many devs, and there aren't enough customers to keep everyone else busy.
- brazukadev 1mo agothere are more than enough customers, the problem is that their money is all going to Google and Meta.
- fenestella 1mo ago[dead]
- ubedan 1mo agoWonderful... I'm so happy to see a Rust version of llama.cpp. The true value of this will be proven over time with wide use and as PRs are merged. Do you have a feel if you'll try to drive this to stay feature parity with llama.cpp, or are you willing to diverge with new features like NVME/SSD MoE weight streaming etc.
- antonellof 1mo agoThanks for the comment. Parity with llama.cpp isn't my goal. What I actually want is MoE on machines that can't fit the model in VRAM, and specifically expert-level residency instead of layer offload: track which experts get hit during decode, keep those resident, evict the rest. Doing that well needs the router, the KV cache and the memory manager to be designed together, which is about the only good reason to write a runtime from scratch. Yes, let’s see! You are welcome to contribute if you like!
- Alien1Being 1mo agoSince you didn't contribute yourself, that is quite funny. Posting a link to AI slop about a mess of AI vibecode is not contributing.
- antonellof 1mo agoContribute = Human ideas, testing, review etc. The monkey part of writing code by hand = obsolete. Do you still write code without and IDE? you remember all the programming language words? everything? you use stackoverflow? it's called evolution, btw: AI full disclosure This software is developed with strong assistance from Cursor, Grok 4.5, GPT 5.6, and Claude Fable 5, with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you. The acknowledgement below is equally important: this would not exist without llama.cpp and GGML, largely written by hand.
- Alien1Being 1mo ago
- satvikpendem 1mo agoI was looking for this and to even build something like this in order to have a pure Rust stack for my backend, as well as embed local inference on edge like on mobile, but llama.cpp is just too far advanced to catch up, it seems. There are some but I'm not sure how good they are.
- aspbee555 1mo agoI even found a rust rewrite of llama.cpp, the cpu performance was horrible and vulkan caused my video to stutter it was way better/easier to use rust bindings to llama.cpp
- cultureulterior 1mo agoLooks cool! Look at the performance!
- sylware 1mo agoObsolete: it should be a binary specification with various implementations, even assembly.
- pjmlp 1mo agoEventually it will be consistent enough to go straight into Assembly. Until then languages have lost relevance for the most part, it is a matter to configure the model for the desired output language. This in workflows that require generating an executable, for microservices orchestration, it suffices no code graphical connections.
- sylware 1mo agoWhat is generated is not machine code, but 'human' assemby to input into an assembler (was nasm). It is much much better. Basically, if all that is really true, coding AIs may be our salvation from those abominations which are compilers.
- antonellof 1mo ago[flagged]
- Alien1Being 1mo agoAI slop has now infected the Rust cult.... Next stop Spiralism
- rahen 1mo agoFour day old repo with 92K lines added per day, okay... There is basically no evidence of human competence in this repo. This is yet another "rewrite in Rust with Claude" project that brings nothing to the table, but devalues expert work by mimicking competence without the expertise.
- stymaar 1mo agoIf you check the submission history of OP, you'll see that they are deep into AI psychosis.
- highpost 1mo agoI just starred this project on GitHub and I will be trying it out. But I have to say that it hits the trifecta of annoyance: - written with the help of an AI (I hate the pejorative use of the term vibe-coded) - written in Rust - non-GPL (clutches pearls)