sanchitmonga22

6 ms·

1.
by sanchitmonga22 6mo ago
please check our main repo: https://github.com/RunanywhereAI/runanywhere-sdks/ We are running anywhere, hence RunAnywhere, MetalRT is the fastest inference engine we made for Apple silicon, and we'll be cover
2.
by sanchitmonga22 6mo ago
Yes, that's the plan. MetalRT will ship as part of the RunAnywhere SDK so other developers can integrate it into their own apps. We're working on making that available. If you want to be in the early access group, drop me a line a
3.
by sanchitmonga22 6mo ago
That's a fair read. Tool calling reliability with sub-4B models is genuinely the hardest unsolved problem in on-device AI right now. The inference engine (MetalRT) is production-grade, the pipeline architecture is solid, but the models
4.
by sanchitmonga22 6mo ago
That tracks with what we've seen too. For agent workflows with reliable tool calling, you really do need the larger models. Larger model support is a priority for us. Thanks for the data point.
5.
by sanchitmonga22 6mo ago
Fair criticism. Our benchmarks are on small models because MetalRT was built for the voice pipeline use case, where decode latency on 0.6B-4B models is the bottleneck. You're right that the bigger opportunity on Apple Silicon is large
6.
by sanchitmonga22 6mo ago
Good correction, thanks. You're right that NAX and ANE are distinct, I shouldn't have conflated them. NAX's ability to accelerate LLM prefill is exactly the kind of capability that could complement MetalRT's decode-focus
7.
by sanchitmonga22 6mo ago
Yes, mobile is our primary offering and it is on the roadmap. The same Metal GPU pipeline that powers MetalRT on macOS maps directly to iOS (same Apple Silicon, same Metal API)
8.
by sanchitmonga22 6mo ago
Agreed for a lot of use cases. RCLI supports text-only mode (--no-speak flag or just type in the TUI instead of using push-to-talk). TTS makes sense for hands-free / eyes-free scenarios, but we dont force it.
9.
by sanchitmonga22 6mo ago
We use AI tools in our workflow, same as a lot of teams at this point. The pipeline architecture, Metal integration, and engine design are ours. The code is MIT and open for anyone to read and judge the quality directly.
10.
by sanchitmonga22 6mo ago
RCLI includes local RAG out of the box. You can ingest PDFs, DOCX, and plain text, then query by voice or text: rcli rag ingest ~/Documents/notes rcli ask --rag ~/Library/RCLI/index "summarize the project plan&
11.
by sanchitmonga22 6mo ago
Fair point. The install script shouldn't silently install Homebrew without explicit consent. We'll update it to detect when Homebrew is missing and prompt the user before installing anything beyond RCLI itself. In the meantime, if
12.
by sanchitmonga22 6mo ago
Cool, just checked out dlgo. Looks like you're targeting Go bindings for on-device inference? Different approach but same conviction that this should run locally. Happy to compare notes if you want to chat about Metal optimization or p
13.
by sanchitmonga22 6mo ago
Apple has the silicon, the frameworks (MLX, CoreML), and the models. The gap is putting it all together into a fast, unified on-device pipeline. That's what we're focused on, and honestly, we think Apple will eventually ship somet
14.
by sanchitmonga22 6mo ago
Absolutely, we'd welcome a Portfile contribution. Happy to review and merge. If halostatue wants to co-maintain, even better. Feel free to open a PR or issue on the RCLI repo and we'll coordinate.
15.
by sanchitmonga22 6mo ago
Understood, you want dictation, not a chatbot. That's a valid and different use case. RCLI is Apple Silicon only today because MetalRT is built on Metal. For Linux, the closest thing to what you're describing would be building a v
16.
by sanchitmonga22 6mo ago
This is a great idea. A virtual audio device that sits in the path of any audio stream and provides live transcription, that would be huge for video conferencing, lectures, podcasts. MetalRT's STT numbers make this feasible: 70 seconds
17.
by sanchitmonga22 6mo ago
This is exactly the problem we're trying to solve. The models themselves have gotten surprisingly capable at small sizes, Qwen3.5 4B with 262K context, LFM2 1.2B for fast tool calling, but the inference infrastructure hasn't kept
18.
by sanchitmonga22 6mo ago
The default TTS voice (Piper) is a lightweight model optimized for speed over quality. It's fast but yeah, it doesn't sound great. If you install Kokoro TTS (rcli models > TTS section), the voice quality is dramatically better,
19.
by sanchitmonga22 6mo ago
Fair criticism. The action executed on the LLM side but didn't translate to the correct macOS action, the model hallucinated success instead of routing to the open_url tool. This is a known limitation with small LLMs (0.6B-1.2B) doing
20.
by sanchitmonga22 6mo ago
Thanks for trying it and for filing the bug, we're looking into the homebrew install issue. On unsloth quants: agreed, they're consistently better bit-for-bit. Adding broader quantization format support (including unsloth's a
21.
by sanchitmonga22 6mo ago
Ha, not yet. Metal 4 is interesting and we're keeping an eye on it. MetalRT currently targets Metal 3.1 GPU compute because that's where we get the most control over the decode pipeline. Neural Engine / ANE is powerful for fi
22.
by sanchitmonga22 6mo ago
Not yet with MetalRT, right now we support models up to ~4B parameters (Qwen3 4B, Llama 3.2 3B, LFM2.5 1.2B). These are optimized for the voice pipeline use case where decode speed and latency matter more then model size. Expanding to large
23.
by sanchitmonga22 6mo ago
Fair feedback on the README clarity, we've updated it to make the licensing distinction between RCLI (MIT) and MetalRT (proprietary) more prominent. That should have been clearer from day one. On why we built MetalRT instead of using C
24.
by sanchitmonga22 6mo ago
uzu is a strong engine, it beat us on Llama-3.2-3B (222 vs 184 tok/s) and we reported that honestly in our benchmarks. But looking at the full picture across all four models tested: Qwen3-0.6B: MetalRT 658, uzu 627 Qwen3-4B: MetalRT 18
25.
by sanchitmonga22 6mo ago
Respectfully, the benchmarks show it is different. MetalRT and mlx-lm use the exact same model files, identical 4-bit MLX weights. That makes it a pure engine-to-engine comparison: LLM decode: MetalRT is 1.10-1.19x faster across all models
26.
by sanchitmonga22 6mo ago
Fully public Metal APIs, no private frameworks, no Neural Engine, no undocumented entitlements. MetalRT is built on the public Metal API. The performance comes from how we use the GPU, not from accessing anything Apple doesn't document
27.
by sanchitmonga22 6mo ago
Fair question, let me clarify. RunAnywhere is an inference company. We build the runtime layer for on-device AI. There are two pieces: MetalRT, a proprietary GPU inference engine for Apple Silicon. It runs LLMs, speech-to-text, and text-to-
28.
Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon(github.com)
240 points by sanchitmonga22 6mo ago|153 comments
29.
by sanchitmonga22 6mo ago
It was hand written hand optimized kernels.
30.
Fastest LLM decode engine on Apple Silicon. 658 tok/s on M4-max,beats mlx by 19%(runanywhere.ai)
5 points by sanchitmonga22 6mo ago|3 comments

More ›