KingJoker

6 ms·

1. 1.▲
   by KingJoker 2mo ago

   Glad to hear! Let me know if you run into any issues!
2. 2.▲
   by KingJoker 2mo ago

   You're welcome, I appreciate the engagement! Your intuitions are right, faster switching between agent calls, and tighter packing of agents in the same amount of compute space. There's a second thing besides just efficiency that I
3. 3.▲
   by KingJoker 2mo ago

   I addressed this a little bit in the comment below, but the cycles add up. I'm doing some pretty crazy things higher up in the stack, that I'm not quite ready to release yet. But even micro optimizations here add up at the scale I
4. 4.▲
   by KingJoker 2mo ago

   To start with, it eliminates the IPC and Sidecar overhead of using something like Ollama. It might seem trivial when comparing the speed of a REST request to that of inferencing, but the ms add up at scale. And for what I'm doing highe
5. 5.▲
   by KingJoker 2mo ago

   Thanks! It was a fun challenge getting an AI agent to pair with me while adhering to strict c-struct padding and memory alignment. Let me know if anything sticks out you'd like to discuss deeper!
6. 6.▲
   Show HN: Low-latency local LLM runner via OpenJDK Panama FFM (Java 22)(github.com)

   38 points by KingJoker 2mo ago|10 comments