Ninfer is the real deal. 160 tokens/sec. You can parallelize 4 chats at once and get ~400 tokens/sec I’ve heard. Although that just compounds the context size i

Ninfer is the real deal. 160 tokens/sec. You can parallelize 4 chats at once and get ~400 tokens/sec I’ve heard. Although that just compounds the context size issue, still absolutely incredible for running locally. I made a web based battle chess game with sound effects running on a pi harness in about 15 minutes, complete with a (very unskilled) “AI” (not an llm) that plays against you if you want!