6 ms·
Agents easily spend >90% of their time waiting for LLMs to reply and optionally executing API calls in other services (HTTP APIs and DBs). In my experience the
by lunarcave 1y ago
Agents easily spend >90% of their time waiting for LLMs to reply and optionally executing API calls in other services (HTTP APIs and DBs).
In my experience the performance of the language runtime rarely matters.
If there ever was a language feature that matters for agent performance and scale, it's actually the performance of JSON serialization and deserialization.
- fritzo 1y agoIn my experience, the 2nd most costly function in agents (after LLM calls) is diffing/patching/merging asynchronous edits to resolve conflicts. Those conflict resolution operations can call out to low-level libraries, but they are still quite expensive optimization problems, compared to serialization etc.
- autogn0me 1y agocan you be more specific about this?
- fritzo 1y ago1. follow rich hickey's advice and orchestrate all llms to mutate a single shared state 2. let those llms operate asynchronously in parallel 3. when an llm wants to mutate the global state but the state has changed since it's checkout, try to safely merge changes using an expensive diff algorithm (which is still cheaper than the llm); on failure retry
- energy123 1y agoWhat diffing/patching/merging library are you working with? Or are you building your own?
- fritzo 1y agoI've used google's old diff-match-patch, a faster python binding of that C++ library fast-diff-match-patch, and biopython (which amazingly supports unicode!)
- fixprix 1y agoYep exactly, might as well use a language that works with JSON natively like TypeScript; which has arguably far more powerful type system than Go.
- zveyaeyv3sfye 1y ago> like TypeScript; which has arguably far more powerful type system than Go. "arguably". Typescript is just a thin wrapper over javascript who doesnt have these types at all.
- Hasnep 1y agoAnd yet its types are turning complete
- hombre_fatal 1y agoWait, you can't be saying that TypeScript doesn't have a much more powerful type system than Go. AGDTs, mapped types, conditional types, template literal types, partial higher-kinded types, and real inference on top of all that. It had one of the most fully loaded type systems out there while the Go team was asking for community examples of where generics might be useful because they're not sure it might be worth it.