5 ms·

LLMs have a limited context window - similar to the limited attention span and memory of humans. LLMs also have trouble attending to many constraints at once.

by jaen 4mo ago

LLMs have a limited context window - similar to the limited attention span and memory of humans. LLMs also have trouble attending to many constraints at once. Therefore the best language for agents is likely the one that, on one hand erases all irrelevant details (ie. raises the level of abstraction and does not force focusing on eg. memory management), and on the other hand encodes any domain-relevant details in the code (eg. using advanced type systems, annotations, contracts, spec-like tests eg. property-based). Human readability is a separate concern and still relevant, but the two mentioned properties actually generally improve on that as well (at least for engineers persistent enough to scale the tower of abstraction). Based on this, it seems Go is certainly not that "agent endgame" language. It has large amounts of boilerplate, a general lack of safety around concurrency features, a pretty middling static safety story overall with a generally underpowered type system. I don't think the perfect language exists, yet, but just wildly imagining, it would probably be something like a cross between Scala, Elixir and Lean (or equivalents). Unfortunately none of those languages also have the large training corpus required to make them perform well in all agenting engineering situations (yet). For any language comparison, one must separate the expressiveness of the language, which limits the long-term possibilities for agents, and the training corpus, which is what mostly gives it the current standing. I think we are still in the phase where the languages are separated by essentially random non-design factors such as the amount of training environments the frontier labs are willing to create for them. Given that, the syntax does not matter all that much, as long as the base language itself is flexible enough - as a another wild idea, it's also possible that eg. Python could mostly swallow all these features through external tools (eg. the pre-existing type checkers or linters), and if the frontier labs bother to RL on those tools, that would also work (see also: Mojo).

devin 4mo ago
I know Clojure misses the mark for you in some major areas, but having a real, proper REPL for an agent to interact with makes for an extremely strong feedback loop. IMO, a strong candidate for an "agent endgame" language has this.
- jaen 4mo ago
  I love REPL-driven development, and exploratory "programming" via small snippets is likely part of the endgame (the more the agent strays outside its comfort zone, the more it's needed). It also looks like it paradoxically saves context, since piling up many small snippets is still better than trying to fix a one-shot gone horribly wrong. And yeah, if Clojure had a better static safety story, it would actually rank high, since it's high-abstraction with metaprogramming capabilities, excellent runtime specifications, and has a good "garden path" ie. natural code is good code. (As a devil's advocate though, REPLs can be replaced with gluing together scripts with ad-hoc state/caching via the file system, which LLMs seem to be pretty good at already...) So as an addendum, introspectability is also important ie. allowing to discover the state and composition of the system at any moment - though that's partially a language (eg. reflection) and partially a tooling (eg. debuggers) issue.
  - iLemming 4mo ago
    It sounds almost squarely written by someone without practical experience of REPL-driven development. Specifically with Lisp REPLs. Sure, other languages also have REPLs, but if you dig just a bit deeper, you'd learn that every single step there in R[ead] E[val] P[rint] L[oop] has differences. That makes the entire holistic experience of using the language drastically different. Not universally better for every domain and every case, just different. Here's one of many practical examples I can give you - my WM on Mac is Yabai, it is hooked up to Hammerspoon, which can be scripted with Lua, which means I can use Fennel, which means I can have Lispy REPL. And from here, it is a bit difficult to explain the difference. The challenge is that the magic is invisible to people who haven't felt it. The key insight is "the image vs the file". In most languages, your program is a description that gets turned into a running thing. The REPL is bolted on - a convenience wrapper around that same lint/compile/run/restore-the-state cycle. Python, Ruby, Lua, C#, etc. REPLs work that way. You're still fundamentally working with files that produce processes. In a Lisp, the running system is the environment. There's no gap between "the code" and "the live thing". When I connect to Hammerspoon's Lua runtime via Fennel, I'm not sending scripts to a subprocess - I'm reaching into a living system and reshaping it while it runs. The missing vocabulary here is "liveness", not "fast feedback" - that's a pale shadow of it. Liveness means the environment has no opinion about what's "done" versus "in progress". Everything is always mid-flight and accessible. So I can actually reach out in live REPL session to let's say Slack app window and extract the data about every single element in the app, get the content, compare and continuously reiterate, without having to restart anything, without even saving the code - just pure data extraction without compiling, dealing with state changes, etc. I can interactively move the window, resize it, hide, or maximize it - all that programmatically. Imagine DevTools on steroids, only it works for everything, not just web apps. Learning Lisp and Clojure allowed me to truly experience the genuine joy of programming, because it makes it feel like you're playing a video game. And now, can you even imagine what happens when you open up access to all that awesomeness and grant it to an LLM? Most people have zero idea what it feels and looks like, when you can point an agent to a REPL running in a k8s cluster and it introspects things on the fly, while you let another agent poke through the UI and they work as a team to fix something or develop a new feature. > if Clojure had a better static safety story, it would actually rank high This framing reveals an assumption - the primary value of a type system is catching errors early, and that Clojure is just a dynamically typed language that would be improved by adding that. In a Lisp image, "early" and "late" barely exist as meaningful categories. The feedback loop isn't compile-time vs runtime - it's just... now. You evaluate a form and you know immediately. The error is right there, in context, with the live data that caused it. Static types are, in a real sense, a compensation for the gap I just described - the gap between the description and the running thing. When you can't easily inspect or reshape the live system, you want the compiler to tell you as much as possible before you cross that gap. Clojure has Spec (which can do things most other type systems would struggle to express), it has instrumentation, it has a rich data inspection story. You said "debugger", Clojure has Flowstorm, which is one of the best debugger experiences I have ever encountered in any language, and I have used more than a few.

by jaen 4mo ago

devin 4mo ago

I know Clojure misses the mark for you in some major areas, but having a real, proper REPL for an agent to interact with makes for an extremely strong feedback loop. IMO, a strong candidate for an "agent endgame" language has this.

jaen 4mo ago
I love REPL-driven development, and exploratory "programming" via small snippets is likely part of the endgame (the more the agent strays outside its comfort zone, the more it's needed). It also looks like it paradoxically saves context, since piling up many small snippets is still better than trying to fix a one-shot gone horribly wrong. And yeah, if Clojure had a better static safety story, it would actually rank high, since it's high-abstraction with metaprogramming capabilities, excellent runtime specifications, and has a good "garden path" ie. natural code is good code. (As a devil's advocate though, REPLs can be replaced with gluing together scripts with ad-hoc state/caching via the file system, which LLMs seem to be pretty good at already...) So as an addendum, introspectability is also important ie. allowing to discover the state and composition of the system at any moment - though that's partially a language (eg. reflection) and partially a tooling (eg. debuggers) issue.
- iLemming 4mo ago
  It sounds almost squarely written by someone without practical experience of REPL-driven development. Specifically with Lisp REPLs. Sure, other languages also have REPLs, but if you dig just a bit deeper, you'd learn that every single step there in R[ead] E[val] P[rint] L[oop] has differences. That makes the entire holistic experience of using the language drastically different. Not universally better for every domain and every case, just different. Here's one of many practical examples I can give you - my WM on Mac is Yabai, it is hooked up to Hammerspoon, which can be scripted with Lua, which means I can use Fennel, which means I can have Lispy REPL. And from here, it is a bit difficult to explain the difference. The challenge is that the magic is invisible to people who haven't felt it. The key insight is "the image vs the file". In most languages, your program is a description that gets turned into a running thing. The REPL is bolted on - a convenience wrapper around that same lint/compile/run/restore-the-state cycle. Python, Ruby, Lua, C#, etc. REPLs work that way. You're still fundamentally working with files that produce processes. In a Lisp, the running system is the environment. There's no gap between "the code" and "the live thing". When I connect to Hammerspoon's Lua runtime via Fennel, I'm not sending scripts to a subprocess - I'm reaching into a living system and reshaping it while it runs. The missing vocabulary here is "liveness", not "fast feedback" - that's a pale shadow of it. Liveness means the environment has no opinion about what's "done" versus "in progress". Everything is always mid-flight and accessible. So I can actually reach out in live REPL session to let's say Slack app window and extract the data about every single element in the app, get the content, compare and continuously reiterate, without having to restart anything, without even saving the code - just pure data extraction without compiling, dealing with state changes, etc. I can interactively move the window, resize it, hide, or maximize it - all that programmatically. Imagine DevTools on steroids, only it works for everything, not just web apps. Learning Lisp and Clojure allowed me to truly experience the genuine joy of programming, because it makes it feel like you're playing a video game. And now, can you even imagine what happens when you open up access to all that awesomeness and grant it to an LLM? Most people have zero idea what it feels and looks like, when you can point an agent to a REPL running in a k8s cluster and it introspects things on the fly, while you let another agent poke through the UI and they work as a team to fix something or develop a new feature. > if Clojure had a better static safety story, it would actually rank high This framing reveals an assumption - the primary value of a type system is catching errors early, and that Clojure is just a dynamically typed language that would be improved by adding that. In a Lisp image, "early" and "late" barely exist as meaningful categories. The feedback loop isn't compile-time vs runtime - it's just... now. You evaluate a form and you know immediately. The error is right there, in context, with the live data that caused it. Static types are, in a real sense, a compensation for the gap I just described - the gap between the description and the running thing. When you can't easily inspect or reshape the live system, you want the compiler to tell you as much as possible before you cross that gap. Clojure has Spec (which can do things most other type systems would struggle to express), it has instrumentation, it has a rich data inspection story. You said "debugger", Clojure has Flowstorm, which is one of the best debugger experiences I have ever encountered in any language, and I have used more than a few.