Interesting to see Factor and J so far to the bottom and right in the zstd test, but much closer to the rest in the Pandoc test (with Asm taking their place). T

Interesting to see Factor and J so far to the bottom and right in the zstd test, but much closer to the rest in the Pandoc test (with Asm taking their place). This suggests that both the task and language (not just the language) influence the efficiency. I try to use LLMs for Kotlin, Python, Emacs Lisp, and Smalltalk (among many others, but these are what I have ongoing projects in). You'd think that Kotlin and Python would be much easier to generate than the other two, right? But that's not what I observed: Elisp is very close to Python in terms of how fast and how many tokens it takes to generate the code! The generated Elisp code is often better on the first try than generated Kotlin code for a comparable task. Smalltalk is... complex. It's meant to be developed interactively in a running image, but running Codex on API pricing is too expensive, and Codex CLI cannot interact with the image without a lot of plumbing. I ended up building multiple tools that live in the image and a protocol for calling them, and a set of skills for using them - including code search, docs search, test runner, and script/string evaluator. I also defined a way of annotating types for method arguments and return values (without having a type checker), which helped a lot. Still, it's an uphill battle; I wouldn't go there on API pricing! My takeaway is that it's not obvious which language fits the LLMs and a given task best.