12 ms·

> A Lisp program that writes a Lisp program really just needs to produce a list of (nested lists) of tokens. A JavaScript program that writes a javascript progr

> A Lisp program that writes a Lisp program really just needs to produce a list of (nested lists) of tokens. A JavaScript program that writes a javascript program needs to generate a string that is syntactically valid JavaScript code. That is a much bigger task than just constructing a (nested) list. > Because Lisp syntax is so much simpler than that of JavaScript etc. it is much easier to avoid errors when generating code. In JavaScript you can use JSON to generate data, but JSON can not carry functions around. First of all, the LLM does not produce structured tokens, it produces (tokens of) text. It does not have a concept of nested or structured tokens. Which means that producing a Lisp program and a JavaScript program is basically the same difficulty, i.e. LLMs producing function foo () {} is about the same task as producing (defun foo () ()). In fact, because most Lisps uses the same token ( and ) for almost all delimiters, the LLM in fact gets confused a lot more than Algol-family languages that uses various different tokens for different purposes. It generates thinking traces that are a few screens long while trying to count the closing parenthesis and the depth, something that I have not found to be the case for other languages, even with languages much more obscure than Lisp. (And no, it is not a training data issue, because the Lisp family as a whole is pretty well represented in the data set, due to Emacs Lisp.) > I think this idea makes a lot sense. Instead of making the LLM generate JSON or XML, why not make it generate Lisp, which can carry both programs and data? You do realize that all programming languages contains both programs and data, right? i.e. JSON is literally a subset of the JavaScript language, all JSON documents are valid JavaScript code, can be embedded in JavaScript programs, and so on. This isn't even a JavaScript-specific thing, almost all recent programming languages have data structure literals. The thing that makes Lisp unique is that it can modify programs as data in the language easily, and why would that be a unique capability that would be beneficial for LLMs, when it can just sed/awk or tree-sitter-parse programs with more conventional languages and modify it as easily?

zingar 2mo ago
Now that you mention it, Claude has had trouble at times with balancing brackets in the elisp I have it write. Yet I still had the idea that LLMs should be better at lisp than other languages. A fascinating contradiction, thanks for pointing this out.
- ijk 2mo ago
  Yeah, unfortunately "LLMs are bad at counting" seems to apply to counting parentheses. With modern tool calling I wonder if a better way to go about it is for the LLM to express the program changes as a function or otherwise use an editor that auto-balances parens. There's a lot of relatively simple tooling that makes it easier to write in a Lisp. The languages tend to lend themselves to being straightforward to check like that. What's special about Lisp's repl is that it's perfectly possible to construct your entire program in the repl, testing each addition live as you write it. (Many Lisp-focused editors assume you'll want to do this, such as Emacs making it easy to run the interpreter on a single function in a file.) That tooling is lost if you just try to one-shot the file, and before 2026 the majority of LLMs originally just tried to one-shot every file. But, just like a lot of early LLMs had huge problems with whitespace and numbers because the tokenization was taking efficiency shortcuts that made sense for text but absolutely wrecked code syntaxe, I wonder if the current optimizations are badly formatted for Lisp. At the very least, using a varient like Clojure that also uses [] and {} in addition to () might help.
- Zambyte 2mo ago
  I believe the problem stems from ) and )) both being one token wide. I wonder if repeated parentheses being removed from the token vocabulary would improve accuracy for Lisp code generation.
- cma 2mo ago
  This deepmind paper showed big limitations on nested structures with transformers, LSTMs actually did better: Neural Networks and the Chomsky Hierarchy https://arxiv.org/abs/2207.02098 https://arxiv.org/abs/2207.02098
  - galaxyLogic 2mo ago
    Interesting. Can someone quickly summarize their results? Are they saying that using languages with simpler syntax Lisp lead to LLM being able to generalize their knowledge better?
    - 2mo ago
galaxyLogic 2mo ago
> JSON is literally a subset of the JavaScript language Isn't it more like when you write JSON in JavaScript you write it as a String? JavaScript has the data-type "String" but does it also have a built-in data-type "JSON"? You can pass such a string containing JSON to (built-in) library function JSON.parse() and you can take a data-structure written in JavaScript can call JSON.stringify() on it. But if your JSON is malformed, does the JS-interpreter tell you that, before you call the library-function JUSON.parse() ? If you have a JavaScript parser/interpreter, does it parse JSON too? If not I would say JSON is not part of the JavaScript language, even if JS runtime provides a library function for parsing it.

zingar 2mo ago

Now that you mention it, Claude has had trouble at times with balancing brackets in the elisp I have it write. Yet I still had the idea that LLMs should be better at lisp than other languages. A fascinating contradiction, thanks for pointing this out.

ijk 2mo ago
Yeah, unfortunately "LLMs are bad at counting" seems to apply to counting parentheses. With modern tool calling I wonder if a better way to go about it is for the LLM to express the program changes as a function or otherwise use an editor that auto-balances parens. There's a lot of relatively simple tooling that makes it easier to write in a Lisp. The languages tend to lend themselves to being straightforward to check like that. What's special about Lisp's repl is that it's perfectly possible to construct your entire program in the repl, testing each addition live as you write it. (Many Lisp-focused editors assume you'll want to do this, such as Emacs making it easy to run the interpreter on a single function in a file.) That tooling is lost if you just try to one-shot the file, and before 2026 the majority of LLMs originally just tried to one-shot every file. But, just like a lot of early LLMs had huge problems with whitespace and numbers because the tokenization was taking efficiency shortcuts that made sense for text but absolutely wrecked code syntaxe, I wonder if the current optimizations are badly formatted for Lisp. At the very least, using a varient like Clojure that also uses [] and {} in addition to () might help.
Zambyte 2mo ago
I believe the problem stems from ) and )) both being one token wide. I wonder if repeated parentheses being removed from the token vocabulary would improve accuracy for Lisp code generation.
cma 2mo ago
This deepmind paper showed big limitations on nested structures with transformers, LSTMs actually did better: Neural Networks and the Chomsky Hierarchy https://arxiv.org/abs/2207.02098 https://arxiv.org/abs/2207.02098
- galaxyLogic 2mo ago
  Interesting. Can someone quickly summarize their results? Are they saying that using languages with simpler syntax Lisp lead to LLM being able to generalize their knowledge better?
  - 2mo ago

galaxyLogic 2mo ago

> JSON is literally a subset of the JavaScript language Isn't it more like when you write JSON in JavaScript you write it as a String? JavaScript has the data-type "String" but does it also have a built-in data-type "JSON"? You can pass such a string containing JSON to (built-in) library function JSON.parse() and you can take a data-structure written in JavaScript can call JSON.stringify() on it. But if your JSON is malformed, does the JS-interpreter tell you that, before you call the library-function JUSON.parse() ? If you have a JavaScript parser/interpreter, does it parse JSON too? If not I would say JSON is not part of the JavaScript language, even if JS runtime provides a library function for parsing it.