5 ms·
> The parsers produce a sequence to tokens, not a full syntax tree. Writing a tokenizer is much easier than parsing full syntax trees...Most other editors don’t
by diffxx 4y ago
> The parsers produce a sequence to tokens, not a full syntax tree. Writing a tokenizer is much easier than parsing full syntax trees...Most other editors don’t construct the full syntax tree.
Syntax highlighting is nice, but fast semantic analysis is the real holy grail.
I have come to think that the best way to develop a new language would be to implement a text editor in that language before releasing the language. The editor should (ideally) have emacs and vim key bindings, though it would surely at least have the bindings that the language author uses. The compiler/interpreter for the language would embedded in the editor. This would allow for a much richer editing experience that goes beyond syntax highlighting. Indeed, the source code would become like a living document in the editor where they editor could display inline information about both syntax _and_ semantics. The editor need not be fancy. It could be written as a terminal application, kind of like a language specific nano or vim.
If the language/editor author is careful in how they design the editor, all of the syntactical and semantic tooling should be exportable into packages that can be consumed by other editors with plugin systems/lsp like vscode or neovim. Then the rich editing experience can be relatively easily exported to any other text editor. Tool authors would also then be able to write static analysis/linting/formatting/whatever tools on top of the semantic tooling that the language supports.
In some ways what I am describing is a more minimalist version of what one would get out of an image/IDE based language like smalltalk but the code representation would still remain as text files. The editor then becomes like a REPL except rather than being an ephemeral process, the REPL state is continually being written to disk and is resumable at any time from any computer capable of running the editor.
- maxbond 4y agoI think it would be sufficient & more valuable to implement a language server for your new language, which keeps you focused on the parts related to your language rather than the struggles of implementing an editor, and then you'll be able to drop into VSCode, neovim, etc.
- thechao 4y agoDoes anyone have an good intro tutorial for writing a language server in, say, C for some simple language? I find most of the docs a little too inscrutable to follow for just a bit of dabbling.
- diffxx 4y agoIf you write your own editor with language server features, writing a language server implementation _should_ (dangerous word) be relatively easy. By writing the editor in your language, you prove that your language is actually useful for a real world problem. You also have the flexibility to add any functionality that you wish without contorting your thinking to whatever may be imposed on you by the lsp model. It isn't necessarily clear to me that writing a language server implementation is that much easier than writing an editor. To be fair, I have personally done neither (though I have written REPLs) so perhaps I am wildly off in my estimation.
- maxbond 4y agoI don't really disagree, but would point out that, for the languages I hack on, they're DSLs that wouldn't be appropriate to write an editor in. I agree that you should be writing code in your language & solving problems with it though, even before there's a working implementation. It does really help hone your vision and focus on the problems you're solving. I've not written a language server either, and I'm positive it's one or two orders of magnitude more difficult than a simple editor, but I think the time is better spent for many projects because it gets you into the nitty gritty mechanics of your language & sets you up for a good developer experience. If your language specializes in writing things like editors though, that would certainly not be the case. Tangentially, my language development hot take is that YAML is a really great platform to start with, so you can focus on prototyping your runtime & semantics and kick the can on parsing & syntax. Parsing takes up a lot of time and you might not know if this language idea is worthwhile or not yet. Additionally, YAML is pretty darn good and you may never need to move off of it.
- Gibbon1 4y agoI agree with the parent. The big advantage of what he's suggesting is code objects exist as first class objects. They don't have to creased by parsing a text file. Which means the editor can directly manipulate them. If you've ever used CAD programs especially Schematic Capture and PCB Layout programs you'll get the idea. Everything displayed on the screen is an object in a database. Instead of a parser blindly parsing a text file and coming across a struct definition, a struct is object created by the programmer. Which means you can have hard links between the objects that make up the program. And those can be directly manipulated with manually or programmatically. The big advantage comes from maintenance and refactoring. Change the name of a field? Happens in exactly one place in the program. So instead of a diff with 1537 files changes you have 'renamed struct fobar to struct foobar'. Change a comment? Well it's just a changed comment.
- maxbond 4y agoI'd like to see languages like that, but it's a massive lift. Our systems of source control and CI/CD are built around the assumption that languages are text files that are somewhat line oriented. It's valuable for people to be able to use different editors and tools, and text files are the integration point that makes this work. Fallible parsers are a bit of a hack but they're a hack that works and is widely deployed already, and I'm not convinced the value of moving to a symbolic/binary representation creates enough value to justify the risk involved in the migration. That being said I'd love to see it happen. We'd end the tabs and spaces debate once and for all!
- bbkane 4y agoI know of two languages trying something similar to this: - https://dion.systems/faq.html https://dion.systems/faq.html - https://www.roc-lang.org/ https://www.roc-lang.org/ They're both experimental, but fun to read about!
- theta_d 4y agoSo smalltalk?