5 ms·

Lingo: A Go micro language framework for building Domain Specific Languages

133 points by adityasaky 2y ago

dlock17 2y ago
This seemed like nuclear overkill for most problems I can think of. And where it should be used, I can't imagine you can't find a pre existing language (Cuelang maybe) instead. I was expecting a section at the end where they demonstrate which services need a new language written just for it's configuration, but nope, just general examples. Also, this should have a (2022) in the title.
- tony-allan 2y ago
  I think that a Domain Specific Language without the scaffolding of a language like python to generate test data is a great use case. You have a specific DSL to generate test data and store code written using your DSL with your test cases. You then have a security model that separates the responsibilities of the test team from that of other developers. The team can generate many test cases in a secure environment. You could then seek community input into your test processes with having to worry about rogue code.
- jerf 2y ago
  It is nuclear overkill for most problems you can think of. But when you hit a problem that you need something like this for... you need something like this. The attempts to get around it or avoid it or do some unbelievably hacky thing leads to piles and piles of terrible, terrible code. In 2024, though, I do try very hard to embed my DSLs in an existing serialization. It doesn't always work out, but, the case they show of directly embedding an AST into YAML is a worst-case scenario. In real life I've done things like specify a particular field carries an expr[1] expression to do that sort of thing, and then the structure of the rest of the file just follows normal serialization format. [1]: https://github.com/expr-lang/expr https://github.com/expr-lang/expr , but I'm sure many static languages have something like this. If you don't know one, it's a good tool to put in the belt in case you ever need it.
  - dlock17 2y ago
    To be clear, my problem isn't with requiring a single DSL, it's with writing so many DSL's at work that there's need for a DSL writing framework to crap them out even faster. At what point do you sit back and ask, is there a way to reduce complexity here? Is it not until you're joining a team where the last guy made dozens of DSL's for all his tools? That was the main reason I was expecting more than a few real world use cases rather than generalized examples and no examples from their workplace.
    - randomdata 2y ago
      > is there a way to reduce complexity here? Yes, you can generate your test data in the host language. That dramatically reduces upfront complexity. But that doesn't come free. Now it is not easily transferrable to other languages. If you recall rsc's somewhat recent talk on testing, they benefited greatly by being able to use similar DSLs from other projects in tests to ensure feature parity. It can be a huge boon to have your test data scripts in an "agnostic" language for 30 years down the line when you are no longer working in the same environment. Tradeoffs, as always. > I was expecting more than a few real world use cases Maybe that's the only use-case they know of? The project was clearly built for a specific purpose. That's not to say that it cannot be found useful elsewhere, but that doesn't mean they have found it to be. If someone else finds it useful in other applications, I'm sure they'll write their own document about it. That's what the web is for, after all.
- 0xCMP 2y ago
  I wouldn't think of it for generating configuration. The example given implies they use it for fuzzing program inputs. It's purpose in the article is simple: generate a CSV with two columns and floats. However, a DSL could be made to do anything. Generate particular kinds of PDFs, Excel files, intentionally incorrect CSV/TSV files, zipbombs, and etc. All that is required is to wire it up in the Go side of things and then write a script to make use of it.
- kubectl_h 2y ago
  > I was expecting a section at the end where they demonstrate which services need a new language written just for it's configuration, but nope, just general examples. Heh that was my first instinct as to why they built this as well but more for providing ergonomic ways to generate k8s objects for complex gitlab specific CRDs.
- randomdata 2y ago
  > Also, this should have a (2022) in the title. Is there something about the current state of world that has invalidated some or all of the content in the article?
  - danmur 2y ago
    Why have a title at all?
Alifatisk 2y ago
Regarding the Ruby example, why did they use Float("...") instead of #to_f?
- intelekshual 2y ago
  Probably because `Float()` is stricter and will raise an error if the string isn't a valid number, whereas `#to_f` will silently return 0.0 or do a best-effort conversion (e.g. "1.2abc".to_f => 1.2)
  - Alifatisk 2y ago
    Ohhh!
Lyngbakr 2y ago
> Some popular DSLs most software developers use on a regular basis include Regular Expressions for pattern matching, AWK for text transformation or Standard Query Language for interacting with databases. Isn't it Structured Query Language? Or are both variants used?
- shakna 2y ago
  > In the early days of the system there was divided preference between Standard Query Language and Structured Query Language but it did not make a whole lot of difference since most people most of the time called it by the acronym SQL. Now the overwhelming but not complete preference is for Structured Query Language. [0] https://www.sjsu.edu/faculty/watkins/sql.htm https://www.sjsu.edu/faculty/watkins/sql.htm
  - Lyngbakr 2y ago
    Thanks for that!
- randomdata 2y ago
  Strictly, it is just SQL. It doesn't stand for anything. The project was originally known as SEQUEL, a fun word play on QUEL (from the Ingress/Postgres camp), but trademark issues saw it later lose the vowels. "Structured Query Language" is a so-called backronym. "Standard Query Language" fits the pattern as well. As does "Super Quick Lookup", if you want to have some fun with it.
wahern 2y ago
When I think of a DSL, I think of a language with specialized syntax, grammar, or constructs suited to the problem domain. Think SQL, AWK, or regular expressions. This is just a LISP variant with a typical host-side API for registering function names. I'll never get how merely having function names that reflect the use case, plus a stripped down or absent standard library, qualifies as a DSL. I know some people have long used "DSL" in this way, especially among LISP fans, but... I just don't get it. If I want a DSL it's because I want something that gives me, e.g., novel control flow constructs a la AWK, or highly specialized semantics a la regular expressions, that directly suit the problem domain. If I'm not getting that kind of power, why tie myself to some esoteric dependency? Either way you're adopting a tremendous maintenance burden; it better be worth your while. I'm a huge fan of Lua and have used it for many projects in different roles, but never once thought of any particular case as having created a DSL, even when stripping the environment to just a few, well-named, self-describing functions. I don't mean to criticize this particular project. Good code is good code. It's just the particular conceptualization (one shared by many others, to be fair) of what a "DSL" means that bugs me.
- Trufa 2y ago
  This is a very good and interesting point, but what if the point itself is to reduce the power and increase things like legibility. If I create this kind of mapped functions DSLs I can assure that things will be done a a certain way vs the borderline infinite possibilities of code.
- skydhash 2y ago
  I don’t think there’s need to have such strict requirements. No need to invent a whole new paradigm, you can go with the usual ones as long as it works well. More often than not the domain is not that complex or flexible to require a language, it may only requires a few algorithms (libraries) ie even if you do invent a language, few programs will be built with it. You may as well tweak an existing language for a nicer DX
- wbl 2y ago
  Lisp macros can make their own control flow: it's like the relation of LinQ to F#.
bitexploder 2y ago
Disclaimer: this is a neat project. DSLs are such a trap for most projects that think they need them. Use lua or something off the shelf for scripting. CEL exists for Go and (safely) solves many of the problems you might also want a DSL for. The case for DSLs is often hard to justify in a project that has to be maintained for years.
- kristianp 2y ago
  CEL seems to be for proto-buffers[1], not text? [1] https://github.com/google/cel-spec/blob/master/doc/intro.md https://github.com/google/cel-spec/blob/master/doc/intro.md CEL-go: https://github.com/google/cel-go https://github.com/google/cel-go
  - bitexploder 2y ago
    Yes, cel-go. But it is all the same expression language. It is used in a number of places in GCP for example. Attribute based ACLs, and such.
    - __turbobrew__ 2y ago
      Kubernetes too: https://kubernetes.io/docs/reference/using-api/cel/ https://kubernetes.io/docs/reference/using-api/cel/
- deleted 2y ago
  [deleted]
- deleted 2y ago
  [deleted]
- zelphirkalt 2y ago
  CEL seems very much in line with the Golang ideology. It looks like CEL doesn't really have any upsides at all, except for being non-turing-complete. It looks like it is more of a convenience food, for people, for whom any other syntax than what they know from mainstream programming languages is "too adventurous" or "too foreign". As if they cannot be trusted to be able to cope with another syntax. This might even be true, considering the character of Golang itself, which was created as a dumbed down lowest common denominator of several mainstream languages, so that everyone gets it. It was even missing generics for a long time, deeming them to be too complicated. As a DSL CEL is kinda pointless, since it does not create any additional convenience beyond the usual mainstream programming language syntax. It therefore leaves potential on the table, and as a tradeoff appeals to familiarity of syntax. As a configuration language it is usable, probably with reduced risk, compared to using Golang itself (no turing-completeness!). I don't think it actually appeals to anyone, who considers creating a DSL for a good reason.
codetrotter 2y ago
Not to be confused with the Lingo programming language that was used in Macromedia Director to create Shockwave content ;) https://en.wikipedia.org/wiki/Lingo_(programming_language) https://en.wikipedia.org/wiki/Lingo_(programming_language) https://en.wikipedia.org/wiki/Adobe_Shockwave https://en.wikipedia.org/wiki/Adobe_Shockwave https://en.wikipedia.org/wiki/Adobe_Director https://en.wikipedia.org/wiki/Adobe_Director
- giancarlostoro 2y ago
  Which is still in use today by hobbyists mainly, but also Habbo Hotel released their old client using their old Lingo code as "Origins" it runs as an EXE that embeds the runtime. People are working on emulators that will run it natively on web, and there's a really interesting decompiler for it as well.
ofrzeta 2y ago
I don't know. Last update two years ago when the project was first posted. Is it worth spending time on it?
- randomdata 2y ago
  What's missing that would necessitate an update?
oldpersonintx 2y ago
[dead]
wejick 2y ago
Unfortunately the given example doesn't give - mere mortal like me- idea on how this make sense to use. Instead of using whatever scripting language available, or just json. Or probably grule? [1] https://github.com/hyperjumptech/grule-rule-engine https://github.com/hyperjumptech/grule-rule-engine
- randomdata 2y ago
  > Instead of using whatever scripting language available This is just another scripting language available. It only differs from something like Lua or Javascript in that it claims to have less complexity. Complexity which, while not fully elaborated on, apparently bogged down their efforts when they originally tried using Lua and an embeddable Go engine.
bezkom 2y ago
There is already programming language called Lingo, though probably nobody uses it anymore: https://en.wikipedia.org/wiki/Lingo_(programming_language) https://en.wikipedia.org/wiki/Lingo_(programming_language)
- yayoohooyahoo 2y ago
  You'd think they'd do a quick check on the name before committing to it. I remember using Lingo with Macromedia Director, it was a nice little language
  - NAHWheatCracker 2y ago
    You'd be surprised how people will stick to a name. At my last job, we spent 3 years building an electrical systems product with the same name as an eye scanning company. We had invested in marketing and sales that was practically useless once the executives finally agreed they needed to rebrand. It was the sort of thing that was pointed out on day zero, but the man in charge was too proud of the name. It wasn't even a very good name for our product. And that was a bet-the-company type of choice. Lingo here is just some little library of no import to Gitlab's business.
justinko 2y ago
The verbosity of “fed through an LLM” articles are becoming unbearable to digest.
allknowingfrog 2y ago
This is a bit of a tangent, but this is how I would actually solve the problem in Ruby: #!/usr/bin/env ruby require "csv" puts CSV.foreach(ARGV[0], headers: true).sum { _1[1].to_f }.round(2)
Mikhail_Edoshin 2y ago
A generic structure is not a notation. XML or S-expressions are a generic structure. A good practical test that shows limitations of this approach would be to try to write something like a picture description similar to Pikchr (PIC). It is possible, but the result will be far more verbose and obtuse than what you get with a real grammar and parser in Pikchr.

133 points by adityasaky 2y ago

This seemed like nuclear overkill for most problems I can think of. And where it should be used, I can't imagine you can't find a pre existing language (Cuelang maybe) instead. I was expecting a section at the end where they demonstrate which services need a new language written just for it's configuration, but nope, just general examples. Also, this should have a (2022) in the title.

tony-allan 2y ago
I think that a Domain Specific Language without the scaffolding of a language like python to generate test data is a great use case. You have a specific DSL to generate test data and store code written using your DSL with your test cases. You then have a security model that separates the responsibilities of the test team from that of other developers. The team can generate many test cases in a secure environment. You could then seek community input into your test processes with having to worry about rogue code.
jerf 2y ago
It is nuclear overkill for most problems you can think of. But when you hit a problem that you need something like this for... you need something like this. The attempts to get around it or avoid it or do some unbelievably hacky thing leads to piles and piles of terrible, terrible code. In 2024, though, I do try very hard to embed my DSLs in an existing serialization. It doesn't always work out, but, the case they show of directly embedding an AST into YAML is a worst-case scenario. In real life I've done things like specify a particular field carries an expr[1] expression to do that sort of thing, and then the structure of the rest of the file just follows normal serialization format. [1]: https://github.com/expr-lang/expr https://github.com/expr-lang/expr , but I'm sure many static languages have something like this. If you don't know one, it's a good tool to put in the belt in case you ever need it.
- dlock17 2y ago
  To be clear, my problem isn't with requiring a single DSL, it's with writing so many DSL's at work that there's need for a DSL writing framework to crap them out even faster. At what point do you sit back and ask, is there a way to reduce complexity here? Is it not until you're joining a team where the last guy made dozens of DSL's for all his tools? That was the main reason I was expecting more than a few real world use cases rather than generalized examples and no examples from their workplace.
  - randomdata 2y ago
    > is there a way to reduce complexity here? Yes, you can generate your test data in the host language. That dramatically reduces upfront complexity. But that doesn't come free. Now it is not easily transferrable to other languages. If you recall rsc's somewhat recent talk on testing, they benefited greatly by being able to use similar DSLs from other projects in tests to ensure feature parity. It can be a huge boon to have your test data scripts in an "agnostic" language for 30 years down the line when you are no longer working in the same environment. Tradeoffs, as always. > I was expecting more than a few real world use cases Maybe that's the only use-case they know of? The project was clearly built for a specific purpose. That's not to say that it cannot be found useful elsewhere, but that doesn't mean they have found it to be. If someone else finds it useful in other applications, I'm sure they'll write their own document about it. That's what the web is for, after all.
0xCMP 2y ago
I wouldn't think of it for generating configuration. The example given implies they use it for fuzzing program inputs. It's purpose in the article is simple: generate a CSV with two columns and floats. However, a DSL could be made to do anything. Generate particular kinds of PDFs, Excel files, intentionally incorrect CSV/TSV files, zipbombs, and etc. All that is required is to wire it up in the Go side of things and then write a script to make use of it.
kubectl_h 2y ago
> I was expecting a section at the end where they demonstrate which services need a new language written just for it's configuration, but nope, just general examples. Heh that was my first instinct as to why they built this as well but more for providing ergonomic ways to generate k8s objects for complex gitlab specific CRDs.
randomdata 2y ago
> Also, this should have a (2022) in the title. Is there something about the current state of world that has invalidated some or all of the content in the article?
- danmur 2y ago
  Why have a title at all?

Alifatisk 2y ago

Regarding the Ruby example, why did they use Float("...") instead of #to_f?

intelekshual 2y ago
Probably because `Float()` is stricter and will raise an error if the string isn't a valid number, whereas `#to_f` will silently return 0.0 or do a best-effort conversion (e.g. "1.2abc".to_f => 1.2)
- Alifatisk 2y ago
  Ohhh!

Lyngbakr 2y ago

> Some popular DSLs most software developers use on a regular basis include Regular Expressions for pattern matching, AWK for text transformation or Standard Query Language for interacting with databases. Isn't it Structured Query Language? Or are both variants used?

shakna 2y ago
> In the early days of the system there was divided preference between Standard Query Language and Structured Query Language but it did not make a whole lot of difference since most people most of the time called it by the acronym SQL. Now the overwhelming but not complete preference is for Structured Query Language. [0] https://www.sjsu.edu/faculty/watkins/sql.htm https://www.sjsu.edu/faculty/watkins/sql.htm
- Lyngbakr 2y ago
  Thanks for that!
randomdata 2y ago
Strictly, it is just SQL. It doesn't stand for anything. The project was originally known as SEQUEL, a fun word play on QUEL (from the Ingress/Postgres camp), but trademark issues saw it later lose the vowels. "Structured Query Language" is a so-called backronym. "Standard Query Language" fits the pattern as well. As does "Super Quick Lookup", if you want to have some fun with it.

wahern 2y ago

When I think of a DSL, I think of a language with specialized syntax, grammar, or constructs suited to the problem domain. Think SQL, AWK, or regular expressions. This is just a LISP variant with a typical host-side API for registering function names. I'll never get how merely having function names that reflect the use case, plus a stripped down or absent standard library, qualifies as a DSL. I know some people have long used "DSL" in this way, especially among LISP fans, but... I just don't get it. If I want a DSL it's because I want something that gives me, e.g., novel control flow constructs a la AWK, or highly specialized semantics a la regular expressions, that directly suit the problem domain. If I'm not getting that kind of power, why tie myself to some esoteric dependency? Either way you're adopting a tremendous maintenance burden; it better be worth your while. I'm a huge fan of Lua and have used it for many projects in different roles, but never once thought of any particular case as having created a DSL, even when stripping the environment to just a few, well-named, self-describing functions. I don't mean to criticize this particular project. Good code is good code. It's just the particular conceptualization (one shared by many others, to be fair) of what a "DSL" means that bugs me.

Trufa 2y ago
This is a very good and interesting point, but what if the point itself is to reduce the power and increase things like legibility. If I create this kind of mapped functions DSLs I can assure that things will be done a a certain way vs the borderline infinite possibilities of code.
skydhash 2y ago
I don’t think there’s need to have such strict requirements. No need to invent a whole new paradigm, you can go with the usual ones as long as it works well. More often than not the domain is not that complex or flexible to require a language, it may only requires a few algorithms (libraries) ie even if you do invent a language, few programs will be built with it. You may as well tweak an existing language for a nicer DX
wbl 2y ago
Lisp macros can make their own control flow: it's like the relation of LinQ to F#.

bitexploder 2y ago

Disclaimer: this is a neat project. DSLs are such a trap for most projects that think they need them. Use lua or something off the shelf for scripting. CEL exists for Go and (safely) solves many of the problems you might also want a DSL for. The case for DSLs is often hard to justify in a project that has to be maintained for years.

kristianp 2y ago
CEL seems to be for proto-buffers[1], not text? [1] https://github.com/google/cel-spec/blob/master/doc/intro.md https://github.com/google/cel-spec/blob/master/doc/intro.md CEL-go: https://github.com/google/cel-go https://github.com/google/cel-go
- bitexploder 2y ago
  Yes, cel-go. But it is all the same expression language. It is used in a number of places in GCP for example. Attribute based ACLs, and such.
  - __turbobrew__ 2y ago
    Kubernetes too: https://kubernetes.io/docs/reference/using-api/cel/ https://kubernetes.io/docs/reference/using-api/cel/
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
zelphirkalt 2y ago
CEL seems very much in line with the Golang ideology. It looks like CEL doesn't really have any upsides at all, except for being non-turing-complete. It looks like it is more of a convenience food, for people, for whom any other syntax than what they know from mainstream programming languages is "too adventurous" or "too foreign". As if they cannot be trusted to be able to cope with another syntax. This might even be true, considering the character of Golang itself, which was created as a dumbed down lowest common denominator of several mainstream languages, so that everyone gets it. It was even missing generics for a long time, deeming them to be too complicated. As a DSL CEL is kinda pointless, since it does not create any additional convenience beyond the usual mainstream programming language syntax. It therefore leaves potential on the table, and as a tradeoff appeals to familiarity of syntax. As a configuration language it is usable, probably with reduced risk, compared to using Golang itself (no turing-completeness!). I don't think it actually appeals to anyone, who considers creating a DSL for a good reason.

codetrotter 2y ago

Not to be confused with the Lingo programming language that was used in Macromedia Director to create Shockwave content ;) https://en.wikipedia.org/wiki/Lingo_(programming_language) https://en.wikipedia.org/wiki/Lingo_(programming_language) https://en.wikipedia.org/wiki/Adobe_Shockwave https://en.wikipedia.org/wiki/Adobe_Shockwave https://en.wikipedia.org/wiki/Adobe_Director https://en.wikipedia.org/wiki/Adobe_Director

giancarlostoro 2y ago
Which is still in use today by hobbyists mainly, but also Habbo Hotel released their old client using their old Lingo code as "Origins" it runs as an EXE that embeds the runtime. People are working on emulators that will run it natively on web, and there's a really interesting decompiler for it as well.

ofrzeta 2y ago

I don't know. Last update two years ago when the project was first posted. Is it worth spending time on it?

randomdata 2y ago
What's missing that would necessitate an update?

oldpersonintx 2y ago

[dead]

wejick 2y ago

Unfortunately the given example doesn't give - mere mortal like me- idea on how this make sense to use. Instead of using whatever scripting language available, or just json. Or probably grule? [1] https://github.com/hyperjumptech/grule-rule-engine https://github.com/hyperjumptech/grule-rule-engine

randomdata 2y ago
> Instead of using whatever scripting language available This is just another scripting language available. It only differs from something like Lua or Javascript in that it claims to have less complexity. Complexity which, while not fully elaborated on, apparently bogged down their efforts when they originally tried using Lua and an embeddable Go engine.

bezkom 2y ago

There is already programming language called Lingo, though probably nobody uses it anymore: https://en.wikipedia.org/wiki/Lingo_(programming_language) https://en.wikipedia.org/wiki/Lingo_(programming_language)

yayoohooyahoo 2y ago
You'd think they'd do a quick check on the name before committing to it. I remember using Lingo with Macromedia Director, it was a nice little language
- NAHWheatCracker 2y ago
  You'd be surprised how people will stick to a name. At my last job, we spent 3 years building an electrical systems product with the same name as an eye scanning company. We had invested in marketing and sales that was practically useless once the executives finally agreed they needed to rebrand. It was the sort of thing that was pointed out on day zero, but the man in charge was too proud of the name. It wasn't even a very good name for our product. And that was a bet-the-company type of choice. Lingo here is just some little library of no import to Gitlab's business.

justinko 2y ago

The verbosity of “fed through an LLM” articles are becoming unbearable to digest.

allknowingfrog 2y ago

This is a bit of a tangent, but this is how I would actually solve the problem in Ruby: #!/usr/bin/env ruby require "csv" puts CSV.foreach(ARGV[0], headers: true).sum { _1[1].to_f }.round(2)

Mikhail_Edoshin 2y ago

A generic structure is not a notation. XML or S-expressions are a generic structure. A good practical test that shows limitations of this approach would be to try to write something like a picture description similar to Pikchr (PIC). It is possible, but the result will be far more verbose and obtuse than what you get with a real grammar and parser in Pikchr.