6 ms·

Reflections on the Lack of Adoption of Domain Specific Languages [pdf]

25 points by bnr 6y ago

bnr 6y ago
I'm thoroughly convinced that software engineering efficiency can be increased by at least one order of magnitude by letting domain experts and product owners directly modify the product through a well defined DSL.
- aszen 6y ago
  To create that ideal dsl one has to both know the domain well enough and have the technical skills which is a rare combination so i'm rather sceptical of this claim. Besides any dsl can only help with well understood repeatable problems, for problems that aren't covered by a dsl, software engineers are still required. The fundamental role of software engineers is to build easy to use and insightful interfaces to understand complex data generated/collected from the real world. To do that one needs to have the skill to organise information and control complexity by data hiding not exactly the skills product owners and domain experts are known for.
- jamesblonde 6y ago
  We built the first open-source feature store for ML, https://github.com/logicalclocks/hopsworks https://github.com/logicalclocks/hopsworks , when every existing proprietary feature store (Uber Michelangelo and Bighead at AirBnb) were shouting about how their DSL for feature engineering was the future. Fast-forward 2 years and it is clear that Data Scientists want to work with Python, not with a DSL. We based our Feature Store on a Dataframe API for Python/PySpark. The DSL can never evolve at the same rate as libraries in a general-purpose programming language. So, your DSL is great for show-casing a Feature Store, but when you need to compute embeddings or train a GAN or done any type of feature engineering that is not a simple time-window aggregation, you pull out Python (or Scala/Java). I am old enough to have seen many DSLs in different domains (GUIs, aspect-oriented programming, feature engineering) have their day in the sun only to be replaced by general-purpose programming languages due to their unmatched utility.
  - bnr 6y ago
    The article (controversely, maybe) classifies libraries for general-purpose programming languages as internal DSLs. One could argue that Data Scientists working with libraries in Pythin are already using a DSL, just with an escape hatch into the general-purpose world. I don't think proper vertical DSLs should be made marketed towards people who are comfortable working in a general-purpose language. I see them as a way to help non-technical domain experts work on code instead of specification. Limiting the possibilities of what one can write, like with MPS' projectional editor, is a feature here and not a bug.
    - jamesblonde 6y ago
      A library is not a DSL, despite what the learned authors may claim. In fact, there are a couple of examples of "successful" DSLs in the data world - you could argue that Talend and DBT are visual programming tools for ETL pipelines. Defintely Zapier- which is just for integrating services that have well-defined REST APIs.
- layoutIfNeeded 6y ago
  I guess you haven't had the "pleasure" of cleaning up the mess made by domain experts and product owners using those well-defined DSLs.
tinco 6y ago
Domain specific languages just didn't settle at the level we expected. We thought they would be for really narrow domains, like a language for each kind of business logic. But instead the domains are more at the architecture level and some very broad purpose (i.e. HTML, SQL, etc). At my company, we use Ruby for web backend and system administration. We use Typescript for web frontend. We use C# for process automation. We use Rust and C++ for real time performance (we've got an in house 3D engine), and a little bit of high performance processing. We use Python for data processing and machine learning. That's six "general purpose" languages, each applied to a specific domain, on six software developers, in three teams of two. Each happy with their choice of languages, effectively solving the business needs. I think at some point in the past this situation was described as a nightmare. But I think at that point people thought developers were something you just opened a tin can of, and then applied to whatever problem you had. Nowadays we hire for a specific purpose (or at least I do). When I hire web dev experts, I expect them to be fluent in the industry standard web programming languages and platforms of our choice (i.e. for us it'd be Ruby and Typescript). When I hire 3D experts I expect them to be fluent in C++, and proficient in Rust (it's the future). And obviously it's Python for the data science types. That's a very clear world to me, and to be extra clear, I'm not saying any of the languages I mentioned are the best choice. As long as we operate within the lines the industries have drawn, you can draw upon the best libraries and ecosystems for your particular problem. I would never approve a 3D engine to be developed in Ruby, or a data science pipeline in Javascript. I'd rather have a Ruby developer learn some Python so they'd be able to work on the data science pipeline (disregarding their complaints about Python's inelegance) than have them trying to kludge together subpar Ruby libraries.
anothernewdude 6y ago
It is a bit telling those were the best reasons they could think of for lack of adoption. Just "ignorance". Not we thought that maybe specifying a DSL rather than a library would leave our users and clients in a state of having to banadage over the constraints of a DSL that doesn't handle future use cases, like say dealing with control flow in a half-assed YAML based language, I'm looking at you Ansible. I don't want MAKE, or whatever DSL, I want to be able to drop into a real programming language when necessary. So libraries. Not frameworks, Not DSLs, libraries. And thankfully, it seems the world agrees with me.
- rualca 6y ago
  > I don't want MAKE, or whatever DSL, I want to be able to drop into a real programming language when necessary. So libraries. That's one opinion, sure. However, DSLs are always in a far better position to solve domain-specific problems because, unlike generic programming libraries, they not only reflect domain-specific knowledge and best practices and represent standardized solutions for recurring problems. Consequently, Makefiles and other DAG-related DSLs are omnipresent and dominate domains such as build systems, and generic library-based proposals always failed to gain any form of traction. And it's not just Makefiles or build systems. There's also markup languages, infrastructure as code, CICD/processing pipelines, configuration, etc etc etc.
  - cle 6y ago
    It’s funny you mention the latter, I’ve seen an industry-wide trend toward using standard programming languages with those things (Pulumi, CDK, et al). I’ve started using them myself. I’m not convinced yet that they’re better. Gradle seems to be an interesting case of a successful build system with an interface in a real language (Groovy), although Groovy is so dynamic that the line between Groovy and DSL is pretty blurry. I think another big issue is the generally terrible interop between programming languages, each language has a silo of excellent libraries and the only way to move things between silos is with unsophisticated data formats. I’ve also made a handful of developer libraries and always pick a cludgy data format for configuration, the ergonomics are quite nice—it’s easy to reason about as a user (no head scratching about what the config will look like at “runtime”), there’s generally no tooling setup required and no additional compilers or interpreters to install, and I’m free to use whatever languages as a tool designer I like and can move freely between them (at this point they all support YAML, JSON, et al). We want to make the barrier to entry as low as possible and simple data formats do just that.
notacoward 6y ago
It seems to me that the main reason DSLs aren't more widely adopted is that any DSL will be unsupported by other tools developers consider more important. Your IDE won't have syntax highlighting and auto completion for it, and a lot of developers nowadays seem crippled without those. Linters and semantic checkers won't understand it. Nor will debuggers. There will be no mocking libraries or makefile rules for it. The list goes on and on. These are problems that every new non-domain-specific language has to address. It's quite a lot, and most of it is pretty tedious compared to designing the language itself. So even those who try to create DSLs often skip most of the "extra" bits, and other developers learn to hate DSLs.
- bnr 6y ago
  Building a DSL on top of JetBrains MPS will give you most, if not all of that. You can distribute an IntelliJ plugin or even a standalone IDE. https://www.jetbrains.com/mps/ https://www.jetbrains.com/mps/
  - notacoward 6y ago
    Thanks, that looks interesting. AFAICT it only covers the editor/IDE parts, though, and that's not even close to "most" of what I mentioned. How does it help create linters and checkers? How does it ease integration with debuggers and build systems? Plus, you have to use JetBrains to get even that. Nothing against JetBrains, but that's not going to help at most companies which have already settled on other tools. It looks like a slightly easier way to create the core part of a DSL, but I'm not sure that solves the problem of the result being an "alien" thing that other developers will develop distaste for.
    - bnr 6y ago
      It uses a projectional editor, so there is only one way of "formatting" the code and no need for a linter. The typesystem is very powerful and allows a language engineer to create arbitrary checks on the language which are executed inside the IDE. I have not tried building a debugger for a DSL in MPS, but it might be achievable, at least if you're targeting Java as a generated language. Build integration is available for Maven and Gradle.
throw_m239339 6y ago
SQL is pretty successful as a DSL which was intended for non programmers. XML,HTML,CSS are also successful. YAML/JSON/... based configurations are also used in a lot of apps. And then there are LISP dialects, which in theory are the best tool to build any kind of DSL quickly, but I have never seen them used in production anywhere I worked, and it doesn't look like end users would find it easy to work with. How many of you had to develop a DSL from scratch for end users and domain experts? How did it go? Did end users actually ended up using it and were they satisfied with the syntax?
dstola 6y ago
IMO a big reason why there is a decline in DSLs could be attributed to de-priotization of teaching compilers in Universities. There are certainly grad level courses, but (my university as an example) did not have an undergrad level compiler course, not even introduction to. I think if you rekindle interests in compiler for people, you will simultaneously increase the likelihood of DSLs being used to solve problems.
diego_moita 6y ago
Because when they are successful they'll stop being "Specific". It's own success conspires against its nature. Look at the history of Lua: it began as a language for config files, similar to makefiles, config.ini, xml or json. But it solved this "problem" so well that people wanted it to become more powerful. And Lua did it without compromising too much it's simplicity. Then it stopped being just a config language. Same goes with JavaScript: in the beginning it was just for small scripts on Webpages, today is much more than that. People will want power and versatility in a language. And they'll find that in Python, JavaScript, R or Lua. They'll not find it in a DSL.
credit_guy 6y ago
Speaking of DSL's, what is the status nowadays in creating DSL's in Python?
- ianandrich 6y ago
  Textx is the only game in town. https://github.com/textX/textX https://github.com/textX/textX Last commit was a week ag
juancn 6y ago
They miss the main problem: tooling. DSLs are nice, but you need to integrate them into a development workflow which means strong IDE support and build systems. Auto complete, testing, backward compatibility, etc. If I normally have code in my IDE, where I right click on a test and choose run, everything is done for me, no fuss. If I integrate a DSL, things stop working as normal. Unless it's a widely supported DSL, such as regular expressions.
permille42 6y ago
This article misses the broader goal/reason to use DSLs. That reason is metaprogramming. DSLs are very powerful and extremely effective when they simplify how some specific task or sequence must be configured. When you do something repeatedly in a programming language where it seems like there is a lot of copy/paste, that is exactly when a DSL should be created and applied to avoid that sort of behavior. In this sense, good DSLs are deeply related to the low-code movement. When enough DSLs are made, need to write all logic in a general purpose programming language will be minimalized. The place where this can most be seen currently is in process management systems and the DSLs used to configure them. Few are familiar with these because they are very expensive enterprise tools used to rapidly setup business processes and related interfaces.

25 points by bnr 6y ago

bnr 6y ago

I'm thoroughly convinced that software engineering efficiency can be increased by at least one order of magnitude by letting domain experts and product owners directly modify the product through a well defined DSL.

aszen 6y ago
To create that ideal dsl one has to both know the domain well enough and have the technical skills which is a rare combination so i'm rather sceptical of this claim. Besides any dsl can only help with well understood repeatable problems, for problems that aren't covered by a dsl, software engineers are still required. The fundamental role of software engineers is to build easy to use and insightful interfaces to understand complex data generated/collected from the real world. To do that one needs to have the skill to organise information and control complexity by data hiding not exactly the skills product owners and domain experts are known for.
jamesblonde 6y ago
We built the first open-source feature store for ML, https://github.com/logicalclocks/hopsworks https://github.com/logicalclocks/hopsworks , when every existing proprietary feature store (Uber Michelangelo and Bighead at AirBnb) were shouting about how their DSL for feature engineering was the future. Fast-forward 2 years and it is clear that Data Scientists want to work with Python, not with a DSL. We based our Feature Store on a Dataframe API for Python/PySpark. The DSL can never evolve at the same rate as libraries in a general-purpose programming language. So, your DSL is great for show-casing a Feature Store, but when you need to compute embeddings or train a GAN or done any type of feature engineering that is not a simple time-window aggregation, you pull out Python (or Scala/Java). I am old enough to have seen many DSLs in different domains (GUIs, aspect-oriented programming, feature engineering) have their day in the sun only to be replaced by general-purpose programming languages due to their unmatched utility.
- bnr 6y ago
  The article (controversely, maybe) classifies libraries for general-purpose programming languages as internal DSLs. One could argue that Data Scientists working with libraries in Pythin are already using a DSL, just with an escape hatch into the general-purpose world. I don't think proper vertical DSLs should be made marketed towards people who are comfortable working in a general-purpose language. I see them as a way to help non-technical domain experts work on code instead of specification. Limiting the possibilities of what one can write, like with MPS' projectional editor, is a feature here and not a bug.
  - jamesblonde 6y ago
    A library is not a DSL, despite what the learned authors may claim. In fact, there are a couple of examples of "successful" DSLs in the data world - you could argue that Talend and DBT are visual programming tools for ETL pipelines. Defintely Zapier- which is just for integrating services that have well-defined REST APIs.
layoutIfNeeded 6y ago
I guess you haven't had the "pleasure" of cleaning up the mess made by domain experts and product owners using those well-defined DSLs.

tinco 6y ago

Domain specific languages just didn't settle at the level we expected. We thought they would be for really narrow domains, like a language for each kind of business logic. But instead the domains are more at the architecture level and some very broad purpose (i.e. HTML, SQL, etc). At my company, we use Ruby for web backend and system administration. We use Typescript for web frontend. We use C# for process automation. We use Rust and C++ for real time performance (we've got an in house 3D engine), and a little bit of high performance processing. We use Python for data processing and machine learning. That's six "general purpose" languages, each applied to a specific domain, on six software developers, in three teams of two. Each happy with their choice of languages, effectively solving the business needs. I think at some point in the past this situation was described as a nightmare. But I think at that point people thought developers were something you just opened a tin can of, and then applied to whatever problem you had. Nowadays we hire for a specific purpose (or at least I do). When I hire web dev experts, I expect them to be fluent in the industry standard web programming languages and platforms of our choice (i.e. for us it'd be Ruby and Typescript). When I hire 3D experts I expect them to be fluent in C++, and proficient in Rust (it's the future). And obviously it's Python for the data science types. That's a very clear world to me, and to be extra clear, I'm not saying any of the languages I mentioned are the best choice. As long as we operate within the lines the industries have drawn, you can draw upon the best libraries and ecosystems for your particular problem. I would never approve a 3D engine to be developed in Ruby, or a data science pipeline in Javascript. I'd rather have a Ruby developer learn some Python so they'd be able to work on the data science pipeline (disregarding their complaints about Python's inelegance) than have them trying to kludge together subpar Ruby libraries.

anothernewdude 6y ago

It is a bit telling those were the best reasons they could think of for lack of adoption. Just "ignorance". Not we thought that maybe specifying a DSL rather than a library would leave our users and clients in a state of having to banadage over the constraints of a DSL that doesn't handle future use cases, like say dealing with control flow in a half-assed YAML based language, I'm looking at you Ansible. I don't want MAKE, or whatever DSL, I want to be able to drop into a real programming language when necessary. So libraries. Not frameworks, Not DSLs, libraries. And thankfully, it seems the world agrees with me.

rualca 6y ago
> I don't want MAKE, or whatever DSL, I want to be able to drop into a real programming language when necessary. So libraries. That's one opinion, sure. However, DSLs are always in a far better position to solve domain-specific problems because, unlike generic programming libraries, they not only reflect domain-specific knowledge and best practices and represent standardized solutions for recurring problems. Consequently, Makefiles and other DAG-related DSLs are omnipresent and dominate domains such as build systems, and generic library-based proposals always failed to gain any form of traction. And it's not just Makefiles or build systems. There's also markup languages, infrastructure as code, CICD/processing pipelines, configuration, etc etc etc.
- cle 6y ago
  It’s funny you mention the latter, I’ve seen an industry-wide trend toward using standard programming languages with those things (Pulumi, CDK, et al). I’ve started using them myself. I’m not convinced yet that they’re better. Gradle seems to be an interesting case of a successful build system with an interface in a real language (Groovy), although Groovy is so dynamic that the line between Groovy and DSL is pretty blurry. I think another big issue is the generally terrible interop between programming languages, each language has a silo of excellent libraries and the only way to move things between silos is with unsophisticated data formats. I’ve also made a handful of developer libraries and always pick a cludgy data format for configuration, the ergonomics are quite nice—it’s easy to reason about as a user (no head scratching about what the config will look like at “runtime”), there’s generally no tooling setup required and no additional compilers or interpreters to install, and I’m free to use whatever languages as a tool designer I like and can move freely between them (at this point they all support YAML, JSON, et al). We want to make the barrier to entry as low as possible and simple data formats do just that.

notacoward 6y ago

It seems to me that the main reason DSLs aren't more widely adopted is that any DSL will be unsupported by other tools developers consider more important. Your IDE won't have syntax highlighting and auto completion for it, and a lot of developers nowadays seem crippled without those. Linters and semantic checkers won't understand it. Nor will debuggers. There will be no mocking libraries or makefile rules for it. The list goes on and on. These are problems that every new non-domain-specific language has to address. It's quite a lot, and most of it is pretty tedious compared to designing the language itself. So even those who try to create DSLs often skip most of the "extra" bits, and other developers learn to hate DSLs.

bnr 6y ago
Building a DSL on top of JetBrains MPS will give you most, if not all of that. You can distribute an IntelliJ plugin or even a standalone IDE. https://www.jetbrains.com/mps/ https://www.jetbrains.com/mps/
- notacoward 6y ago
  Thanks, that looks interesting. AFAICT it only covers the editor/IDE parts, though, and that's not even close to "most" of what I mentioned. How does it help create linters and checkers? How does it ease integration with debuggers and build systems? Plus, you have to use JetBrains to get even that. Nothing against JetBrains, but that's not going to help at most companies which have already settled on other tools. It looks like a slightly easier way to create the core part of a DSL, but I'm not sure that solves the problem of the result being an "alien" thing that other developers will develop distaste for.
  - bnr 6y ago
    It uses a projectional editor, so there is only one way of "formatting" the code and no need for a linter. The typesystem is very powerful and allows a language engineer to create arbitrary checks on the language which are executed inside the IDE. I have not tried building a debugger for a DSL in MPS, but it might be achievable, at least if you're targeting Java as a generated language. Build integration is available for Maven and Gradle.

throw_m239339 6y ago

SQL is pretty successful as a DSL which was intended for non programmers. XML,HTML,CSS are also successful. YAML/JSON/... based configurations are also used in a lot of apps. And then there are LISP dialects, which in theory are the best tool to build any kind of DSL quickly, but I have never seen them used in production anywhere I worked, and it doesn't look like end users would find it easy to work with. How many of you had to develop a DSL from scratch for end users and domain experts? How did it go? Did end users actually ended up using it and were they satisfied with the syntax?

dstola 6y ago

IMO a big reason why there is a decline in DSLs could be attributed to de-priotization of teaching compilers in Universities. There are certainly grad level courses, but (my university as an example) did not have an undergrad level compiler course, not even introduction to. I think if you rekindle interests in compiler for people, you will simultaneously increase the likelihood of DSLs being used to solve problems.

diego_moita 6y ago

Because when they are successful they'll stop being "Specific". It's own success conspires against its nature. Look at the history of Lua: it began as a language for config files, similar to makefiles, config.ini, xml or json. But it solved this "problem" so well that people wanted it to become more powerful. And Lua did it without compromising too much it's simplicity. Then it stopped being just a config language. Same goes with JavaScript: in the beginning it was just for small scripts on Webpages, today is much more than that. People will want power and versatility in a language. And they'll find that in Python, JavaScript, R or Lua. They'll not find it in a DSL.

credit_guy 6y ago

Speaking of DSL's, what is the status nowadays in creating DSL's in Python?

ianandrich 6y ago
Textx is the only game in town. https://github.com/textX/textX https://github.com/textX/textX Last commit was a week ag

juancn 6y ago

They miss the main problem: tooling. DSLs are nice, but you need to integrate them into a development workflow which means strong IDE support and build systems. Auto complete, testing, backward compatibility, etc. If I normally have code in my IDE, where I right click on a test and choose run, everything is done for me, no fuss. If I integrate a DSL, things stop working as normal. Unless it's a widely supported DSL, such as regular expressions.

permille42 6y ago

This article misses the broader goal/reason to use DSLs. That reason is metaprogramming. DSLs are very powerful and extremely effective when they simplify how some specific task or sequence must be configured. When you do something repeatedly in a programming language where it seems like there is a lot of copy/paste, that is exactly when a DSL should be created and applied to avoid that sort of behavior. In this sense, good DSLs are deeply related to the low-code movement. When enough DSLs are made, need to write all logic in a general purpose programming language will be minimalized. The place where this can most be seen currently is in process management systems and the DSLs used to configure them. Few are familiar with these because they are very expensive enterprise tools used to rapidly setup business processes and related interfaces.