5 ms·

Ask HN: Writing DSL in Python

26 points by oltmans 18y ago

Hey Hackers, I was reading this blog article http://weblog.jamisbuck.org/2006/4/20/writing-domain-specific-languages and I've no idea how can one go about writing a DSL in Python. I've rudimentary knowledge of Python and I'm interesting in writing a DSL just for fun and knowledge. Can anyone please point out what do I need to learn to be able to write a DSL in Python? Any help is appreciated.

apgwoz 18y ago
Study up on context managers, decorators, and meta-classes, which will assist in creating DSLs. However, you don't really need to use these features to build DSLs. The use of method chaining (think jQuery) can lead to interesting languages. Or, take Django's chaining for making queries. User.objects.filter(first_name='Andrew').order_by('last_name') Or, you can get more complicated and do something like described at: http://code.activestate.com/recipes/534150/ http://code.activestate.com/recipes/534150/ EDIT: s/be vital/assist/ on line 1.
- cabalamat 18y ago
  > Study up on context managers, decorators, and meta-classes, which will be vital to creating DSLs. However, you don't really need to use these features to build DSLs. Make your mind up. Either they're vital or they're not. You can't have both.
  - apgwoz 18y ago
    No. I won't make my mind up. I'm not providing a recipe for DSLs in Python. I'm simply stating ingredients and saying "now go create."
    - petercooper 18y ago
      I think he/she was more criticizing that you effectively said "X, Y and Z are vital" and then "But X, Y and Z are not vital." If you'd said "useful" instead of "vital" it'd have been fine logically, but "vital" implies necessity.
      - apgwoz 18y ago
        Yeah. I ended up editing the original post when I realized it later.
- russell 18y ago
  apgwoz has a excellent suggestion. Python has a clean syntax, so you may not need to invent your own. Meta-classes allow you to do the DSL magic behind the scenes. It's less fun than inventing a whole new language, but you can get something working in a couple of hours. Even more useful are the mind stretching exercises inherent in understanding meta-classes and their friends. You will see programming in a whole new light.
jlc 18y ago
I ran into this a while back: http://blog.brianbeck.com/post/53538107/python-dsl-i http://blog.brianbeck.com/post/53538107/python-dsl-i
ivank 18y ago
You could look at SQLAlchemy, Storm, Schevo, or Nevow Stan. Or even BeautifulSoup's element-finding.
nuclear_eclipse 18y ago
I still haven't quite grasped the concept behind what a DSL really is, and what makes your code a DSL versus any other program or API. Any insight that can be offered, especially in the context of something other than Ruby and Lisp? I just want to know what all the hubbub is about, and why it matters to me?
- jerf 18y ago
  There's multiple definitions, so what follows is not authoritative, but IMHO it will help you understand the field better, as well as why so many people seem to be talking past each other as they silently adopt one definition while arguing with someone based on another. A strong DSL is a specialized language designed for a specific task, with its own parser and syntax not directly based on another language. Ideally, it is not Turing complete, because that opens a huge can of worms, and much of the point of a strong DSL is to avoid this can of worms. The DSL should be carefully designed in conjunction with the eventual users, who most likely will not be programmers. I've heard of impressive results with this approach, but I've never witnessed them firsthand. This definition is favored by Martin Fowler, and much less importantly/impressively, me. Weak DSLs are basically APIs written such that in their target containing language, the calls into the API read conversationally and with minimal boilerplate not directly related to the domain of the problem. In general (and I emphasize that I am generalizing), they can't be used to their full power without understanding the containing language, so you still need to be a programmer, albeit perhaps with less experience. You may see demos on the home page for the DSL about how it is possible to use without knowing the containing language, but it tends to be demoware; much of the gain is intrinsically that you still have a full Turing-complete language backing you up. People will bikeshed about punctuation and claim that Ruby can build better weak DSLs, but in my experience, like I said, you still have to understand the containing language to really use the language, so the virtue of dropping off punctuation marks is oversold; the syntactic constructs are still there and you still need to understand them, so whether you actually see them in source is much more a matter of taste than a matter of "goodness". Mixing people who use different definitions of DSL is a recipe for much heat and no light. It should also be pointed out that the strong DSL people make certain claims about why DSLs are good, along with the reasons why these things are true, and these claims should not be carelessly translated to weak DSLs. While I use the terms "strong" and "weak", they should be understood as descriptive, not a claim of goodness, much like "strong AI" and "weak AI". If you look at a weak DSL and wonder what the deal is, it is because rather than a magical technology, it is merely a style of API... there really isn't much "there" there, but it can be useful in some cases. I would never write a weak DSL for a DSL's sake, though; just evolve the API and see what happens. Personally, I think it is a continuum between "API" and "weak DSL", not a binary distinction, but for "strong DSL", I would say that if you're backing to a full Turing complete language like Ruby, you do not have a strong DSL and should not expect to reap the benefits. (Which is good, since you're not paying the price, either. Nothing is free.)
tocomment 18y ago
I think it just means writing a programming language but a little one. So use pyparsing as your parser. The May 2008 Python magazine has a good article on making an interpreter/compiler for brainf*ck using PyParsing.
- wilkes 18y ago
  I've really enjoyed using PyParsing. O'Reilly has a ShortCut book on it that is worthwhile. http://oreilly.com/catalog/9780596514235/ http://oreilly.com/catalog/9780596514235/
cabalamat 18y ago
I have implemented two languages in Python, one a DSL, the other a general-purpose language. Both times I used the SPARK parsing framework http://pages.cpsc.ucalgary.ca/~aycock/spark/ http://pages.cpsc.ucalgary.ca/~aycock/spark/ which I recommend.
bayareaguy 18y ago
I would recommend taking a quick look at how SCons and Conary use python. You may also want to consider implementing your DSL using a python parser generator such as Ply. SCons - http://www.scons.org/doc/HTML/scons-man.html#lbBB http://www.scons.org/doc/HTML/scons-man.html#lbBB Conary - http://wiki.rpath.com/wiki/Conary:Recipe_Structure http://wiki.rpath.com/wiki/Conary:Recipe_Structure Ply - http://www.dalkescientific.com/writings/NBN/parsing_with_ply.html http://www.dalkescientific.com/writings/NBN/parsing_with_ply...
nostrademons 18y ago
I used yapps2 for the DSL parts of GameClay. It's probably not the best Python parser generator out there, but it was installable via apt, simple to use, and familiar to people who have used Antlr. Had a simple DSL written in 2 days. Also, if it's just a quick-n-dirty hack, don't underestimate the power of regexp/split() parsing. Yeah, everyone says you shouldn't do it, and you probably shouldn't if you're planning on maintaining things for a while, but you can get going really fast without learning any additional tools. And for maybe 80-90% of the things you'd use a DSL for, it's perfectly adequate.
intellectronica 18y ago
You can customize the behaviour of Python objects in ways which may allow you to implement DSLs (for example, by responding to arbitrary messages, or by constructing callables). A good example to look at might be https://storm.canonical.com/ https://storm.canonical.com/ which utilizes every trick in the book to allows you to write SQL-like queries in Python.
gruseom 18y ago
Although I got excited about it at first, I've come to the conclusion that this DSL stuff is seriously overrated. I've seen few successful examples and many silly ones. There are certainly good little specialized languages -- Common Lisp's LOOP is an example, as are regular expressions -- but these are for programmers, not domain experts.
thorax 18y ago
No joke: take a look at the LOLpython source for an example of creating a variant language that can compile to Python bytecode. It's one of many methods to create a DSL, though you often don't need access to all of Python in the DSL itself.
jgalvez 18y ago
http://github.com/galvez/gae-rest/tree/master/xmlbuilder.py http://github.com/galvez/gae-rest/tree/master/xmlbuilder.py

26 points by oltmans 18y ago

apgwoz 18y ago

Study up on context managers, decorators, and meta-classes, which will assist in creating DSLs. However, you don't really need to use these features to build DSLs. The use of method chaining (think jQuery) can lead to interesting languages. Or, take Django's chaining for making queries. User.objects.filter(first_name='Andrew').order_by('last_name') Or, you can get more complicated and do something like described at: http://code.activestate.com/recipes/534150/ http://code.activestate.com/recipes/534150/ EDIT: s/be vital/assist/ on line 1.

cabalamat 18y ago
> Study up on context managers, decorators, and meta-classes, which will be vital to creating DSLs. However, you don't really need to use these features to build DSLs. Make your mind up. Either they're vital or they're not. You can't have both.
- apgwoz 18y ago
  No. I won't make my mind up. I'm not providing a recipe for DSLs in Python. I'm simply stating ingredients and saying "now go create."
  - petercooper 18y ago
    I think he/she was more criticizing that you effectively said "X, Y and Z are vital" and then "But X, Y and Z are not vital." If you'd said "useful" instead of "vital" it'd have been fine logically, but "vital" implies necessity.
    - apgwoz 18y ago
      Yeah. I ended up editing the original post when I realized it later.
russell 18y ago
apgwoz has a excellent suggestion. Python has a clean syntax, so you may not need to invent your own. Meta-classes allow you to do the DSL magic behind the scenes. It's less fun than inventing a whole new language, but you can get something working in a couple of hours. Even more useful are the mind stretching exercises inherent in understanding meta-classes and their friends. You will see programming in a whole new light.

jlc 18y ago

I ran into this a while back: http://blog.brianbeck.com/post/53538107/python-dsl-i http://blog.brianbeck.com/post/53538107/python-dsl-i

ivank 18y ago

You could look at SQLAlchemy, Storm, Schevo, or Nevow Stan. Or even BeautifulSoup's element-finding.

nuclear_eclipse 18y ago

I still haven't quite grasped the concept behind what a DSL really is, and what makes your code a DSL versus any other program or API. Any insight that can be offered, especially in the context of something other than Ruby and Lisp? I just want to know what all the hubbub is about, and why it matters to me?

jerf 18y ago
There's multiple definitions, so what follows is not authoritative, but IMHO it will help you understand the field better, as well as why so many people seem to be talking past each other as they silently adopt one definition while arguing with someone based on another. A strong DSL is a specialized language designed for a specific task, with its own parser and syntax not directly based on another language. Ideally, it is not Turing complete, because that opens a huge can of worms, and much of the point of a strong DSL is to avoid this can of worms. The DSL should be carefully designed in conjunction with the eventual users, who most likely will not be programmers. I've heard of impressive results with this approach, but I've never witnessed them firsthand. This definition is favored by Martin Fowler, and much less importantly/impressively, me. Weak DSLs are basically APIs written such that in their target containing language, the calls into the API read conversationally and with minimal boilerplate not directly related to the domain of the problem. In general (and I emphasize that I am generalizing), they can't be used to their full power without understanding the containing language, so you still need to be a programmer, albeit perhaps with less experience. You may see demos on the home page for the DSL about how it is possible to use without knowing the containing language, but it tends to be demoware; much of the gain is intrinsically that you still have a full Turing-complete language backing you up. People will bikeshed about punctuation and claim that Ruby can build better weak DSLs, but in my experience, like I said, you still have to understand the containing language to really use the language, so the virtue of dropping off punctuation marks is oversold; the syntactic constructs are still there and you still need to understand them, so whether you actually see them in source is much more a matter of taste than a matter of "goodness". Mixing people who use different definitions of DSL is a recipe for much heat and no light. It should also be pointed out that the strong DSL people make certain claims about why DSLs are good, along with the reasons why these things are true, and these claims should not be carelessly translated to weak DSLs. While I use the terms "strong" and "weak", they should be understood as descriptive, not a claim of goodness, much like "strong AI" and "weak AI". If you look at a weak DSL and wonder what the deal is, it is because rather than a magical technology, it is merely a style of API... there really isn't much "there" there, but it can be useful in some cases. I would never write a weak DSL for a DSL's sake, though; just evolve the API and see what happens. Personally, I think it is a continuum between "API" and "weak DSL", not a binary distinction, but for "strong DSL", I would say that if you're backing to a full Turing complete language like Ruby, you do not have a strong DSL and should not expect to reap the benefits. (Which is good, since you're not paying the price, either. Nothing is free.)

tocomment 18y ago

I think it just means writing a programming language but a little one. So use pyparsing as your parser. The May 2008 Python magazine has a good article on making an interpreter/compiler for brainf*ck using PyParsing.

wilkes 18y ago
I've really enjoyed using PyParsing. O'Reilly has a ShortCut book on it that is worthwhile. http://oreilly.com/catalog/9780596514235/ http://oreilly.com/catalog/9780596514235/

cabalamat 18y ago

I have implemented two languages in Python, one a DSL, the other a general-purpose language. Both times I used the SPARK parsing framework http://pages.cpsc.ucalgary.ca/~aycock/spark/ http://pages.cpsc.ucalgary.ca/~aycock/spark/ which I recommend.

bayareaguy 18y ago

I would recommend taking a quick look at how SCons and Conary use python. You may also want to consider implementing your DSL using a python parser generator such as Ply. SCons - http://www.scons.org/doc/HTML/scons-man.html#lbBB http://www.scons.org/doc/HTML/scons-man.html#lbBB Conary - http://wiki.rpath.com/wiki/Conary:Recipe_Structure http://wiki.rpath.com/wiki/Conary:Recipe_Structure Ply - http://www.dalkescientific.com/writings/NBN/parsing_with_ply.html http://www.dalkescientific.com/writings/NBN/parsing_with_ply...

nostrademons 18y ago

I used yapps2 for the DSL parts of GameClay. It's probably not the best Python parser generator out there, but it was installable via apt, simple to use, and familiar to people who have used Antlr. Had a simple DSL written in 2 days. Also, if it's just a quick-n-dirty hack, don't underestimate the power of regexp/split() parsing. Yeah, everyone says you shouldn't do it, and you probably shouldn't if you're planning on maintaining things for a while, but you can get going really fast without learning any additional tools. And for maybe 80-90% of the things you'd use a DSL for, it's perfectly adequate.

intellectronica 18y ago

You can customize the behaviour of Python objects in ways which may allow you to implement DSLs (for example, by responding to arbitrary messages, or by constructing callables). A good example to look at might be https://storm.canonical.com/ https://storm.canonical.com/ which utilizes every trick in the book to allows you to write SQL-like queries in Python.

gruseom 18y ago

Although I got excited about it at first, I've come to the conclusion that this DSL stuff is seriously overrated. I've seen few successful examples and many silly ones. There are certainly good little specialized languages -- Common Lisp's LOOP is an example, as are regular expressions -- but these are for programmers, not domain experts.

thorax 18y ago

No joke: take a look at the LOLpython source for an example of creating a variant language that can compile to Python bytecode. It's one of many methods to create a DSL, though you often don't need access to all of Python in the DSL itself.

jgalvez 18y ago

http://github.com/galvez/gae-rest/tree/master/xmlbuilder.py http://github.com/galvez/gae-rest/tree/master/xmlbuilder.py