7 ms·

Talk of metaprogramming intrigues me. I'd like to hear what a Lisp user makes of it because I find non-Lisp users are usually amazed by any metaprogramming at a

Talk of metaprogramming intrigues me. I'd like to hear what a Lisp user makes of it because I find non-Lisp users are usually amazed by any metaprogramming at all and can't be as critical about it.

PMunch 7y ago
Not a Lisp user, but a heavy Nim macro user. Nim macros are AST based, which means that after the parser part of the compiler have read the code and created a tree structure of it your macro is called on this tree. Then you can modify this tree to your hearts desire and return a new tree that the Nim compiler will splice back into the main tree before continuing. This means that you rewrite pretty much anything into pretty much anything else. You also have access to reading files on the machine, or doing pretty much anything else that Nim offers. For examples of what you can do with it you can have a look at my macro to create GUIs: https://peterme.net/cross-platform-guis-and-nim-macros.html https://peterme.net/cross-platform-guis-and-nim-macros.html, my article on how I use macros to create more read- and maintainable code: https://peterme.net/metaprogramming-and-read-and-maintainability-in-nim.html https://peterme.net/metaprogramming-and-read-and-maintainabi... or perhaps see how Protobuf was written as a string parsing macro which reads a Protobuf specification and spits out all the types and procedures you need to use it: https://github.com/PMunch/protobuf-nim https://github.com/PMunch/protobuf-nim
- lispm 7y ago
  > which means that after the parser part of the compiler Lisp works slightly different. The macro form is an expression (a list) with the macro operator as the first element. The next elements in the expression can be anything as data.
arc776 7y ago
Metaprogramming is one of the core ideals in the birth of the language so is well supported. Personally I've not used Lisp, so can't comment there, but it is on the list of influences on the homepage. Essentially there's a VM that runs almost all the language barring importc type stuff, and you can chuck around AST node objects to create code, so metaprogramming is done in the core Nim language. You can read files so it's easy to slurp files and use them to generate code or other data processing at compile-time. Several simple utility operations in the stdlib make things really fluid; the easy ability to `quote` blocks of code to AST, and outputting nodes and code to string. This lets you both hack something together quickly and learn over time how the syntax trees work. Quoting looks like this: macro repeat(count: int, code: untyped): untyped = quote do: for i in 0..<`count`: `code` repeat(10): echo "Hello" Inspecting something's AST can be done with dumpTree: dumpTree: let x = 1 y = 2 echo "Hello ", x + y To save even more effort, there's also dumpAstGen which outputs the code to create that node manually, and even a dumpLisp! You can display ASTs inside macros: macro showMe(input: untyped): untyped = echo "Input as written:", input.repr echo "Input as AST tree:", input.treerepr result = quote: `input` + `input` echo result.repr echo result.treerepr So it's really easy to debug what went wrong if you're generating lots of code. Since untyped parameters to a macro don't have to be valid Nim code (though they still follow syntax rules) you can make your own DSLs really easily and reliably because any input is pre-parsed into a nice tree for you. Here's a contrived example of some simple DSL that lets you call procs and store their results for later output: import macros, tables macro process(items: untyped): untyped = result = newStmtList() # Create hash table of 'perform' names to store their result variables. var performers: Table[string, NimNode] for item in items: let command = item[0] param = item[1] paramStr = $param case $command of "perform": # Check if we've already generated a var for holding the return value. var node = performers.getOrDefault(paramStr) if node == nil: # Generate a variable name to store the performer result in. # genSym guarantees a unique name. node = genSym(nskVar, paramStr) performers.add(paramStr, node) # Add the variable declaration result.add(quote do: var `node` = `param`() ) else: # A repeat performance, we don't need to declare the variable and can overwrite the # value in the fetched variable. result.add(quote do: `node` = `param`() ) of "output": let node = performers.getOrDefault(paramStr) if node == nil: quit "Cannot find performer " & paramStr result.add(quote do: echo `node`) else: discard # Display the resultant code. echo result.repr proc foo: string = "foo!" proc bar: string = "bar!" process: perform foo perform bar perform foo output foo output bar The generated output from process looks like: var foo262819 = foo() var bar262821 = bar() foo262819 = foo() echo foo262819 echo bar262821
- arc776 7y ago
  In hindsight maybe the "performers" table would be better named "resultVars", as that's what it really represents.
pjmlp 7y ago
We do metaprogramming all the time on JVM and .NET languages.
thsealienbstrds 7y ago
Disclaimer: I only did a hobby project in Lisp once. But I did use some of the macro functionality. Judging by some of the comments here, it seems like the macro system has a similar approach as Lisp's macro system, which is also AST-based. Something I don't see here is macros that generate other macros, but the question is how much you really want that anyway (when I did that, I thought the syntax was horribly complicated because of all the quoting). I know Lisp also has reader macros (they run before the parser) that allow you to effectively change the language syntax, but I didn't use those.
- lispm 7y ago
  Lisp macros are not based on an AST. The macro forms in Lisp need to be valid s-expressions and begin with a macro operator. That's it.
  - thsealienbstrds 7y ago
    Still, isn't it AST-based in the sense that the input of a Lisp macro is effectively a parsed syntax tree, just as in Nim?
    - lispm 7y ago
      The only restriction is that it is a s-expression: a possibly nested list of data. There is no particular programming language syntax it is parsed against - it's parsed as data. For example this is a valid Lisp macro form (loop for i below 10 and j downfrom 20 when (= (+ i j) 9) sum i into isum of-type integer finally (return (+ j isum))) It returns 10. The LOOP macro parses it on its own - it just sees a list of data. Lisp has other than that no idea what the tokens mean and what syntax the LOOP macro accepts. CL-USER 142 > (defmacro my-macro (&rest stuff) t) MY-MACRO CL-USER 143 > (my-macro we are going to Europe and have a good time) T Works.
      - kazinator 7y ago
        The data has an AST. The AST is made of conses, and atoms. The atoms have a large variety of types. They are not "tokens", but objects. Tokens only exist briefly inside the Lisp reader. 123 and |123| are tokens; one converts to an integer node, one to a symbol node. (1 (2 3)) has an AST which is CONS / \ FIXNUM CONS | / \ 1 CONS SYMBOL / \ \ FIXNUM CONS NIL | / \ 2 FIXNUM SYMBOL | | 3 NIL

Talk of metaprogramming intrigues me. I'd like to hear what a Lisp user makes of it because I find non-Lisp users are usually amazed by any metaprogramming at all and can't be as critical about it.

PMunch 7y ago

Not a Lisp user, but a heavy Nim macro user. Nim macros are AST based, which means that after the parser part of the compiler have read the code and created a tree structure of it your macro is called on this tree. Then you can modify this tree to your hearts desire and return a new tree that the Nim compiler will splice back into the main tree before continuing. This means that you rewrite pretty much anything into pretty much anything else. You also have access to reading files on the machine, or doing pretty much anything else that Nim offers. For examples of what you can do with it you can have a look at my macro to create GUIs: https://peterme.net/cross-platform-guis-and-nim-macros.html https://peterme.net/cross-platform-guis-and-nim-macros.html, my article on how I use macros to create more read- and maintainable code: https://peterme.net/metaprogramming-and-read-and-maintainability-in-nim.html https://peterme.net/metaprogramming-and-read-and-maintainabi... or perhaps see how Protobuf was written as a string parsing macro which reads a Protobuf specification and spits out all the types and procedures you need to use it: https://github.com/PMunch/protobuf-nim https://github.com/PMunch/protobuf-nim

lispm 7y ago
> which means that after the parser part of the compiler Lisp works slightly different. The macro form is an expression (a list) with the macro operator as the first element. The next elements in the expression can be anything as data.

arc776 7y ago

Metaprogramming is one of the core ideals in the birth of the language so is well supported. Personally I've not used Lisp, so can't comment there, but it is on the list of influences on the homepage. Essentially there's a VM that runs almost all the language barring importc type stuff, and you can chuck around AST node objects to create code, so metaprogramming is done in the core Nim language. You can read files so it's easy to slurp files and use them to generate code or other data processing at compile-time. Several simple utility operations in the stdlib make things really fluid; the easy ability to `quote` blocks of code to AST, and outputting nodes and code to string. This lets you both hack something together quickly and learn over time how the syntax trees work. Quoting looks like this: macro repeat(count: int, code: untyped): untyped = quote do: for i in 0..<`count`: `code` repeat(10): echo "Hello" Inspecting something's AST can be done with dumpTree: dumpTree: let x = 1 y = 2 echo "Hello ", x + y To save even more effort, there's also dumpAstGen which outputs the code to create that node manually, and even a dumpLisp! You can display ASTs inside macros: macro showMe(input: untyped): untyped = echo "Input as written:", input.repr echo "Input as AST tree:", input.treerepr result = quote: `input` + `input` echo result.repr echo result.treerepr So it's really easy to debug what went wrong if you're generating lots of code. Since untyped parameters to a macro don't have to be valid Nim code (though they still follow syntax rules) you can make your own DSLs really easily and reliably because any input is pre-parsed into a nice tree for you. Here's a contrived example of some simple DSL that lets you call procs and store their results for later output: import macros, tables macro process(items: untyped): untyped = result = newStmtList() # Create hash table of 'perform' names to store their result variables. var performers: Table[string, NimNode] for item in items: let command = item[0] param = item[1] paramStr = $param case $command of "perform": # Check if we've already generated a var for holding the return value. var node = performers.getOrDefault(paramStr) if node == nil: # Generate a variable name to store the performer result in. # genSym guarantees a unique name. node = genSym(nskVar, paramStr) performers.add(paramStr, node) # Add the variable declaration result.add(quote do: var `node` = `param`() ) else: # A repeat performance, we don't need to declare the variable and can overwrite the # value in the fetched variable. result.add(quote do: `node` = `param`() ) of "output": let node = performers.getOrDefault(paramStr) if node == nil: quit "Cannot find performer " & paramStr result.add(quote do: echo `node`) else: discard # Display the resultant code. echo result.repr proc foo: string = "foo!" proc bar: string = "bar!" process: perform foo perform bar perform foo output foo output bar The generated output from process looks like: var foo262819 = foo() var bar262821 = bar() foo262819 = foo() echo foo262819 echo bar262821

arc776 7y ago
In hindsight maybe the "performers" table would be better named "resultVars", as that's what it really represents.

pjmlp 7y ago

We do metaprogramming all the time on JVM and .NET languages.

thsealienbstrds 7y ago

Disclaimer: I only did a hobby project in Lisp once. But I did use some of the macro functionality. Judging by some of the comments here, it seems like the macro system has a similar approach as Lisp's macro system, which is also AST-based. Something I don't see here is macros that generate other macros, but the question is how much you really want that anyway (when I did that, I thought the syntax was horribly complicated because of all the quoting). I know Lisp also has reader macros (they run before the parser) that allow you to effectively change the language syntax, but I didn't use those.

lispm 7y ago
Lisp macros are not based on an AST. The macro forms in Lisp need to be valid s-expressions and begin with a macro operator. That's it.
- thsealienbstrds 7y ago
  Still, isn't it AST-based in the sense that the input of a Lisp macro is effectively a parsed syntax tree, just as in Nim?
  - lispm 7y ago
    The only restriction is that it is a s-expression: a possibly nested list of data. There is no particular programming language syntax it is parsed against - it's parsed as data. For example this is a valid Lisp macro form (loop for i below 10 and j downfrom 20 when (= (+ i j) 9) sum i into isum of-type integer finally (return (+ j isum))) It returns 10. The LOOP macro parses it on its own - it just sees a list of data. Lisp has other than that no idea what the tokens mean and what syntax the LOOP macro accepts. CL-USER 142 > (defmacro my-macro (&rest stuff) t) MY-MACRO CL-USER 143 > (my-macro we are going to Europe and have a good time) T Works.
    - kazinator 7y ago
      The data has an AST. The AST is made of conses, and atoms. The atoms have a large variety of types. They are not "tokens", but objects. Tokens only exist briefly inside the Lisp reader. 123 and |123| are tokens; one converts to an integer node, one to a symbol node. (1 (2 3)) has an AST which is CONS / \ FIXNUM CONS | / \ 1 CONS SYMBOL / \ \ FIXNUM CONS NIL | / \ 2 FIXNUM SYMBOL | | 3 NIL