5 ms·

Having played around with Clojure and Scheme for a while (but never got too serious), I always thought homoiconicity and macros were extremely cool concepts, bu

by reifyx 4y ago

Having played around with Clojure and Scheme for a while (but never got too serious), I always thought homoiconicity and macros were extremely cool concepts, but I never actually ran into a need for them in my everyday work. >Now, if we agree that the ability to manipulate source code is important to us, what kind of languages are most conducive for supporting it? This is useful for compiler programmers, or maybe also those writing source code analyzers/optimizers, but is that it? On occasion I have had to write DSLs for the user input, but in these cases the (non-programmer) users didn't want to write Lisp so I used something like Haskell's parsec to parse the data. The remote code example given in the post is compelling, but again seems a bit niche. I don't doubt that it's sometimes useful but is it reason enough to choose the language? Are there examples of real-life (non-compilers-related) Lisp programs that show the power of homoiconicity? Same goes with the concept of "being a guest" in the programming language. I have never wanted to change "constant" to "c". Probably I'm not imaginative enough, but this has never really been an issue for me. Perhaps it secretly has though, and some of my problems have been "being a guest" in disguise.

zmgsabst 4y ago
I remember (faintly, from 20 years ago) using Lisp in game programming to create rules for mobs, from within the game.
agumonkey 4y ago
It's not tied to complex DSLs and compilation. I was frustrated by the lack of f-strings in emacs lisp so I hacked a macro, now I have embedded scoped string interpolation. I can write (f "<{class} {fields}>"). Having these freedom is really not a niche, it's mentally and pragmatically important.
momentoftop 4y ago
> This is useful for compiler programmers, or maybe also those writing source code analyzers/optimizers, but is that it? On occasion I have had to write DSLs for the user input, but in these cases the (non-programmer) users didn't want to write Lisp so I used something like Haskell's parsec to parse the data. If you're talking about Haskell, you should be talking about folk who write template Haskell, which is the macro system for GHC. There are plenty of Haskell programmers who know how to write Template Haskell, and there are plenty of Haskell programmers who don't. By contrast, I don't think there's a single Lisp programmer who can't write Lisp macros. That's homoiconicity. Once you learn Lisp, you automatically know how to write Lisp macros. Once you learn Haskell (or Ocaml or Rust), you don't automatically know how to write macros in that language (and the macro system may not even be portable across compilers).
mtlmtlmtlmtl 4y ago
In my experience, macros in lisp codebases are mainly abstracting away common boilerplate patterns. Common Lisp has WITH-OPEN-FILE to open a file and make sure it closes if the stack unwinds. This is a macro based on the UNWIND-PROTECT primitive, which ensures execution of a form if the stack unwinds. Many projects will have various with-foo macros to express this pattern for arbitray things that are useful in the context of that project(though not all with- macros need UNWIND-PROTECT). Another example is custom loops that happen over and over. Let's say I'm writing a chess engine. I regularly want to iterate over the legal moves in a position. So I might make a macro so I can write (do-moves (mv pos) ...) I find that because doing this is so simple, well written lisp codebases tend to be pretty easy to read. There's less of that learning curve in a new codebase of getting used to the codebase's particular boilerplate rituals, and learning to pick out the code that's interesting. In a good lisp codebase all the ritual is hidden away in self-documenting macros. Of course this can get taken too far and then it becomes nightmarish to understand the codebase because so much stuff is in various complex macros that it's hard to tell where the shit goes down.
- turboponyy 4y ago
  What I don't understand is where the advantage of macros over functions lies in your example. In essence, what can't I do with the following? var position = new Position(); var newPositions = GetLegalMoves(position) .Select(move => position.Clone().Perform(move)); Moreover, besides this "sending remote commands example" portrayed in the article, when would I ever want to pass around unevaluated code? Perhaps I should pick up lisp.
  - agumonkey 4y ago
    Well there's an old debate about macros and functions, especially when you have closures, since closures can also delay assembly of bits of logic (half the work macros do, the rest being actual syntactic processing when people do that). You have to understand too that mainstream languages didn't have closure until very recently, so a lot of things look less obvious now.
  - phoe-krk 4y ago
deleted 4y ago
[deleted]
jimbokun 4y ago
> This is useful for compiler programmers But that's incredibly powerful. Now, stuff that would have to be implemented in the compiler to update the language, can now be written just as a "normal" program, that adds whole new features to your programming language. For example, the entire object system in Common Lisp was implemented as macros. Yes, most programming tasks don't require this kind of power. But it does mean programming in Lisp it's very very rare you are going to be stuck because your programming language doesn't implement some feature you need for the task at hand.
- lispm 4y ago
  > For example, the entire object system in Common Lisp was implemented as macros. It wasn't. The original Object System implementation of Common Lisp is has tree layers: at the bottom layer it is object-oriented (especially the Meta-Object Protocol), then there is a functional layer and on top there are macros for the convenience of the user.
TylerE 4y ago
And anyway modern languages have proven that you don't need S-Exps for powerful hygenic macros. E.g. sees Nim's take on it: https://nim-lang.org/docs/tut3.html https://nim-lang.org/docs/tut3.html
- Jtsummers 4y ago
  Dylan also did this, in the 90s, that's not a new idea. Who has been saying you need s-exprs for hygienic macros?
  - TylerE 4y ago
    All the lisp advocates, generally.
    - lispm 4y ago
      Some Lisp advocates may tell you that Lisp even does not have hygienic macros. Lisp dialects like Scheme have. Lisp usually has procedural macros, which transform source code (in the form of nested lists), not ASTs (like Nim). That Nim has 'powerful hygienic macros' is fine, many languages have powerful macros.
    - deleted 4y ago
      [deleted]
haroldl 4y ago
I've been using Java since version 1.1 and I've seen features added that required a new version of the language spec to go through committee and get implemented before we got to use them where you could just add these features yourself at the top of any Lisp file and then immediately start using them. For example consider the try-with-resources [0] syntax sugar. So instead of thinking "I never actually ran into a need for them", think of new language features that you have started using: those are the kinds of things you could've added yourself. Also look at any kind of code-gen tooling like parser generators or record specifications like Protocol Buffers as examples of what you could do within the language. [0] https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html https://docs.oracle.com/javase/tutorial/essential/exceptions...
j-james 4y ago
> This is useful for compiler programmers, or maybe also those writing source code analyzers/optimizers, but is that it? It is also useful for anyone wanting to implement language-level features as simple libraries. Someone else brought up Nim here: it's a great example of what can be done with metaprogramming (and in a non-Lisp language) as it intentionally sticks to a small-but-extendable-core design. There's macro-based libraries that implement the following, with all the elegance of a compiler feature: traits, interfaces, classes, typeclasses, contracts, Result types, HTML (and other) DSLs, syntax sugar for a variety of things (notably anonymous functions `=>` and Option types `?`), pattern matching (now in the compiler), method cascading, async/await, and more that I'm forgetting. https://github.com/ringabout/awesome-nim#language-features https://github.com/ringabout/awesome-nim#language-features
bitwize 4y ago
As part of delivering an e-commerce solution in Scheme, I wrote a module which allowed for SQL queries to be written in s-expression syntax. You could unquote Scheme variables or expressions into the queries and it would do the right thing wrt quoting, so no connection.prepareStatement("INSERT INTO CUSTOMERS (NAME, ADDRESS) VALUES (?, ?)", a, b) type madness. Wanky, you bet. But oh so, so convenient.
eternityforest 4y ago
I actively want people to NOT change constant to c. I want the language to be a predictable shared base, not something I have to relearn and customize for every project. I'm not a language designer. That's hard. That takes time and effort. And I don't do anything that can't be done in Python or Dart or whatever. Customizing the language is time not spent on the project, and batteries included stuff already has everything I need. I think lisp is good for people who "think inside their heads", as in, they think "I want to do this, oh, I could do it this way, then I'd need this resource, let's build it". If you think "Interactively" as in "I want to do this, what does Google tell me others are doing, oh yeah, this was made almost exactly the same use case, I'll start with this resource, now I'll adapt my design to fit it", you might not have any ideas for language tweaks to make in the first place. I basically never code and think "I wish I could do that in this language" aside from minor syntactic sugar and library features. New abstractions and ideas don't just pop up in my mind, what the common mainstream languages have is the entirity of what programs are made of, as far as I'm concerned.
- dusted 4y ago
  I don't entirely disagree that _changing_ the language is a bit of a no no, like changing the behavior of existing keywords and operators. However, any program that declares a variable or new function could be said to extend the language, since, if you declare some function, well, that function is now, at least in any proper language, as much part of the language of that program, as any builtin function is. Sure, if all you have is an empty .c file, you can say "this program is standard C", but as soon as you've declared a variable or funciton, your program becomes in a way a superset of C, it is all the standard C plus the functions, datastructures and variables that you've defined, and to extend that program, it is not enough to keep strictly to the standard language, you must also take into consideration the superset of functionality that is part of the program.. In this way, programming is much more about creating a language that speaks natively in the abstractions of the domain, and then using that language to solve specific tasks within the domain. And so it becomes that, you're always tweaking the language, it's just the degree to which you can tweak it that is different..

by reifyx 4y ago

zmgsabst 4y ago

I remember (faintly, from 20 years ago) using Lisp in game programming to create rules for mobs, from within the game.

agumonkey 4y ago

It's not tied to complex DSLs and compilation. I was frustrated by the lack of f-strings in emacs lisp so I hacked a macro, now I have embedded scoped string interpolation. I can write (f "<{class} {fields}>"). Having these freedom is really not a niche, it's mentally and pragmatically important.

momentoftop 4y ago

> This is useful for compiler programmers, or maybe also those writing source code analyzers/optimizers, but is that it? On occasion I have had to write DSLs for the user input, but in these cases the (non-programmer) users didn't want to write Lisp so I used something like Haskell's parsec to parse the data. If you're talking about Haskell, you should be talking about folk who write template Haskell, which is the macro system for GHC. There are plenty of Haskell programmers who know how to write Template Haskell, and there are plenty of Haskell programmers who don't. By contrast, I don't think there's a single Lisp programmer who can't write Lisp macros. That's homoiconicity. Once you learn Lisp, you automatically know how to write Lisp macros. Once you learn Haskell (or Ocaml or Rust), you don't automatically know how to write macros in that language (and the macro system may not even be portable across compilers).

mtlmtlmtlmtl 4y ago

In my experience, macros in lisp codebases are mainly abstracting away common boilerplate patterns. Common Lisp has WITH-OPEN-FILE to open a file and make sure it closes if the stack unwinds. This is a macro based on the UNWIND-PROTECT primitive, which ensures execution of a form if the stack unwinds. Many projects will have various with-foo macros to express this pattern for arbitray things that are useful in the context of that project(though not all with- macros need UNWIND-PROTECT). Another example is custom loops that happen over and over. Let's say I'm writing a chess engine. I regularly want to iterate over the legal moves in a position. So I might make a macro so I can write (do-moves (mv pos) ...) I find that because doing this is so simple, well written lisp codebases tend to be pretty easy to read. There's less of that learning curve in a new codebase of getting used to the codebase's particular boilerplate rituals, and learning to pick out the code that's interesting. In a good lisp codebase all the ritual is hidden away in self-documenting macros. Of course this can get taken too far and then it becomes nightmarish to understand the codebase because so much stuff is in various complex macros that it's hard to tell where the shit goes down.

turboponyy 4y ago
What I don't understand is where the advantage of macros over functions lies in your example. In essence, what can't I do with the following? var position = new Position(); var newPositions = GetLegalMoves(position) .Select(move => position.Clone().Perform(move)); Moreover, besides this "sending remote commands example" portrayed in the article, when would I ever want to pass around unevaluated code? Perhaps I should pick up lisp.
- agumonkey 4y ago
  Well there's an old debate about macros and functions, especially when you have closures, since closures can also delay assembly of bits of logic (half the work macros do, the rest being actual syntactic processing when people do that). You have to understand too that mainstream languages didn't have closure until very recently, so a lot of things look less obvious now.
- phoe-krk 4y ago

deleted 4y ago

[deleted]

jimbokun 4y ago

> This is useful for compiler programmers But that's incredibly powerful. Now, stuff that would have to be implemented in the compiler to update the language, can now be written just as a "normal" program, that adds whole new features to your programming language. For example, the entire object system in Common Lisp was implemented as macros. Yes, most programming tasks don't require this kind of power. But it does mean programming in Lisp it's very very rare you are going to be stuck because your programming language doesn't implement some feature you need for the task at hand.

lispm 4y ago
> For example, the entire object system in Common Lisp was implemented as macros. It wasn't. The original Object System implementation of Common Lisp is has tree layers: at the bottom layer it is object-oriented (especially the Meta-Object Protocol), then there is a functional layer and on top there are macros for the convenience of the user.

TylerE 4y ago

And anyway modern languages have proven that you don't need S-Exps for powerful hygenic macros. E.g. sees Nim's take on it: https://nim-lang.org/docs/tut3.html https://nim-lang.org/docs/tut3.html

Jtsummers 4y ago
Dylan also did this, in the 90s, that's not a new idea. Who has been saying you need s-exprs for hygienic macros?
- TylerE 4y ago
  All the lisp advocates, generally.
  - lispm 4y ago
    Some Lisp advocates may tell you that Lisp even does not have hygienic macros. Lisp dialects like Scheme have. Lisp usually has procedural macros, which transform source code (in the form of nested lists), not ASTs (like Nim). That Nim has 'powerful hygienic macros' is fine, many languages have powerful macros.
  - deleted 4y ago
    [deleted]

haroldl 4y ago

I've been using Java since version 1.1 and I've seen features added that required a new version of the language spec to go through committee and get implemented before we got to use them where you could just add these features yourself at the top of any Lisp file and then immediately start using them. For example consider the try-with-resources [0] syntax sugar. So instead of thinking "I never actually ran into a need for them", think of new language features that you have started using: those are the kinds of things you could've added yourself. Also look at any kind of code-gen tooling like parser generators or record specifications like Protocol Buffers as examples of what you could do within the language. [0] https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html https://docs.oracle.com/javase/tutorial/essential/exceptions...

j-james 4y ago

> This is useful for compiler programmers, or maybe also those writing source code analyzers/optimizers, but is that it? It is also useful for anyone wanting to implement language-level features as simple libraries. Someone else brought up Nim here: it's a great example of what can be done with metaprogramming (and in a non-Lisp language) as it intentionally sticks to a small-but-extendable-core design. There's macro-based libraries that implement the following, with all the elegance of a compiler feature: traits, interfaces, classes, typeclasses, contracts, Result types, HTML (and other) DSLs, syntax sugar for a variety of things (notably anonymous functions `=>` and Option types `?`), pattern matching (now in the compiler), method cascading, async/await, and more that I'm forgetting. https://github.com/ringabout/awesome-nim#language-features https://github.com/ringabout/awesome-nim#language-features

bitwize 4y ago

As part of delivering an e-commerce solution in Scheme, I wrote a module which allowed for SQL queries to be written in s-expression syntax. You could unquote Scheme variables or expressions into the queries and it would do the right thing wrt quoting, so no connection.prepareStatement("INSERT INTO CUSTOMERS (NAME, ADDRESS) VALUES (?, ?)", a, b) type madness. Wanky, you bet. But oh so, so convenient.

eternityforest 4y ago

I actively want people to NOT change constant to c. I want the language to be a predictable shared base, not something I have to relearn and customize for every project. I'm not a language designer. That's hard. That takes time and effort. And I don't do anything that can't be done in Python or Dart or whatever. Customizing the language is time not spent on the project, and batteries included stuff already has everything I need. I think lisp is good for people who "think inside their heads", as in, they think "I want to do this, oh, I could do it this way, then I'd need this resource, let's build it". If you think "Interactively" as in "I want to do this, what does Google tell me others are doing, oh yeah, this was made almost exactly the same use case, I'll start with this resource, now I'll adapt my design to fit it", you might not have any ideas for language tweaks to make in the first place. I basically never code and think "I wish I could do that in this language" aside from minor syntactic sugar and library features. New abstractions and ideas don't just pop up in my mind, what the common mainstream languages have is the entirity of what programs are made of, as far as I'm concerned.

dusted 4y ago
I don't entirely disagree that _changing_ the language is a bit of a no no, like changing the behavior of existing keywords and operators. However, any program that declares a variable or new function could be said to extend the language, since, if you declare some function, well, that function is now, at least in any proper language, as much part of the language of that program, as any builtin function is. Sure, if all you have is an empty .c file, you can say "this program is standard C", but as soon as you've declared a variable or funciton, your program becomes in a way a superset of C, it is all the standard C plus the functions, datastructures and variables that you've defined, and to extend that program, it is not enough to keep strictly to the standard language, you must also take into consideration the superset of functionality that is part of the program.. In this way, programming is much more about creating a language that speaks natively in the abstractions of the domain, and then using that language to solve specific tasks within the domain. And so it becomes that, you're always tweaking the language, it's just the degree to which you can tweak it that is different..