16 ms·

> When was the last time your Unix workstation was as useful as a Macintosh? Some of that discussion has not aged well :)

by bagrow 8y ago

deleted 8y ago
[deleted]
pytester 8y ago
The core critique - that everything is stringly typed - still holds pretty well though. >The receiving and sending processes must use a stream of bytes. Any object more complex than a byte cannot be sent until the object is first transmuted into a string of bytes that the receiving end knows how to reassemble. This means that you can’t send an object and the code for the class definition necessary to implement the object. You can’t send pointers into another process’s address space. You can’t send file handles or tcp connections or permissions to access particular files or resources.
- majewsky 8y ago
  > You can’t send pointers into another process’s address space. Thank goodness.
- jimktrains2 8y ago
  To be fair, the same critisim could be used for a socket? I think the issue is that some people want pipes to be something magical that connects their software, not a dumb connection between them.
  - AnIdiotOnTheNet 8y ago
    Yes. The compromise of just using an untyped byte stream in a single linear pipeline was a fair tradeoff in the 70s, but it is nearly 2020 and we can do better.
    - laumars 8y ago
      We have done better. The shell I'm writing is typed and I know I'm not the only person to do this (eg Powershell). The issue here is really more with POSIX compatibility but if you're willing to step away from that then you might find an alternative that better suits your needs. Thankfully switching shells is as painless as switching text editors.
      - da_chicken 8y ago
        > Thankfully switching shells is as painless as switching text editors. So, somewhere between, "That wasn't as bad as I feared," and, "Sweet Jesus, what fresh new hell have I found myself in"?
        laumars 8y ago
        haha yes. I was thinking more about launching the shell but you're absolutely right that learning the syntax of a new shell is often non-trivial.
  - __MatrixMan__ 8y ago
    I don't want all my pipes to be magical all the time, but occasionally I do want to write a utility that is "pipeline aware" in some sense. For example, I'd like to pipe mysql to jq and have one utility or the other realize that a conversion to json is needed in the middle for it work. Im working on a library for this kind of intra-pipeline negitiation. It's all drawing-board stuff right now but I coobbled together a proof of concept: https://unix.stackexchange.com/a/495338/146169 https://unix.stackexchange.com/a/495338/146169 Do you think this is a reasonable way to achieve the magic that some users want in their pipelines? Or are ancient Unix gods going to smite me for tampering with the functional consistency of tools by making their behavior different in different contexts?
    - jshen 8y ago
      I wonder if something like HTTP’s content negotiation is a good model for this.
      - __MatrixMan__ 8y ago
        That sounds reasonable, I'll look into it--thanks. I was imagining an algorithm where each pipeline-aware utility can derive port numbers to use to talk/listen to its neighbors. I may be able to use http content negotiation wholesale in that context.
        cryptonector 8y ago
        I was thinking something similar, buried in a library that everyone could link. It seems... awfully awkward to build, much less portably. This reminds me of how busted Linux is for not having a SO_PEERCRED. You can actually get that information by walking /proc/net/tcp or using AF_NETLINK sockets and inet_diag, but there is a race condition such that this isn't 100% reliable. SO_PEERCRED would [have to] be.
        laumars 8y ago
        I've been trying to solve the exact same problem with my shell too. It's pipes are typed and all the builtin commands can than automatically decode those data types via shared libraries. So commands don't need to worry about how to decode and re-encode the data. This means that JSON, YAML, TOML, CSV, Apache log files, S-Expressions and even tabulated data from `ps` (for example) can all be transparently handled the same way and converted from one to another without the tools ever needing to know how to marshal nor unmarshal that data. For example: you could take a JSON array that's not been formatted with cartridge returns and still grep through it item by item as if it was a multi-line string. However the problem I face is how do you pass that data type information over a pipeline from tools that exist outside of my shell? It's all well and good having builtins that all follow that convention but what if someone else wants to write a tool? My first thought was to use network sockets, but then you break piping over SSH, eg: local-command | ssh user@host "| remote-command" My next thought was maybe this data should be in-lined - a bit like how ANSI escape sequences are in-lined and the terminals don't render them as printable characters. Maybe something like the following as a prefix to STDIN? <null>$SHELL<null> But then you have the problem of tainting your data if any tools are sent that prefix in error. I also wondered if setting environmental variables might work but that also wouldn't be reliable for SSH connections. So as you can see, I'm yet to think up a robust way of achieving this goal. However in the case of builtin tools and shell scripts, I've got it working for the most part. A few bugs here and there but it's not a small project I've taken on. If you fancy comparing notes on this further, I'm happy to oblige. I'm still hopeful we can find a suitable workaround to the problems described above.
- laumars 8y ago
  I'm not going to argue that UNIX got everything right because I don't believe that to be the case either but I don't agree with those specific points: > This means that you can’t send an object and the code for the class definition necessary to implement the object. To some degree you can and I do just this with my own shell I've written. You just have to ensure that both ends of the pipe understands what is being sent (eg is it JSON, text, binary data, etc)? Even with typed terminals (such as Powershell), you still need both ends of the pipe to understand what to expect to some extent. Having this whole thing happen automatically with a class definition is a little optimistic though. Not least of all because not every tool would be suited for every data format (eg a text processor wouldn't be able to do much with a GIF even if it has a class definition). > You can’t send pointers into another process’s address space. Good job too. That seems a very easy path for exploit. Thankfully these days it's less of an issue though because copying memory is comparatively quick and cheap compared to when that handbook was written. > You can’t send file handles Actually that's exactly how piping works as technically the standard streams are just files. So you could launch a program with STDIN being a different file from the previous processes STDOUT. > or tcp connections You can if you pass it as a UNIX socket (where you define a network connection as a file). > or permissions to access particular files or resources. This is a little ambiguous. For example you can pass strings that are credentials. However you cannot alter the running state of another program via it's pipeline (aside what files it has access to). To be honest I prefer the `sudo` type approach but I don't know how much of that is because it's better and how much of that is because it's what I am used to.
  - zwp 8y ago
    >> You can’t send file handles > Actually that's exactly how piping works Also SCM_RIGHTS, which exists exactly for this purpose (see cmsg(3), unix(7) or https://blog.cloudflare.com/know-your-scm_rights/ https://blog.cloudflare.com/know-your-scm_rights/ for a gentler introduction and application). That's been around since BSD 4.3, which predates the Hater's Handbook 1ed by 4 years or so.
    - laumars 8y ago
      Yeah I had mentioned UNIX domain sockets. However your post does add a lot of good detail on them which I had left off.
- rumcajz 8y ago
  Look at the alternatives though. Would you really want to use something like Spring in shell scripting?
  - pytester 8y ago
    No. I typically use python as a drop in replacement for shell scripts > ~10 lines of code.
OliverJones 8y ago
MacOs is layered on a UNIX-like OS. You can use pipes in your command windows.
- amdavidson 8y ago
  The Unix Haters Handbook was published in 1994, when System 7 was decidedly not unix-like.
- DavidWoof 8y ago
  This comment makes me feel really old. MacOs wasn't always layered on unix, and the unix-haters' handbook predates the switch to the unix-based MacOs X.
  - chrisfinazzo 8y ago
    Of course not, but the switch to BSD fixed a bunch of the underpinnings in the OS and was a sane base to work off of. Not to put too fine a point on it, but they found religion. Unlike Classic (and early versions of Windows for that matter), there was more to be gained by ceding some control to the broader community. Microsoft has gotten better (PowerShell - adapting UNIX tools to Windows, and later WSL, where they went all in) Still, for Apple it meant they had to serve two masters for a while - old school Classic enthusiasts and UNIX nerds. Reading the back catalog of John Siracusa's (one of my personal nerd heroes) old macOS reviews gives you some sense of just how weird this transition was.
- deleted 8y ago
  [deleted]
- deleted 8y ago
  [deleted]
- JdeBP 8y ago
  You can also drop the "-like". (-: * https://unix.stackexchange.com/questions/1489/ https://unix.stackexchange.com/questions/1489/
jcelerier 8y ago
... has it ? most people using macs never ever open a terminal.

> When was the last time your Unix workstation was as useful as a Macintosh? Some of that discussion has not aged well :)

by bagrow 8y ago

> When was the last time your Unix workstation was as useful as a Macintosh? Some of that discussion has not aged well :)

deleted 8y ago

[deleted]

pytester 8y ago

The core critique - that everything is stringly typed - still holds pretty well though. >The receiving and sending processes must use a stream of bytes. Any object more complex than a byte cannot be sent until the object is first transmuted into a string of bytes that the receiving end knows how to reassemble. This means that you can’t send an object and the code for the class definition necessary to implement the object. You can’t send pointers into another process’s address space. You can’t send file handles or tcp connections or permissions to access particular files or resources.

majewsky 8y ago
> You can’t send pointers into another process’s address space. Thank goodness.
jimktrains2 8y ago
To be fair, the same critisim could be used for a socket? I think the issue is that some people want pipes to be something magical that connects their software, not a dumb connection between them.
- AnIdiotOnTheNet 8y ago
  Yes. The compromise of just using an untyped byte stream in a single linear pipeline was a fair tradeoff in the 70s, but it is nearly 2020 and we can do better.
  - laumars 8y ago
    We have done better. The shell I'm writing is typed and I know I'm not the only person to do this (eg Powershell). The issue here is really more with POSIX compatibility but if you're willing to step away from that then you might find an alternative that better suits your needs. Thankfully switching shells is as painless as switching text editors.
    - da_chicken 8y ago
      > Thankfully switching shells is as painless as switching text editors. So, somewhere between, "That wasn't as bad as I feared," and, "Sweet Jesus, what fresh new hell have I found myself in"?
      - laumars 8y ago
        haha yes. I was thinking more about launching the shell but you're absolutely right that learning the syntax of a new shell is often non-trivial.
- __MatrixMan__ 8y ago
  I don't want all my pipes to be magical all the time, but occasionally I do want to write a utility that is "pipeline aware" in some sense. For example, I'd like to pipe mysql to jq and have one utility or the other realize that a conversion to json is needed in the middle for it work. Im working on a library for this kind of intra-pipeline negitiation. It's all drawing-board stuff right now but I coobbled together a proof of concept: https://unix.stackexchange.com/a/495338/146169 https://unix.stackexchange.com/a/495338/146169 Do you think this is a reasonable way to achieve the magic that some users want in their pipelines? Or are ancient Unix gods going to smite me for tampering with the functional consistency of tools by making their behavior different in different contexts?
  - jshen 8y ago
    I wonder if something like HTTP’s content negotiation is a good model for this.
    - __MatrixMan__ 8y ago
      That sounds reasonable, I'll look into it--thanks. I was imagining an algorithm where each pipeline-aware utility can derive port numbers to use to talk/listen to its neighbors. I may be able to use http content negotiation wholesale in that context.
      - cryptonector 8y ago
        I was thinking something similar, buried in a library that everyone could link. It seems... awfully awkward to build, much less portably. This reminds me of how busted Linux is for not having a SO_PEERCRED. You can actually get that information by walking /proc/net/tcp or using AF_NETLINK sockets and inet_diag, but there is a race condition such that this isn't 100% reliable. SO_PEERCRED would [have to] be.
      - laumars 8y ago
        I've been trying to solve the exact same problem with my shell too. It's pipes are typed and all the builtin commands can than automatically decode those data types via shared libraries. So commands don't need to worry about how to decode and re-encode the data. This means that JSON, YAML, TOML, CSV, Apache log files, S-Expressions and even tabulated data from `ps` (for example) can all be transparently handled the same way and converted from one to another without the tools ever needing to know how to marshal nor unmarshal that data. For example: you could take a JSON array that's not been formatted with cartridge returns and still grep through it item by item as if it was a multi-line string. However the problem I face is how do you pass that data type information over a pipeline from tools that exist outside of my shell? It's all well and good having builtins that all follow that convention but what if someone else wants to write a tool? My first thought was to use network sockets, but then you break piping over SSH, eg: local-command | ssh user@host "| remote-command" My next thought was maybe this data should be in-lined - a bit like how ANSI escape sequences are in-lined and the terminals don't render them as printable characters. Maybe something like the following as a prefix to STDIN? <null>$SHELL<null> But then you have the problem of tainting your data if any tools are sent that prefix in error. I also wondered if setting environmental variables might work but that also wouldn't be reliable for SSH connections. So as you can see, I'm yet to think up a robust way of achieving this goal. However in the case of builtin tools and shell scripts, I've got it working for the most part. A few bugs here and there but it's not a small project I've taken on. If you fancy comparing notes on this further, I'm happy to oblige. I'm still hopeful we can find a suitable workaround to the problems described above.
laumars 8y ago
I'm not going to argue that UNIX got everything right because I don't believe that to be the case either but I don't agree with those specific points: > This means that you can’t send an object and the code for the class definition necessary to implement the object. To some degree you can and I do just this with my own shell I've written. You just have to ensure that both ends of the pipe understands what is being sent (eg is it JSON, text, binary data, etc)? Even with typed terminals (such as Powershell), you still need both ends of the pipe to understand what to expect to some extent. Having this whole thing happen automatically with a class definition is a little optimistic though. Not least of all because not every tool would be suited for every data format (eg a text processor wouldn't be able to do much with a GIF even if it has a class definition). > You can’t send pointers into another process’s address space. Good job too. That seems a very easy path for exploit. Thankfully these days it's less of an issue though because copying memory is comparatively quick and cheap compared to when that handbook was written. > You can’t send file handles Actually that's exactly how piping works as technically the standard streams are just files. So you could launch a program with STDIN being a different file from the previous processes STDOUT. > or tcp connections You can if you pass it as a UNIX socket (where you define a network connection as a file). > or permissions to access particular files or resources. This is a little ambiguous. For example you can pass strings that are credentials. However you cannot alter the running state of another program via it's pipeline (aside what files it has access to). To be honest I prefer the `sudo` type approach but I don't know how much of that is because it's better and how much of that is because it's what I am used to.
- zwp 8y ago
  >> You can’t send file handles > Actually that's exactly how piping works Also SCM_RIGHTS, which exists exactly for this purpose (see cmsg(3), unix(7) or https://blog.cloudflare.com/know-your-scm_rights/ https://blog.cloudflare.com/know-your-scm_rights/ for a gentler introduction and application). That's been around since BSD 4.3, which predates the Hater's Handbook 1ed by 4 years or so.
  - laumars 8y ago
    Yeah I had mentioned UNIX domain sockets. However your post does add a lot of good detail on them which I had left off.
rumcajz 8y ago
Look at the alternatives though. Would you really want to use something like Spring in shell scripting?
- pytester 8y ago
  No. I typically use python as a drop in replacement for shell scripts > ~10 lines of code.

OliverJones 8y ago

MacOs is layered on a UNIX-like OS. You can use pipes in your command windows.

amdavidson 8y ago
The Unix Haters Handbook was published in 1994, when System 7 was decidedly not unix-like.
DavidWoof 8y ago
This comment makes me feel really old. MacOs wasn't always layered on unix, and the unix-haters' handbook predates the switch to the unix-based MacOs X.
- chrisfinazzo 8y ago
  Of course not, but the switch to BSD fixed a bunch of the underpinnings in the OS and was a sane base to work off of. Not to put too fine a point on it, but they found religion. Unlike Classic (and early versions of Windows for that matter), there was more to be gained by ceding some control to the broader community. Microsoft has gotten better (PowerShell - adapting UNIX tools to Windows, and later WSL, where they went all in) Still, for Apple it meant they had to serve two masters for a while - old school Classic enthusiasts and UNIX nerds. Reading the back catalog of John Siracusa's (one of my personal nerd heroes) old macOS reviews gives you some sense of just how weird this transition was.
deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
JdeBP 8y ago
You can also drop the "-like". (-: * https://unix.stackexchange.com/questions/1489/ https://unix.stackexchange.com/questions/1489/

jcelerier 8y ago

... has it ? most people using macs never ever open a terminal.