6 ms·

The Mighty Named Pipe

• amelius 12y ago
  If you like pipes, then you will love lazy evaluation. It is unfortunate, though, that Unix doesn't support that (operations can block when "writing" only, not when "nobody is reading").
  • ajuc 12y ago
    I don't know, I love pipes and I'm on the fence regarding strict vs lazy. BTW: when is nobody reading in pipes? There's always implicit &> stdout added. EDIT: oh, right, named pipes.
    • jquast 12y ago
      If you write to a named pipe, the call to write(2) will block until somebody opens it for reading and begins to read(2) it.
  • falcolas 12y ago
    If nobody is reading, you will eventually fill the pipe buffer (which is about 4k), and the writing will stop. It's a bigger queue than most of us would expect when compared to generator expressions, but it can and does create back pressure while making reads efficient.
    • noselasd 12y ago
      *about 4k 64k on linux these days.
  • valarauca1 12y ago
    Lazy evaluation with pipes would be problematic because alerts/echos would be ignored by default as they are necessarily part of the stdin/stdout chain. I.E.: This section of code let a = "some_file_name".as_string(); println!("Opening: {}", a); let path = std::path(a); let mut fd = std::io::open(path); would get optimized to let mut fd = std::io::open(std::path("some_file_name".as_string())); with strict lazy evaluation. The user feedback is removed, which is a big part of shell scripting.
    • amelius 12y ago
      I guess you found another issue with Unix. The user does not care in general how something is performed, just that it is performed correctly and with good performance. I guess an OS should be functional at its interface to the user, and only imperative deep down to keep things running efficiently. However, note that this hypothetical functional layer on top also would ensure efficiency, as it enables lazy evaluation. This type of efficiency could under certain circumstances be even more valuable than the bare-metal performance of system programming languages.
      • valarauca1 12y ago
        >The user does not care in general how something is performed, just that it is performed correctly and with good performance. This is the crux of the matter. With BASH scripting the user does care how a task is preformed as that task maybe system administration, involve sensitive system components, OR sensitive data. Lazy evaluation is great for binary/cpu level optimization. But passing system administration tasks though the same process is scary as you lose the 1:1 mapping you previously had.
        • amelius 12y ago
          Well, in any case, the problem can be resolved by adding a kernel-level api function that allows one to wait (block) until results are requested from the other end of the pipe.
          • valarauca1 12y ago
            Why? The opposite is already true and has the same effect. Each stage of the pipeline is executed when it has data to execute. So ultimately the main blocking event is IO (normally the first stage in a pipeline). Every other process is automatically marked as blocked, until its stdin is populated by the output of the former. Once its task is complete it re-checks stdin, and if nothing is present blocks itself. So the execution of each task is controlled by the process who's data that task needs to operate. In your system why would you want to block the previous step? This would just interfere with the previous+1 step, and you'd have to populate that message further up the chain. This seems needlessly complicated. As you have to add extra IPC.
• Malarkey73 12y ago
  Vince Buffalo is author of the best book on bioinformatics: Bioinformatics Data Skills (O'Reilly). It's worth a read for learning unix/bash style data science of any flavour. Or even if you think you know unix/bash and data there are new and unexpected snippets every few pages that surprise you.
• mhax 12y ago
  I've used *nix for ~15 years and never used a named pipe or process substitution before. Great to know about!
  • a3n 12y ago
    Named pipes have been rare for me, but simple process substitution is every day. Very often I do something like this in quick succession. Command line editing makes this trivial. $ find . -name "*blarg*.cpp" # Some output that looks like what I'm looking for. # Run the same find again in a process, and grep for something. $ grep -i "blooey" $(find . -name "*blarg*.cpp") # Yep, those are the files I'm looking for, so dig in. # Note the additional -l in grep, and the nested processes. $ vim $(grep -il "blooey" $(find . -name "*blarg*.cpp"))
    • icebraining 12y ago
      That's actually command substitution, not process substitution :)
      • hollerith 12y ago
        And command substitution has nothing to do with pipes, named or unnamed.
      • a3n 12y ago
        Thanks for the correction. The unix is large and I'm so very small.
    • jbnicolai 12y ago
      You could just use a pipe here though, which would also make it more easy to read. e.g.: $ find . -name '*blarg*.cpp' | grep -li blooey | vi -
      • icebraining 12y ago
        Your version searches for blooey in the filenames, not in the files themselves.
• frankerz 12y ago
  How does the > process substitution differ from simply piping the output with | ? For example (from Wikipedia) tee >(wc -l >&2) < bigfile | gzip > bigfile.gz vs tee < bigfile | wc -l | gzip > bigfile.gz
  • joosters 12y ago
    It allows multiple, parallel pipes to each individual command, where the | allows just one.
  • aidos 12y ago
    In the tee case the substitution is actually going somewhere different than standard out (that's what tee does). so: cmd1 | tee out.txt | cmd2 So tee is splitting the stream into two outputs, one that carries on out stdout (into cmd2) and the other one that is redirected into out.txt. With process substitution you can do extra stuff on the way out, I guess (I've never seen it used for output before). It looks like in the example given they're writing wc stuff to stderr while zipping the content (over stdout). Nice to see that example, I hadn't even thought about the usefulness of process substitution for outputting like this!
  • unhammer 12y ago
    Say that you have a program that splits its output into two files, each given by command line arguments. A normal run would be <input.txt munge-data-and-split -o1 out1.txt -o2 out2.txt but since the output is huge and your disk is old and dying, you want to run xz on it before saving it to disk, so use >(): <input.txt munge-data-and-split -o1 >(xz - > out1.txt) -o2 >(xz - > out2.txt) If you want to do several things in there, I recommend defining a function for clarity: pp () { sort -k2,3 -t$'\t' | xz - ; } <input.txt munge-data-and-split -o1 >(pp > out1.txt) -o2 >(pp > out2.txt)
  • cnvogel 12y ago
    When you connect to processes in a pipe such as ... a | b you connect stdout (fd #1) of a to stdin (fd #0) of b. Technically, the shell process will create a pipe, which is two filedescriptors connected back to back. It then will fork two times (create two copies of itself) where it replaces standard output (filedescriptor 1) of the first copy by one end of the pipe and replaces standard input (filedescriptor 0) of the second copy by the other end of the pipe. Then the first copy will replace itself (exec) by a, the second copy will replace itself (exec) by b. Everything that a writes to stdout will appear on stdin of b. But nothing prevents the shell from replacing any other filedescriptor by pipes. And when you create a subprocess by writing "<(c)" in your commandline, it's just one additional fork for the shell, and one additional filedescriptor pair to be created. One side, as in the simple case, will replace stdin (fd #0) of "c"... and because the input side of this pipe doesn't have a predefined output of "a" (stdout is already taken by "|b") the shell will somehow have to tell "a" what filedescriptor the pipe uses. Under Linux one can refer to opened filedescriptors as "/dev/fd/<FDNUM>" (symlink to /proc/self/fd/<FDNUM> which itself is a symlik to /proc/<PID>/fd/<FDNUM>), so that's what's replaced as a "name" to refer to the substituted process on "a"'s command line: Try this: $ echo $$ 12345 # <--- PID of your shell $ tee >( sort ) >( sort ) >( sort ) otherfile | sort and in a second terminal $ pstree 12345 # <--- PID of your shell zsh,301 ├─sort,3600 # <-- this one reads from the other end of the shell's fd #14 ├─sort,3601 # <-- this one reads from the other end of the shell's fd #15 ├─sort,3602 # <-- this one reads from the other end of the shell's fd #15 ├─sort,3604 # <-- this one reads from stdout of tee └─tee,3603 /proc/self/fd/14 /proc/self/fd/15 /proc/self/fd/16 otherfile If your system doesn't support the convenient /proc/self/fd/<NUM> shortcut, the shell might decide not to create a pipe, but rather create temporary fifos in /tmp and use those to connect the filedescriptors. http://man7.org/linux/man-pages/man2/pipe.2.html http://man7.org/linux/man-pages/man2/pipe.2.html http://linux.die.net/man/2/dup http://linux.die.net/man/2/dup You can watch the syscalls as they are made: $ strace -fe fork,pipe,close,dup,dup2,execve bash -c 'tee <(sort) <(sort)'
• aidos 12y ago
  Nice article. Really easy to follow introduction. I only discovered process substitution a few months ago but it's already become a frequently used tool in my kit. One thing that I find a little annoying about unix commands sometimes is how hard it can be to google for them. '<()', nope, "command as file argument to other command unix," nope. The first couple of times I tried to use it, I knew it existed but struggled to find any documentation. "Damnit, I know it's something like that, how does it work again?..." Unless you know to look for "Process Substitution" it can be hard to find information on these things. And that's once you even know these things exist.... Anyone know a good resource I should be using when I find myself in a situation like that?
  • icebraining 12y ago
    man pages! $ man bash /<\( Drops you right into the Process Substitution section.
    • pygy_ 12y ago
      /<\( For those wondering, man, which uses less as a pager, has vi-like key bindings. "/<\(" starts a regex-based search for "<(" (you must escape the open paren). This is the origin of the regex literal syntax in most programming languages that have them. It was first introduced by Ken Thompson in the "ed" text editor.
    • aidos 12y ago
      Ahhhh..haaa...ha....DOH! I've never even thought of looking at the manpage for bash before. Thanks, you've just made my life better.
      • omaranto 12y ago
        That's weird, isn't it? You wanted to know how to use a feature of bash and didn't check the manual?
        • vertex-four 12y ago
• unhammer 12y ago
  Once you discover <() it's hard not to (ab)use it everywhere :-) # avoid temporary files when some program needs two inputs: join -e0 -o0,1.1,2.1 -a1 -a2 -j2 -t$'\t' \ <(sort -k2,2 -t$'\t' freq/forms.${lang}) \ <(sort -k2,2 -t$'\t' freq/lms.${lang}) # gawk doesn't care if it's given a regular file or the output fd of some process: gawk -v dict=<(munge_dict) -f compound_translate.awk <in.txt # prepend a header: cat <(echo -e "${word}\t% ${lang}\tsum" | tr [:lower:] [:upper:]) \ <(coverage ${lang})
  • repsilat 12y ago
    > # gawk doesn't care if it's given a regular file or the output fd of some process: Something wonderful I found out the other day: Bash executes scripts as it parses them, so you can do all kinds of awful things. For starters, bash <(yes echo hello) will have bash execute an infinite script that looks like echo hello echo hello echo hello ... without trying to load the whole thing first. After that, you can move onto having a script append to itself and whatever other dreadful things you can think of.
    • unhammer 12y ago
      That's actually one of the things that I really dislike with bash, that it doesn't read the whole script before executing it. I've been bitten by it before, when I write some long-running script, then e.g. write a comment at the top of it as it's running, and then when bash looks for the next command, it's shifted a bit and I get (at best) a syntax error and have to re-run :-(
      • lsaferite 12y ago
        Ewww. Thats... nasty and dangerous. Very dangerous.
      • LukeShu 12y ago
        There are several ways to get Bash to read the whole thing before executing. My preferred method is to write a main() function, and call main "$@" at the very end of the script. Another trick, useful for shorter scripts, is to just wrap the body of the script in {}, which causes the script to be a giant compound command that is parsed before any of it is executed; instead of a list of commands that is executed as read.
• Dewie 12y ago
  Pipes are very cool and useful, but it's hard for me to understand this common worship of something like that. Yes, it's useful and elegant, but is it really the best thing since Jesus Christ?
  • AndrewWright 12y ago
    Maybe it's not the best thing since Jesus, but it's worth celebrating its birthday http://blog.fugue.it/2013-10-07-pipeday.html http://blog.fugue.it/2013-10-07-pipeday.html
  • Dewie 12y ago
    Wow. I guess that's what I get for not being totally enamoured of Unix.
    • JustSomeNobody 12y ago
      No, that's not why you were down voted. You were down voted because you were condescending to the people who enjoy working with *nix.
      • Dewie 12y ago
        It really is a question that I've been having for a long time. I didn't just say that to piss people off. I guess that's the risk you run of coming across when you try to insert yourself in a conversation where the other participants have already agreed on a set of shared opinions - this is great - and you try to question that common assumption/opinion. I have honestly been questioning my own understanding of pipe, since I've failed to see the significance before; first I thought it was just `a | b` as in "first do a, then b". So then it just seemed like a notational way of composing programs. Then I thought, uh, ok say what? Composing things is the oldest trick in the conceptual book. But then I read more about it and saw that it had this underlying wiring of standard input and output and forking processes that made me realize that I had underestimated it. So, given that, I was wondering if there is even more that I've been missing. I have for that matter read about named pipes before and tried it out a bit. It's definitely a cool concept.
• hitlin37 12y ago
  i heard somewhere that go follows unix pipe link interfaces.
• AndrewSB 12y ago
  Does anyone have a working link to Gary Bernhardt's The Unix Chainsaw, as mentioned in the article?
  • agumonkey 12y ago
    That's the kind of video I might have downloaded. At least I hope so. Gonna check my backups. update 1 : found it, time to upload.
    • AndrewSB 12y ago
      Thank you kind sir. Waiting for a link
      • agumonkey 12y ago
        http://goo.gl/X5jo83 http://goo.gl/X5jo83
  • sikhnerd 12y ago
    I found a high quality copy I had downloaded: http://sikhnerd.com/downloaded_vids/02-gary-bernhardt.mp4 http://sikhnerd.com/downloaded_vids/02-gary-bernhardt.mp4 (703M)
  • dtmooreiv 12y ago
    https://www.youtube.com/watch?v=sCZJblyT_XM https://www.youtube.com/watch?v=sCZJblyT_XM
• larsf 12y ago
  Pipes are probably the original instantiation of dataflow processing (dating back to the 1960s). I gave a tech talk on some of the frameworks: https://www.youtube.com/watch?v=3oaelUXh7sE https://www.youtube.com/watch?v=3oaelUXh7sE And my company creates a cool dataflow platform - https://composableanalytics.com https://composableanalytics.com
  • noselasd 12y ago
    http://doc.cat-v.org/unix/pipes/ http://doc.cat-v.org/unix/pipes/ . And there's a bit more about how pipes came to be in unix here: http://cm.bell-labs.com/who/dmr/hist.html http://cm.bell-labs.com/who/dmr/hist.html
• baschism 12y ago
  AFAIK process substitution is a bash-ism (not part of POSIX spec for /bin/sh). I recently had to go with the slightly less wieldy named pipes in a dash environment and put the pipe setup, command execution and teardown in a script.
• jamesrom 12y ago
  Is this guy a bioinformatician? I think he's a bioinformatician. Can't be sure if he is a bioinformatician because he never really mentions that he is a bioinformatician.
  • bsenftner 12y ago
    The tone made me stop reading. It reads like a child's thoughts in a comic book.
  • pmags 12y ago
    Seems entirely appropriate given his blog post, and others like it on his site as well as the book he wrote, are clearly aimed at people interested in learning bioinformatics.
• leni536 12y ago
  moreutils [1] has some really cool programs for pipe handling. pee: tee standard input to pipes sponge: soak up standard input and write to a file ts: timestamp standard input vipe: insert a text editor into a pipe [1] https://joeyh.name/code/moreutils/ https://joeyh.name/code/moreutils/
• chuckcode 12y ago
  Anybody know of a way to increase the buffer size of pipes? I've experienced cases where piping a really fast program to a slow one caused them both to go slower as the OS pauses first program writing when pipe buffer is full. This seemed to ruin the caching for the first program and caused them both to be slower even though normally pipes are faster as you're not touching disk.
  • jquast 12y ago
    Both mbuffer and pv by default contain fairly large in-memory buffers for pipe data, and accept parameters for particularly large buffers. http://www.maier-komor.de/mbuffer.html http://www.maier-komor.de/mbuffer.html http://www.ivarch.com/programs/pv.shtml http://www.ivarch.com/programs/pv.shtml
    • chuckcode 12y ago
      Thanks - hoping that there was a built in solution but a buffer program makes sense
• dbbolton 12y ago
  In zsh, =(cmd) will create a temporary file, <(cmd) will create a named pipe, and $(cmd) creates a subshell. There are also fancy options that use MULTIOS. For example: paste <(cut -f1 file1) <(cut -f3 file2) | tee >(process1) >(process2) >/dev/null can be re-written as: paste <(cut -f1 file1) <(cut -f3 file2) > >(process1) > >(process2) http://zsh.sourceforge.net/Doc/Release/Expansion.html#Process-Substitution http://zsh.sourceforge.net/Doc/Release/Expansion.html#Proces... http://zsh.sourceforge.net/Doc/Release/Redirection.html#Redirection http://zsh.sourceforge.net/Doc/Release/Redirection.html#Redi...
• anateus 12y ago
  In fish shell the canonical example is this: diff (sort a.txt|psub) (sort b.txt|psub) The psub command performs the process substitution.
  • frankerz 12y ago
    It seems like fish shell's ">" process substitution equivalence is not working as well as bash's though https://github.com/fish-shell/fish-shell/issues/1786 https://github.com/fish-shell/fish-shell/issues/1786