11 ms·
Linux Pipes Are Slow
- jheriko 2y agojust never use pipes. they are some weird archaism that need to die :P the only time ive used them is external constraints. they are just not useful.
- w0m 2y agoYou can replace a 10k line python or shell script with a single creative line of pipes/xargs/etc on the cli. It's incredibly valuable on the day to day.
- henearkr 2y agoPipes are extremely useful. But I guess it just depends on your use case. I do a lot of scripting. If you dislike their (relative) slowness, it's open source, you can participate in making them faster. And I'm sure that after this HN post we'll see some patches and merge requests.
- noloblo 2y ago+1 yes pipes are what shell scripting quite useful and allow for easy composition of the different unix shell utilities
- hnlmorg 2y agoThe very thing that makes pipes useful is what also makes them slow. I don't think there is much we can do to fix that without breaking POSIX compatibility entirely. Personally I think there's much worse ugliness in POSIX than pipes. For example, I've just spent the last couple of days debugging a number of bugs in a shell's job control code (`fg`, `bg`, `jobs`, etc). But despite its warts, I'm still grateful we have something like POSIX to build against.
- effie 2y agoWhat possible bugs can there be in those? They are quite simple to use and work as expected.
- khafra 2y agoThey work as expected on Redhat and Debian. "POSIX" leaves open a lot of possibility for less-well-tested systems. They could be writing shellscripts on Minix or HelenOS.
- hnlmorg 2y agoI'm talking about shell implementation not shell usage. To implement job control, there are several signals you need to be aware of: - SIGSTSP (what the TTY sends if it receives ^Z) - SIGSTOP (what a shell sends to a process to suspend it) - SIGCONT (what a shell sends to a process to resume it) - SIGCHLD (what the shell needs to listen for to see there is a change in state for a child process -- this is also sometimes referred to as SIGCLD) - SIGTIN (received if a process read from stdin) - SIGTOU (received if a process cannot write to stdout nor set its modes) Some of these signals are received by the shell, some are by the process. Some are sent from the shell and others from the kernel. SIGCHLD isn't just raised for when a child process goes into suspend, it can be raised for a few different changes of state. So if you receive SIGCHLD you then need to inspect your children (of course you don't know what child has triggered SIGCHLD because signals don't contain metadata) to see if any of them have changed their state in any way. Which is "fun".... And all of this only works if you manage to fork your children with special flags to set their PGID (not PID, another meta ID which represents what process group they belong to), and send magic syscalls to keep passing ownership of the TTY (if you don't tell the kernel which process owns the TTY, ie is in the foreground, then either your child process and/or your shell will crash due to permission issues). None of this is 100% portable (see footnote [1]) and all of this also depends on well behaving applications not catching signals themselves and doing something funky with them. The bug I've got is that Helix editor is one of those applications doing something non-standard with SIGTSTP and assuming anything that breaks as a result is a parent process which doesn't support job control. Except my shell does support job control and still crashes as a result of Helix's non-standard implementation. In fairness to Helix, my shell does also implement job control in a non-standard way because I wanted to add some wrappers around signals and TTYs to make the terminal experience a little more comfortable than it is with POSIX-compliant shells like Bash. But because job control (and signals and TTYs in general) are so archaic, the result is that there are always going to be edge case bugs with applications (like Helix) that have implemented things a little differently themselves too. So they're definitely not easy to use and can break in unexpected ways if even just one application doesn't implement things in expected ways. [1] By the way, this is all ignoring subtle problems that different implementations of PTYs (eg terminal emulators, terminal multiplexors, etc) and different POSIX kernels can introduce too. And those can be a nightmare to track down and debug!
- akira2501 2y ago> just never use pipes. vmslice doesn't work with every type of file descriptor. eschewing some technology entirely because it seems archaic or because it makes writing "the fastest X software" seem harder is just sloppy engineering. > they are just not useful. Then you have not written enough software yet to discover how they are useful.
- gpderetta 2y agoMost importantly the fast fizbuz toy vmsplices into /dev/null. Nothing ever touches those pages on the consumer side and they can be refused immediately. If you actually want a functional program using vmsplice, with a real consumer, things get hairy very quickly.
- hagbard_c 2y agoThat's like telling a builder never to use nails but turn to adhesives instead. He will look at his hammer and his nails as well as a stack of 2x4s, grin and in no time slap together a box into which he will stuff you with a bottle of glue with the advice to now go and play while the grown-ups take care of business. Sure, you could build that box with glue and clamps and ample time, sure it would look neater and weigh less than the version that's currently holding you imprisoned and if done right it will even be stronger but it takes more time and effort as well as those glue clamps and other specialised tools to create perfect matching surfaces while the builder just wielded that hammer and those nails and now is building yet another utilitarian piece of work with the same hammer and nails. Sometimes all you need is a hammer and some nails. Or pipes.
- duped 2y agoI agree with this but with a much more nuanced take: avoid pipes if either reader or writer expects to do async i/o and you don't own both the reader and writer. In fact if you ever set O_NONBLOCK on a pipe you need to be damn sure both the reader and writer expect non-blocking i/o because you'll get heisenbugs under heavy i/o when either the reader/writer outpace each other and one expects blocking i/o. When's the last time you checked the error code of `printf` and put it in a retry loop?
- teo_zero 2y agoGenuine question: why does printf need a retry loop when using pipes?
- duped 2y agoIt doesn't that's why no one does it. But for pipes what it means is that if whoever is reading or writing the pipe expects non blocking semantics, the other end needs to agree. And if they don't you'll eventually get an error because the reader or writer outpaced the other, and almost no program handles errors for stdin or stdout.
- nitwit005 2y agoJust about every form of IPC is "slow". You have decided to pay a performance cost for safety.
- marcosdumay 2y agoYou shouldn't have to pay that much. Pipes give you almost nothing, so they should cost almost nothing. Specifically, there aren't many reasons for your fastest IPC to be slower than a long function call.
- nitwit005 2y agoIf you don't think pipes offer much, don't use them. Saying "long function call" doesn't mean much since a function can take infinitely long.
- marcosdumay 2y agoA long distance function call, that invalidates everything on your cache.
- saagarjha 2y ago…which is quite expensive.
- marcosdumay 2y agoYes, it is. But it's much cheaper than interacting by pipe. Linux is optimizing sockets with a similar goal. And it's quite far on that direction. But there's still some margin to gain.
- brigade 2y agoPipes don’t exist for safety, they exist as an optimization to pass data between existing programs.
- djaouen 2y agoSo is Python, but I'm still gonna use it lol
- RevEng 2y agoI didn't quite grasp why the original splice has to be so slow. They pointed out what made it slower than vmsplice - in particular allocating buffers and using scalar instructions - but why is this necessary? Why couldn't splice just be reimplemented as vmsplice? I'm sure there is a good reason, but I've missed it.
- Izkata 2y ago> Why couldn't splice just be reimplemented as vmsplice? A possible answer that's currently just below your comment: https://news.ycombinator.com/item?id=41351870 https://news.ycombinator.com/item?id=41351870 > vmslice doesn't work with every type of file descriptor.
- koverstreet 2y agoOne of my sideprojects is intended to address this: https://lwn.net/Articles/976836/ https://lwn.net/Articles/976836/ The idea is a syscall for getting a ringbuffer for any supported file descriptor, including pipes - and for pipes, if both ends support using the ringbuffer they'll map the same ringbuffer: zero copy IO, potentially without calling into the kernel at all. Would love to find collaborators for this one :)
- wakawaka28 2y agoBuffering is there for a reason and this approach will lead to weird failure modes and fragility in scripts. The core issue is that any stream producer might go slower than any given consumer. Even a momentary hiccup will totally mess up the pipe unless there is adequate buffering, and the amount needed is system-dependent.
- Spivak 2y agoWhat makes this any different than other buffer implementations that have a max size? Buffer fills, writes block. What failure mode are you worried about that can't occur with pipes which are also bounded?
- foota 2y agoMaybe I misunderstand, but if the ring buffer is full isn't it ok for the sender to just block?
- mort96 2y agoYeah, and if the ring buffer is empty it's okay for the receiver to just block... exactly as happens today with pipes
- hackernudes 2y agoI think the OP's proposal has buffering. It is different from a pipe - instead of using read/write to copy data from/to a kernel buffer, it gives user space a mapped buffer object and they need to take care to use it properly (using atomic operations on the head/tail and such). If you own the code for the reader and writer, it's like using shared memory for a buffer. The proposal is about standardizing an interface.
- 0xbadcafebee 2y agoCalling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.
- AkBKukU 2y agoI have a project that uses a proprietary SDK for decoding raw video. I output the decoded data as pure RGBA in a way FFMpeg can read through a pipe to re-encode the video to a standard codec. FFMpeg can't include the Non-Free SDK in their source, and it would be wildly impracticable to store the pure RGBA in a file. So pipes are the only way to do it, there are valid reasons to use high throughput pipes.
- whartung 2y agoWhat about domain sockets? It's clumsier, to be sure, but if performance is your goal, the socket should be faster.
- AkBKukU 2y agoIt looks like FFmpeg does support reading from sockets natively[1], I didn't know that. That might be a better solution in this case, I'll have to look into some C code for writing my output to a socket to try that some time. [1] https://ffmpeg.org/ffmpeg-protocols.html#unix https://ffmpeg.org/ffmpeg-protocols.html#unix
- ptx 2y agoWhy should sockets be faster?
- uncanneyvalley 2y agoSockets remap pages without moving any data while pipes have to copy the data between fds.
- 2y ago
- JoshTriplett 2y agoThis is a side note to the main point being made, but on modern CPUs, "rep movsb" is just as fast as the fastest vectorized version, because the CPU knows to accelerate it. The name of the kernel function "copy_user_enhanced_fast_string" hints at this: the CPU features are ERMS ("Enhanced Repeat Move String", which makes "rep movsb" faster for anything above a certain length threshold) and FSRM ("Fast Short Repeat Move String", which makes "rep movsb" faster for shorter moves too).
- jeffbee 2y agoAlso worth noting that Linux has changed the way it uses ERMS and FSRM in x86 copy multiple times since kernel 6.1 used in the article. As a data-dote, my machine that has FSRM and ERMS — surprisingly, the latter is not implied by the former — hits 17GB/s using plain old pipes and a 32KiB buffer on Linux 6.8
- Lockal 2y agoThis is not the full truth, "rep movsb" is fast until another threshold, after which either normal or non-temporal store is faster. All thresholds are described in https://codebrowser.dev/glibc/glibc/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S.html#19 https://codebrowser.dev/glibc/glibc/sysdeps/x86_64/multiarch... And they are not final, i. e. Noah Goldstein still updates them every year.
- jeffbee 2y agoWhich is these is "faster" depends greatly on whether you have the very rare memcpy-only workload, or if your program actually does something useful. Many people believe, often with good evidence, that the most important thing is for memcpy to occupy as few instruction cache lines as is practical, instead of being something that branches all over kilobytes of machine code. For comparison, see the x86 implementations in LLVM libc. https://github.com/llvm/llvm-project/blob/main/libc/src/string/memory_utils/x86_64/inline_memcpy.h#L158 https://github.com/llvm/llvm-project/blob/main/libc/src/stri...
- adrian_b 2y agoIt depends on the CPU. There is no good reason for "rep movsb" to be slower at any big enough data size. On a Zen 3 CPU, "rep movsb" becomes faster than or the same as anything else above a length slightly greater than 2 kB. However there is a range of multi-megabyte lengths, which correspond roughly with sizes below the L3 cache but exceeding the L2 cache, where for some weird reason "rep movsb" becomes slower than SIMD non-temporal stores. At lengths exceeding the L3 size, "rep movsb" becomes again the fastest copy method. The Intel CPUs have different behaviors.
- nyanpasu64 2y agoHow do you gather profiling information for kernel function calls from a user program?
- qsantos 2y agoI'll write an article on the flamegraphs specifically, but to get the data, just follow Julia's article! https://jvns.ca/blog/2017/03/19/getting-started-with-ftrace/ https://jvns.ca/blog/2017/03/19/getting-started-with-ftrace/
- ismaildonmez 2y agoCould you clarify how are you testing the speed of the first example where you are not writing anything to stdout? Thanks.
- qsantos 2y agoFor the first Rust program, where I just write to memory, I just use the time utility when running the program from zsh. Then, I divide the number of bytes written by the number of seconds elapsed. That's why it's not an infinite loop ;)
- ismaildonmez 2y agoThanks!
- cowsaymoo 2y agoWhat is the library used to profile the program?
- tzury 2y agopv https://linux.die.net/man/1/pv https://linux.die.net/man/1/pv it is in the pipe command `... | pv > /dev/null`
- throw12390 2y ago`pv --discard` is faster by 8% (on my system). % pv </dev/zero >/dev/null 54.0GiB/s % pv </dev/zero --discard 58.7GiB/s
- IWeldMelons 2y agoWhich is suspiciously close to the speed of DDR4.
- qsantos 2y agoI am again getting the hug of death of Hacker News. The situation is better than the last time thanks to caching WordPress pages, but loading the page can still take a few seconds, so bear with me!
- Borg3 2y agoHaha. When I read the title I smiled. Linux pipes slow? Moook.. Now try Cygwin pipes. Thats what I call slow! Anyway, nice article, its good to know whats going on under the hood.
- MaxBarraclough 2y agoI'd assumed Cygwin pipes are just Windows pipes, is that not the case?
- tyingq 2y agoNot a comprehensive list of problems, and not current but a good illustrative post of the kind of issues that people have run into in this post: https://cygwin.com/pipermail/cygwin-patches/2016q1/008301.html https://cygwin.com/pipermail/cygwin-patches/2016q1/008301.ht...
- Borg3 2y agoIts not that easy. Yeah, they are, but there is a lot of POSIX like glue inside so they work correctly with select() and other alarms. Code is very complicated. But still, kudos for Cygwin Developers for creating Cygwin :) Great work, even tho it have some issues.
- stabbles 2y agoA bold claim for a blog that takes about 20 seconds to load.
- yas_hmaheshwari 2y agoThis post has gone to the top of hacker news, so I think we should give him some slack Looks like an amazing article, and so much to learn on what happens under the hood
- ben-schaaf 2y agoHN generates ~20k page views over the course of a day with a peak of 2k/h: https://harrisonbroadbent.com/blog/hacker-news-traffic-spike-anatomy/ https://harrisonbroadbent.com/blog/hacker-news-traffic-spike.... At ~1MB per page load - not sure how accurate this is, I don't think it fully loaded - this static blogpost requires 0.55MB/s to meet demand. An original raspberry pi B (10mpbs ethernet) on the average french mobile internet connection (8mbps) provides double that. I don't mean this as a slight to anyone, I just want to point out the HN "hug of death" can be trivially handled by a single cheap VPS without even breaking a sweat.
- qsantos 2y agoTotally agree, my server should definitely be able to handle the load. But this is a WordPress install, which is definitely doing too much work for what it is when just serving the pages. I plan to improve on this!
- wvh 2y agoI believe that when it's a .fr, they call it nonchalance...
- deleted 2y ago[deleted]
- goodpoint 2y agoExcellent article even if, to be honest, the title is clickbait.
- chmaynard 2y agoAgreed. Titles that don't use quantifiers are almost always misleading at best.
- sixthDot 2y ago> I do not know why the JMP is not just a RET, however. The jump seems generated by the expansion of the `ASM_CLAC` macro, which is supposed to change the EFLAGS register ([1], [2]). However in this case the expansion looks like it does nothing (maybe because of the target ?). I 'd be interested to know more about that. Call to the wild. [1]: https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/smap.h#L17 https://github.com/torvalds/linux/blob/master/arch/x86/inclu... [2]: https://stackoverflow.com/a/60579385 https://stackoverflow.com/a/60579385
- fatcunt 2y ago> I do not know why the JMP is not just a RET, however. This is caused by the CONFIG_RETHUNK option. In the disassembly from objdump you are seeing the result of RET being replaced with JMP __x86_return_thunk. https://github.com/torvalds/linux/blob/v6.1/arch/x86/include/asm/linkage.h#L23 https://github.com/torvalds/linux/blob/v6.1/arch/x86/include... https://github.com/torvalds/linux/blob/v6.1/arch/x86/lib/retpoline.S#L121 https://github.com/torvalds/linux/blob/v6.1/arch/x86/lib/ret... > The NOP instructions at the beginning and at the end of the function allow ftrace to insert tracing instructions when needed. These are from the ASM_CLAC and ASM_STAC macros, which make space for the CLAC and STAC instructions (both of them three bytes in length, same as the number of NOPs) to be filled in at runtime if X86_FEATURE_SMAP is detected. https://github.com/torvalds/linux/blob/v6.1/arch/x86/include/asm/smap.h#L17-L26 https://github.com/torvalds/linux/blob/v6.1/arch/x86/include... https://github.com/torvalds/linux/blob/v6.1/arch/x86/include/asm/cpufeatures.h#L264 https://github.com/torvalds/linux/blob/v6.1/arch/x86/include... https://github.com/torvalds/linux/blob/v6.1/arch/x86/kernel/alternative.c#L265 https://github.com/torvalds/linux/blob/v6.1/arch/x86/kernel/...
- qsantos 2y agoThanks a lot for the information! I was not quite sure what to look for in this case. I have added in note in the article.
- ndesaulniers 2y agoThere are perhaps only a handful of kernel developers that: 1. would know the above 2. would choose such an obnoxious throwaway handle
- michaelcampbell 2y agoI believe there are a lot more of your 2nd point than you might think.
- opello 2y agoAnd certainly those interested in the performance implications of the retpoline mitigation for the various speculative execution information leaks.
- arendtio 2y agoI know pipes primarily from shell scripts. Are they being used in other contexts as extensively, too? Like C or Rust programs?
- guenthert 2y agoMost shells are C programs, so it's clearly possible to use pipes there (and consequently any language with a C FFI, including Rust) and it's done. It's cumbersome though. That's why there are so called glue languages, including shells. Compiler and similar tools might establish pipes and will do so directly, rather than via a glue language. Perhaps a grep through github could tell how 'extensive' this truly is. For the data transfer rate it doesn't matter how (using which language) the pipe is established; C and Rust and the like will have a (small) edge up in the start-up time (latency) though.
- rwmj 2y agoBe interesting to see a version using io_uring, which I think would let you pre-share buffers with the kernel avoiding some copies, and avoid syscall overhead (though the latter seems negligible here).
- qsantos 2y agoThat sounds like a good idea!
- rwmj 2y agoI'm not claiming it'll be faster! Additionally io_uring has its own set of challenges, such as whether it's better to allocate one ring per core or one ring per application (shared by some or all cores). Pre-sharing buffers has trade-offs too, particularly in application complexity [alignment, you have to be careful not to reuse a buffer before it is consumed] versus the efficiency of zero copy.
- donaldihunter 2y agoSomething I didn't see mentioned in the article about AVX512, aside from the xsave/xrstor overhead, is that AVX512 is power hungry and causes CPU frequency scaling. See [1], [2] for details and as an example of how nuanced it can get. [1] https://www.intel.com/content/dam/www/central-libraries/us/en/documents/cryptography-processing-with-3rd-gen-intel-xeon-scalable-processors-19-may-2021.pdf https://www.intel.com/content/dam/www/central-libraries/us/e... [2] https://www.intel.com/content/www/us/en/developer/articles/technical/accelerating-x265-with-intel-advanced-vector-extensions-512-intel-avx-512.html https://www.intel.com/content/www/us/en/developer/articles/t...
- Narishma 2y agoThat is only the case in specific Intel CPU models.
- up2isomorphism 2y agoSomeone tasted a bread thinking it is not sweet enough, which is fine. But calling the bread bland is funny because it does not mean to taste sweet.
- mparnisari 2y agoI get PR_CONNECT_RESET_ERROR when trying to open the page
- qsantos 2y agoMy server struggles a bit with the load on the WordPress site. You should be fine just reloading. I will make sure to improve things for the next time!
- jvanderbot 2y ago> Although SSE2 is always available on x86-64, I also disabled the cpuid bit for SSE2 and SSE to see if it could nudge glibc into using scalar registers to copy data. I immediately got a kernel panic. Ah, well. I think you need to recompile your compiler, or disable those explicitly via link / cc flags. Compilers are fairly hard to get to coax / dissuade SIMD instructions, IMHO.
- faizshah 2y agoThis is a really cool post and that is a massive amount of throughput. In my experience in data engineering, it’s very unlikely you can exceed 500mb/s throughput of your business logic as most libraries you’re using are not optimized to that degree (SIMD etc.). That being said I think it’s a good technique to try out. I’m trying to think of other applications this could be useful for. Maybe video workflows?
- jeremyscanvic 2y agoGreat post! I didn't know about vmsplice(2). I'm glad to see a former ENSL student here as well!
- qsantos 2y agoHey!
- yencabulator 2y agoFUSE can be a bit trickier than a single queue of data chunks. Reads from /dev/fuse actually pick the right message to read based on priorities, and there's cases where the message queue is meddled with to e.g. cancel requests before they're even sent to userspace. If you naively switch it to eagerly putting messages into a userspace-visible ringbuffer, you might significantly change behavior in cases like interrupting slow operations. Imagine having to fulfill a ringbuf worth of requests to a misbehaving backend taking 5sec/op, just to see the cancellations at the very end.
- Marthcass 2y ago[flagged]