10 ms·
Io_uring, kTLS and Rust for zero syscall HTTPS server
- sandeep-nambiar 1y agoThis is really cool. I've been thinking about something similar for a long time and I'm glad someone has finally done it. GG! I can recommend writing even the BPF side of things with rust using Aya[1]. [1] - https://github.com/aya-rs/aya https://github.com/aya-rs/aya
- boredatoms 1y agoWhats the goto instead of strace, if you wanted to see what was going on?
- abrookewood 1y agoI think you have to use eBPF-based tools
- fuy 1y agoperf and look at stack traces (or off-cpu events for waits/locks). also, ebpf
- Unirely01 1y ago[dead]
- bmcahren 1y agoThis was a good read and great work. Can't wait to see the performance tests. Your write up connected some early knowledge from when I was 11 where I was trying to set up a database/backend and was finding lots of cgi-bin online. I realize now those were spinning up new processes with each request https://en.wikipedia.org/wiki/Common_Gateway_Interface https://en.wikipedia.org/wiki/Common_Gateway_Interface I remember when sendfile became available for my large gaming forum with dozens of TB of demo downloads. That alone was huge for concurrency. I thought I had swore off this type of engineering but between this, the Netflix case of extra 40ms and the GTA 5 70% load time reduction maybe there is a lot more impactful work to be done. https://netflixtechblog.com/life-of-a-netflix-partner-engineer-the-case-of-extra-40-ms-b4c2dd278513 https://netflixtechblog.com/life-of-a-netflix-partner-engine... https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times-by-70/ https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...
- kev009 1y agoIt wasn't just CGI, every HTTP session was commonly a forked copy of the entire server in the CERN and Apache lineage! Apache gradually had better answers, but their API with common addons made it a bit difficult to transition so webservers like nginx took off which are built closer to the architecture in the article with event driven I/O from the beginning.
- avar 1y agoevery HTTP session was commonly a forked copy of the entire server in the CERN and Apache lineage! And there's nothing wrong with that for application workers. On *nix systems fork() is very fast, you can fork "the entire server" and the kernel will only COW your memory. As nginx etc. showed you can get better raw file serving performance with other models, but it's still a legitimate technique for application logic where business logic will drown out any process overhead.
- josephg 1y agoSo long as you have something like nginx in front of your server. Otherwise your whole site can be taken down by a slowloris attack over a 33.6k modem.
- tsimionescu 1y agoForking for anything other than calling exec is still a horrible idea (with special exceptions like shells). Forking is a very unsafe operation (you can easily share locks and files with the child process unless both your code and every library you use is very careful - for example, it's easy to get into malloc deadlocks with forked processes), and its performance depends a lot on how you actually use it.
- zbentley 1y agoI think it's not quite that bad (and I know that this has been litigated to death all over the programmer internet). If you are forking from a language/ecosystem that is extremely thread-friendly, (e.g. Go, Java, Erlang) fork is more risky. This is because such runtimes mean a high likelihood of there being threads doing fork-unsafe things at the moment of fork(). If you are forking from a language/ecosystem that is thread-unfriendly, fork is less risky. That isn't to say "it's always safe/low risk to run fork() in e.g. Python, Ruby, Perl", but in those contexts it's easier to prove/test invariants like "there are no threads running/so-and-so lock is not held at the point in my program when I fork", at which point the risks of fork(2) are much reduced. To be clear, "reduced" is not the same as "gone"! You still have to reason about explicitly taken locks in the forking thread, file descriptors, signal handlers, and unexpected memory growth due to CoW/GC interactions. But that's a lot more tractable than the Java situation of "it's tricky to predict how many Java threads are active when I want to fork, and even trickier to know if there are any JNI/FFI-library-created raw pthreads running, the GC might be threaded, and checking for each of those things is still racy with my call to fork(2)". You still have to make sure that that fork-safety invariants are true. But the effort to do that is extremely different depending on language platform. Rust/C/C++ don't cleanly fit into either of those two (already mushy/subjective) categorizations, though. Whether forking is feasible in a given Rust/C/C++ codebase depends on what the code does and requires a tricky set of judgement calls and at-a-distance knowledge going forward to make sure that the codebase doesn't become fork-unsafe in harmful ways.
- 6r17 1y agoI really want to see the benchmarks on this ; tried it like 4 days ago and then built a standard epoll implementation ; I could not compete against nginx using uring but that's not the easiest task for an arrogant night so I really hope you get some deserved sweet numbers ; mine were a sad deception but I did not do most of your implementation - rather simply tried to "batch" calls. Wish you the best of luck and much fun
- ValtteriL 1y agoExcellent read. I'd like to see DPDK style full kernel bypass next
- spaintech 1y agoNot sure if you are aware of this, but LUNA does this already. https://www.usenix.org/system/files/atc23-zhu-lingjun.pdf https://www.usenix.org/system/files/atc23-zhu-lingjun.pdf
- Seattle3503 1y ago> For example when submitting a write operation, the memory location of those bytes must not be deallocated or overwritten. > The io-uring crate doesn’t help much with this. The API doesn’t allow the borrow checker to protect you at compile time, and I don’t see it doing any runtime checks either. I've seen comments like this before[1], and I get the impression that building a a safe async Rust library around io_uring is actually quite difficult. Which is sort of a bummer. IIRC Alice from the tokio team also suggested there hasn't been much interest in pushing through these difficulties more recently, as the current performance is "good enough". [1] https://boats.gitlab.io/blog/post/io-uring/ https://boats.gitlab.io/blog/post/io-uring/
- JoshTriplett 1y agoI think the right way to build a safe interface around io_uring would be to use ring-owned buffers, ask the ring for a buffer when you want one, and give the buffer back to the ring when initiating a write.
- pingiun 1y agoThis is something that Amos Wenger (fasterthanlime) has worked on: https://github.com/bearcove/loona/blob/main/crates/buffet/README.md https://github.com/bearcove/loona/blob/main/crates/buffet/RE...
- Tuna-Fish 1y agoThis works perfectly well, and allows using the type system to handle safety. But it also really limits how you handle memory, and makes it impossible to do things like filling out parts of existing objects, so a lot of people are reluctant to take the plunge.
- johncolanduoni 1y agoThat’s annoying for people writing bespoke low-level networking code, but for a high-level HTTP library it’s a rounding error in the overall complexity on display. I think the bigger barrier for Tokio is that the interplay between having an epoll instance and a io_uring instance on the same pool is problematic and can erase performance gains. If done greenfield you could implement the “normal” APIs with ‘IORING_OP_POLL_ADD’, but not all of the exposed ‘mio’ surface area can work this way - only the oneshot API.
- up2isomorphism 1y ago[flagged]
- kev009 1y agoFWIW Rust advice is maybe 15% of the bottom of the article, most of the decisions apply equally to C and the article is a fairly sensible survey of APIs.
- Imustaskforhelp 1y agoSuch a good read. I am patient to wait for the benchmarks so take your time ,but I honestly love how the author doesn't care about benchmarks right now and wanted to clean the code first. Its kinda impressive that there are people who have such line of thinking in this world where benchmarks gets maxxed and whole project's sole existence is to satisfy benchmarks. Really a breath of fresh air and honestly I admire the author so much for this. It was such a good read, loved it a lot thank you. Didn't know ktls existed or Io_uring could be used in such a way.
- mgaunard 1y ago"zero syscall" > In order to avoid busy looping, both the kernel and the web server will only busy-loop checking the queue for a little bit (configurable, but think milliseconds), and if there’s nothing new, the web server will do a syscall to “go to sleep” until something gets added to the queue.
- KolmogorovComp 1y agoIt’s good to read an article until the end > This means that a busy web server can serve all of its queries without even once (after setup is done) needing to do a syscall. As long as queues keep getting added to, strace will show nothing.
- nly 1y agoLike all polling I/O models (that don't spin) it also means you have to wait milliseconds in the worst case to start servicing a request. That's a long time. For comparison a read/write over a TCP socket on loopback between two process is a few microseconds using BSD sockets API.
- klabb3 1y ago> Like all polling I/O models (that don't spin) it also means you have to wait milliseconds in the worst case to start servicing a request. That's a long time. No? What they're saying is the busy loop will spin until an event occurs, for at most x ms. And if it does park the thread (the only syscall required), it can be immediately woken up on the first event too. Only if multiple events occurred since the last call would you receive them together. This normally happens only under high load, when event processing takes enough time to have a buildup of new events in the background. Increased latency is the intended outcome on high loads. To be fair, it was a while ago I read the io-uring paper. But I distinctly recall the mix of poll and park behavior, plus configurable wait conditions. Please correct me if I'm wrong (someone here certainly knows).
- thomashabets2 1y agoUnder load it's zero syscall (barring any rare allocations inside rustls for the handshake. I can't guarantee that it never does). Without load the overhead of calling (effectively) sleep() is, while technically true, not relevant. But sure, you can tweak the busyloop timers and burn 100% CPU on kernel and user side indefinitely if you want to avoid that sleep-when-idle syscall. It's just… not a good idea.
- LAC-Tech 1y agoI think rusts glacial compile times prevent it from being a useful platform for web apps. Yes it's a nice language, and very performant, but it's horrible devex to have to wait seconds for your server to recompile after a change.
- maeln 1y ago> but it's horrible devex to have to wait seconds for your server to recompile after a change. What a time to be alived that seconds to recompile is consider horrible devex.
- craftkiller 1y agoAt my first job out of college it took 30 minutes to recompile and launch the server. Now the kids complain about 10 seconds. It's just impossible for me to take their complaints seriously. 10 seconds isn't even enough time for a mental context-switch, its just slightly more time than "instant". Back in the day, something like this wasn't an exaggeration: https://xkcd.com/303/ https://xkcd.com/303/
- LAC-Tech 1y agoI can remember instant reloads of application servers on a job 10+ years ago grandpa. This isn't new.
- craftkiller 1y agoYeah, we had hot reloading of code too but hot-reloading for instant "reloads" was needed back then. Nowadays, you can do a full relaunch of the server in 10 seconds so hot reloads no longer matter.
- hu3 1y agoIt was already horrible devex 40 years ago when turbo pascal could compile millions of lines almost instantly with a processor that was slower than my current watch processor.
- phrotoma 1y agoAnybody know what the state of kTLS is? I asked one of the Cilium devs about it a while ago'cause I'd seen Thomas Graf excitedly talking about it and he told me that kernel support in many distros was lacking so they aren't ready to enable it by default.
- drewg123 1y agoThat's a shame. How hard is it to enable? Do you need a custom kernel, or can you enable it at runtime? On FreeBSD, its been in the kernel / openssl since 13, and has been one runtime toggle (sysctl kern.ipc.tls.enable=1) away from being enabled. And its enabled by default in the upcoming FreeBSD-15. We (at Netflix) have run all of our tls encrypted streaming over kTLS for most of a decade.
- tempaccount420 1y agokTLS just sounds like a bad idea all around.
- evrennetwork 1y ago[dead]
- bullen 1y agoSo far everything after epoll that I have compared with falls short. So to reimplement my foundation (with all the bugs) will not be worth it. I will however compare Javas NIO (epoll) with the new Virtual Threads IO (without pinning). http://github.com/tinspin/rupy http://github.com/tinspin/rupy
- ozgrakkurt 1y agoThis wiki page might be useful for anyone that is looking into this https://github.com/axboe/liburing/wiki/io_uring-and-networking-in-2023 https://github.com/axboe/liburing/wiki/io_uring-and-networki... Also there is napi support in uring which uses polled io on sockets instead of interrupt based io from what I understand. You can see examples using it in liburing github
- api 1y agoThis is impressive but it’s also an amazing amount of complexity and difficult programming to work around the fact that syscalls are so slow. It seems like there’s these fundamental things in OSes that we just can’t improve, or I suppose can’t without breaking too much backward compatibility, so we are forced to do this.
- j_seigh 1y agoI don't think it has to be. Conceptually it's just a couple of queues. There's a software equivalent of the Peter Principle where software or an API becomes increasingly complex to the point where no one understands it. They then attempt to fix that by adding more functionality (complexity).
- selfmodruntime 1y agoI do wonder if this would make for an excellent exfil implant since it doesn‘t register syscalls.
- zbentley 1y agoIt would, hence why major cloud providers currently disable io_uring in many of their compute environments.
- selfmodruntime 1y agoInteresting!
- klaussilveira 1y agoFor anyone wanting to learn more about how to create a small server with io_uring: https://unixism.net/2020/04/io-uring-by-example-article-series/ https://unixism.net/2020/04/io-uring-by-example-article-seri...
- npalli 1y agoSo, current status on async Rust - you need to understand: Futures, Pin, Waker, async runtimes, Send/Sync bounds, async trait objects, etc. C++20, coroutines. Go, goroutines. Java21+, virtual threads
- thomashabets2 1y agoRust: Well yes. Rust does force you to understand the things, or it won't compile. It does have drawbacks. Go: goroutines are not async. And you can't understand goroutines without understanding channels. And channels are weirdly implemented in Go, where the semantics of edge cases, while well defined, are like rolling a D20 die if you try to reason from first principles. Go doesn't force you to understand things. I agree with that. It has pros and cons. I see what you mean but "cheap threads" is not the same thing as async. More like "current status of massive concurrency". Except that's not right either. tarweb, the subject of the blog post in question, is single threaded and uses io_uring as an event loop. (the idea being to spin up one thread per CPU core, to use full capacity) So it's current status of… what exactly? Cheap threads have a benefit over an async loop. The main one being that they're easier to reason about. It also has drawbacks. E.g. each thread may be light weight, but it does need a stack.
- ori_b 1y ago> Go: goroutines are not async Sure they are. The abstraction they provide is a synchronous API, but it's accomplished using an async runtime.
- thomashabets2 1y agoI'm trying to understand the context in which the parent commenter uses the term, since it can mean multiple things. They said "async" and then enumerated some wildly different things. Like, do you need async runtimes to do epoll async in Rust? No. Ok, so that excludes many definitions. Do you need coroutines in C++ to do aio for reading and writing? No. So like I said, what do they mean by "async"? The blog post refers to a web server that does "async" in Rust without any async runtime, and without the `async` keyword. In other words, that parent commenter is what's called "not even wrong".
- hnaccountme 1y agoHey, Is there a working HTTP server with all these features? I am working on something like this for work. But with plain old C
- alde 1y agoUnfortunately io_uring is disabled by default on most cloud workload orchestrators, like CloudRun, GKE, EKS and even local Docker. Hope this will change soon, but until then it will remain very niche.
- nicce 1y agoBack to self-hosting!
- superb_dev 1y agoWhy do they disable io_uring?
- arianvanp 1y agoSandboxing like gvisor is based on syscalls and iouring makes your code syscallless
- alpb 1y agoSecurity reasons. https://news.ycombinator.com/item?id=44632240 https://news.ycombinator.com/item?id=44632240 There are also other edge cases around cgroups accounting that renders some isolation/throttling mechanisms not fully effective.
- butterisgood 1y agoWhere do people get the idea that one thread per core is correct on a system that deals with time slices? In my experience “oversubscribing” threads to cores (more threads than cores) provides a wall-clock time benefit. I think one thread per core would work better without preemptive scheduling. But then we aren’t talking about Unix.
- gorset 1y agoIsolating a core and then pinning a single thread is the way to go to get both low latency and high throughput, sacrificing efficiency. This works fine on Linux, and common approach for trading systems where it’s fine to oversubscribe a bunch of cores for this type of stuff. The cores are mostly busy spinning and doing nothing, so it’s very inefficient in terms of actual work, but great for latency and throughput when you need it.
- butterisgood 1y agoI just wish people who give this advice for 1 thread per core would "expand their reasoning" or "show the work". It's not blanket good advice for all things.
- lossolo 1y agoCheck out Scylla and its underlying framework Seastar. They expand their reasoning and show the work.
- thinkharderdev 1y agoIt is definitely not good advice for all things. For workloads that are either end of the CPU/IO spectrum (e.g. almost all waiting on IO or almost all doing CPU work) it can be a huge win as you can get very good L1 cache utilization, are not context-switching and don't need to handle thread synchronization in your code because not state is shared between threads. For workloads that are a mix of IO and non-trivial CPU work, it can still work but is much, much harder to get right.
- jandrewrogers 1y ago
- WJW 1y agoPretty cool! Adding kTLS is definitely an improvement. I made an actually zero-syscall per request server a few years ago (and blogged about it at https://wjwh.eu/posts/2021-10-01-no-syscall-server-iouring.html https://wjwh.eu/posts/2021-10-01-no-syscall-server-iouring.h...) but as TFA notes it comes at a heavy cost of constantly busy-looping. io_uring is very cool tech though and has been progressing at an impressive pace the last few years.