6 ms·

Yes, io_uring is significantly faster than epoll (I think I had like 20% faster req/s with io_uring.) The catch is that its kernel opt-in and disabled just abou

Yes, io_uring is significantly faster than epoll (I think I had like 20% faster req/s with io_uring.) The catch is that its kernel opt-in and disabled just about everywhere for security reasons. I think that it has direct memory sharing between the kernel and user-land which is kind of yikes. There's been multiple exploits that hit io_uring in recent times. It's because of this that even engineering projects that try to reach the highest performance possible (like Go) don't really bake io_uring in as a sane default. Though if you want to take the risk you can always run it yourself for your favourite language. It is faster but the cost is possible exploits.

happyPersonR 3mo ago
Any kind of poll mode networking: Rdma, dpdk, io_uring it’s really kind of up to the user to do the memory isolation In io_urings case tho, you can’t do much because the rings are in the kernel. I’m hopeful though that with Llm things will get better. But it’s just hard problem to solve . Very difficult to do in the kernel itself, and folks don’t really even understand tuning for it.
- kshri24 3mo ago
  The ring buffers are in shared memory not kernel private. The ring buffers (submission and completion) are shared between kernel space and user space. User publishes requests via submission queue entries (updates tail of buffer while kernel reads head of the buffer), kernel shifts the submission queue buffer on its end and returns a completion queue event by publishing to completion buffer. User pulls from this buffer (specifically the head, kernel updates tail of buffer) in user space.
omcnoe 3mo ago
For a project like Go, wouldn't it be an option to do one-time iouring feature detection in the runtime startup? Exploits are an issue for the entire OS, not the program choosing to use iouring, yeah?
csdreamer7 3mo ago
RHEL 9 and 10 now fully support io_uring by default. It is very recent, but this covers a lot of corporate Linux installs. Gemini 'said' Ubuntu and SuSE support it as well, but did not provide any links to prove it. https://access.redhat.com/solutions/4723221 https://access.redhat.com/solutions/4723221 Go should reconsider support. They should have a 'go' at it.
- insanitybit 3mo ago
  It's still seccomp'd off in most environments because io-uring is still a seccomp bypass that doesn't play well with kernel security systems (audit subsystem), even if it weren't also like the #1 or #2 exploit vector for privesc.
  - Asmod4n 3mo ago
    That’s solved as of last week, you can use cBPF now to disable functionality.
    - insanitybit 3mo ago
      How solved? AFAIK it's not meaningfully shipped but happy to hear otherwise.
      - Asmod4n 3mo ago
        you can now disable opertions with cBPF, like you would be able to with seccomp for normal syscalls.
        cyphar 3mo ago
        Their point is that that functionality is not available on older kernels (such as those in RHEL 9 and 10) and so most sandboxes will continue to block it outright for a while, though eventually one would expect Red Hat to backport it. (We haven't even added support for the new cBPF io_uring stuff to low-level container runtimes like runc yet, though I did review the patchset on LKML earlier this year and planned to get working on it when I have time. But as it requires spec changes, expect it to take 6-12 months at best...)
Cloudef 3mo ago
Quite depends, I had times when my posix emulation of io_uring (with poll, not epoll) was faster than io_uring. For large zero-copy buffers, io_uring is king however. Also io_uring is useful even for non asynchronous IO as it can implement chain of operations as single atomic operation (mkdir + open it for example). For something like networking, if you are maximizing packets per second, you'll hit kernel limits[1] very quickly and instead have to start leveraging features like GSO/GRO or completely bypass the network stack. 1: https://github.com/axboe/liburing/discussions/1346 https://github.com/axboe/liburing/discussions/1346
- lukeh 3mo ago
  Also it’s nice for things like SPI which have no user space non-blocking API.
  - nottorp 3mo ago
    SPI the bus?
Asmod4n 3mo ago
The main reason why it gets disabled is fixed now, the latest RC got cBPF support and as such you can restrict what OPs can be run now instead of just fully disabling it.
- mort96 3mo ago
  Well the reason it's disabled now is the recent history of pretty bad vulnerabilities. It probably needs to go a while without new vulnerabilities before it makes sense to enable by default. It's pretty complex completely unsafe C code, after all.
  - tempaccount420 3mo ago
    It's not complex actually, but it is C...
  - Asmod4n 3mo ago
    Then just disable that singular function which causes the issue instead of a whole kernel subsystem. The current release can do that.
    - mort96 3mo ago
      What's the singular function that has a chance of turning out to be exploitable in the future?
      - Asmod4n 3mo ago
        With seccomp you can disable a single syscall which might be exploitable since you didn’t get around to patch your kernel, with cBPF you can now do the same for io_uring
        mort96 3mo ago
        And which single syscall might be exploitable? I don't wanna wait until after an exploit to disable it

happyPersonR 3mo ago

Any kind of poll mode networking: Rdma, dpdk, io_uring it’s really kind of up to the user to do the memory isolation In io_urings case tho, you can’t do much because the rings are in the kernel. I’m hopeful though that with Llm things will get better. But it’s just hard problem to solve . Very difficult to do in the kernel itself, and folks don’t really even understand tuning for it.

kshri24 3mo ago
The ring buffers are in shared memory not kernel private. The ring buffers (submission and completion) are shared between kernel space and user space. User publishes requests via submission queue entries (updates tail of buffer while kernel reads head of the buffer), kernel shifts the submission queue buffer on its end and returns a completion queue event by publishing to completion buffer. User pulls from this buffer (specifically the head, kernel updates tail of buffer) in user space.

omcnoe 3mo ago

For a project like Go, wouldn't it be an option to do one-time iouring feature detection in the runtime startup? Exploits are an issue for the entire OS, not the program choosing to use iouring, yeah?

csdreamer7 3mo ago

RHEL 9 and 10 now fully support io_uring by default. It is very recent, but this covers a lot of corporate Linux installs. Gemini 'said' Ubuntu and SuSE support it as well, but did not provide any links to prove it. https://access.redhat.com/solutions/4723221 https://access.redhat.com/solutions/4723221 Go should reconsider support. They should have a 'go' at it.

insanitybit 3mo ago
It's still seccomp'd off in most environments because io-uring is still a seccomp bypass that doesn't play well with kernel security systems (audit subsystem), even if it weren't also like the #1 or #2 exploit vector for privesc.
- Asmod4n 3mo ago
  That’s solved as of last week, you can use cBPF now to disable functionality.
  - insanitybit 3mo ago
    How solved? AFAIK it's not meaningfully shipped but happy to hear otherwise.
    - Asmod4n 3mo ago
      you can now disable opertions with cBPF, like you would be able to with seccomp for normal syscalls.
      - cyphar 3mo ago
        Their point is that that functionality is not available on older kernels (such as those in RHEL 9 and 10) and so most sandboxes will continue to block it outright for a while, though eventually one would expect Red Hat to backport it. (We haven't even added support for the new cBPF io_uring stuff to low-level container runtimes like runc yet, though I did review the patchset on LKML earlier this year and planned to get working on it when I have time. But as it requires spec changes, expect it to take 6-12 months at best...)

Cloudef 3mo ago

Quite depends, I had times when my posix emulation of io_uring (with poll, not epoll) was faster than io_uring. For large zero-copy buffers, io_uring is king however. Also io_uring is useful even for non asynchronous IO as it can implement chain of operations as single atomic operation (mkdir + open it for example). For something like networking, if you are maximizing packets per second, you'll hit kernel limits[1] very quickly and instead have to start leveraging features like GSO/GRO or completely bypass the network stack. 1: https://github.com/axboe/liburing/discussions/1346 https://github.com/axboe/liburing/discussions/1346

lukeh 3mo ago
Also it’s nice for things like SPI which have no user space non-blocking API.
- nottorp 3mo ago
  SPI the bus?

Asmod4n 3mo ago

The main reason why it gets disabled is fixed now, the latest RC got cBPF support and as such you can restrict what OPs can be run now instead of just fully disabling it.

mort96 3mo ago
Well the reason it's disabled now is the recent history of pretty bad vulnerabilities. It probably needs to go a while without new vulnerabilities before it makes sense to enable by default. It's pretty complex completely unsafe C code, after all.
- tempaccount420 3mo ago
  It's not complex actually, but it is C...
- Asmod4n 3mo ago
  Then just disable that singular function which causes the issue instead of a whole kernel subsystem. The current release can do that.
  - mort96 3mo ago
    What's the singular function that has a chance of turning out to be exploitable in the future?
    - Asmod4n 3mo ago
      With seccomp you can disable a single syscall which might be exploitable since you didn’t get around to patch your kernel, with cBPF you can now do the same for io_uring
      - mort96 3mo ago
        And which single syscall might be exploitable? I don't wanna wait until after an exploit to disable it