benlwalker

6 ms·

1.
by benlwalker 1y ago
SPDK will be able to fully saturate the PCIe bandwidth from a single CPU core here (no secret 6 threads inside the kernel). The drives are your bottleneck so it won't go faster, but it can use a lot less CPU. But with SPDK you'll
2.
by benlwalker 1y ago
For an expanding array in a 64 bit address space, reserving a big region and mmaping it in as you go is usually the top performing solution by a wide margin. At least on Linux, it is faster to speculatively mmap ahead with MAP_POPULATE rath
3.
by benlwalker 2y ago
Compared to libraries like bgfx and sokol at least, I think there are two key differences. 1) SDL_gpu is a pure C library, heavily focused on extreme portability and no depedencies. And somehow it's also an order of magnitude less code
4.
by benlwalker 2y ago
The eBPF programs are strictly bounded. And they're scoped to their own memory that you have to pre-load from the actual storage with separate commands issued from the CPU (presumably from the kernel driver which is doing access contro
5.
by benlwalker 2y ago
We tried to standardize exactly this - eBPF programs offloaded onto the device. The NVMe standard now has a lot of infrastructure for this standardized, including commands to discover device memory topology, transfer to/from that memor
6.
by benlwalker 3y ago
Yes, I've seen some clearer cases made for networking. In networking there is no standard for the hardware interface. Every vendor does their own thing. Except many can at least handle virtqueues carrying virtio-net messages for the da
7.
by benlwalker 3y ago
I don't get it either, and I'm a maintainer of SPDK which provides multiple implementations of virtualized devices and is frequently used inside DPUs to present storage devices. If I'm implementing a hardware device anyway, w
8.
by benlwalker 3y ago
Is it really that much code? I don't know GPU hardware, but the NVMe spec header file in SPDK is around 4k lines[0]. If there's 7 of them and they're twice as complicated each, we're still well under 100k from register m
9.
by benlwalker 3y ago
Pre-commit means before committed to the canonical repo, not before commit locally. The SPDK project has an elaborate pre-commit review and test system all in public. See https://spdk.io/development . I wouldn't want t
10.
by benlwalker 3y ago
iWARP maybe, but I don't think you want to offload all of TCP to hardware. You want to leave congestion control and all that to software. I don't entirely know if that's why iWARP isn't very popular, but I suspect that&#
11.
by benlwalker 3y ago
At 200+ Gbps, the copy from LLC where the packet landed to the userspace buffer dominates the performance profiles on most systems. The TCP processing isn't bad and the expensive parts can often be offloaded. I'd contend that this
12.
by benlwalker 3y ago
RDMA is great and similar, but behaves very differently from TCP in the face of network congestion and longer distance traversals. This is essentially trying to get the best parts of TCP and the best parts of RDMA combined.
13.
Using SPDK as Device Firmware(spdk.io)
3 points by benlwalker 3y ago|0 comments
14.
by benlwalker 3y ago
XRP is a regular BPF hook in Linux and requires no additional standardization. The device never "calls out to BPF programs in the driver" - it generates a normal completion interrupt and Linux runs a BPF hook in the completion pat
15.
by benlwalker 3y ago
I am very closely tied to what the NVMe vendors want, having written the first internal draft of the proposal to the standards body (since that draft many smart people have taken the pen and done a lot of great work). XRP is unrelated to of
16.
by benlwalker 3y ago
Associating the desire of NVMe vendors to allow users to ship down eBPF programs to run on the device and XRP is a major mistake in the article. XRP has nothing to do with what the NVMe vendors want to do, and XRP is a pure kernel solution
17.
by benlwalker 3y ago
For many syscalls, the primary overhead is the transition itself, not the work the kernel does. So doing 50 operations one by one may take, say, 10x as much time as a single call to io_uring_enter for the same work. It really shouldn't
18.
by benlwalker 3y ago
You can also use umwait on the next completion entry in the ring
19.
by benlwalker 3y ago
Without the submission batching you lose your system call reduction and that is far and away the biggest benefit.
20.
by benlwalker 3y ago
Imagine you have a piece of software that runs in an event loop (as many things do). On each loop, queue up all system calls you'd like to perform. At the end of the loop, do one syscall to execute the batch. At the start of the loop,
21.
by benlwalker 3y ago
IOCP certainly was ahead of its time, but it only does the completion batching, not the submission batching. io_uring is significantly better than anything available on Windows right now.
22.
by benlwalker 3y ago
I've spent essentially the last year trying to find the best way to use io_uring for networking inside the NVMe-oF target in SPDK. Many of my initial attempts were also slower than our heavily optimized epoll version. But now I feel li
23.
by benlwalker 4y ago
And io_uring itself was more directly inspired by NVMe and RDMA, which of course work with these same queues as GFX cards. The original io_uring patch compares itself to SPDK, whose premise is "what if we expose an abstraction for a ha
24.
by benlwalker 4y ago
Windows did already have async ("overlapped") IO, and a completion aggregator (IOCP) kind of like io_uring. What Windows didn't have, and the reason they're now adding their own IORing, is the ability to submit batches o
25.
by benlwalker 4y ago
As an alternative, may I offer you a wasm port? https://play.classicuo.org/ (I've never used this but I heard it runs well)
26.
by benlwalker 4y ago
There's also now an open source recreation of the original client, all written in C# that actually uses a GPU so it renders 4k at 250fps instead of 800x600 at 12.5fps. It's a very mature and stable reproduction at this point. htt
27.
by benlwalker 4y ago
For me, the killer use case for this is presenting logical volumes to containers. There just has not been an efficient mechanism for a local storage service in one container to serve logical volumes to another container on the same system u
28.
by benlwalker 4y ago
The SPDK project is certainly looking to use this to replace our limited use of NBD, as well as present SPDK block devices as kernel block devices, including devices backed by userspace implementations of iSCSI, NVMe-oF, and various other n
29.
by benlwalker 4y ago
It's the architectural choices more than the language (one thread per core, async, event loops). But I'm sure the c++ does have some benefit over Java.
30.
by benlwalker 5y ago
Nope. The reason this is so complex is that these devices are actually highly parallel machines with multiple queues accepting commands. It's quite difficult to even define "before" in terms of command sequence. For example,

More ›