6 ms·

We are talking about an insane amount of data here. It was 56 Gbit/s (or 56 x 1 Gbit servers 100% saturated!). This is not something a "caching server" could ha

We are talking about an insane amount of data here. It was 56 Gbit/s (or 56 x 1 Gbit servers 100% saturated!). This is not something a "caching server" could handle. We are talking on the order of CDN networks, like Cloudflare, to be able to handle this.

wyager 1y ago
> or 56 x 1 Gbit servers 100% saturated Presumably a caching server would be 10GbE, 40GbE, or 100GbE 56Gbit/sec of pre-generated data is definitely something that you can handle from 1 or 2 decent servers, assuming each request doesn't generate a huge number of random disk reads or something
ndriscoll 1y ago
I'd be somewhat surprised if nginx couldn't saturate a 10Gbit link with an n150 serving static files, so I'd expect 6x $200 minipcs to handle it. I'd think the expensive part would be the hosting/connection.
Sesse__ 1y ago
> We are talking about an insane amount of data here. It was 56 Gbit/s. This is not something a "caching server" could handle. You are not talking about an insane amount of data if it's 56 Gbit/s. Of course a caching server could handle that. Source: Has written servers that saturated 40gig (with TLS) on an old quadcore.
- hyperknot 1y ago
  OK, technically there might exist such server, I guess Netflix and friends are using those. But we are talking about a community supported, free service here. Hetzner servers are my only options, because of their unmetered bandwidth.
  - Sesse__ 1y ago
    It really depends on the size of the active set. If it fits into RAM of whatever server you are using, then it's not a problem at all, even with completely off-the-shelf hardware and software. Slap two 40gig NICs in it, install Varnish or whatever and you're good to go. (This is, of course, assuming that you have someone willing to pay for the bandwidth out to your users!) If you need to go to disk to serve large parts of it, it's a different beast. But then again, Netflix was doing 800gig already three years ago (in large part from disk) and they are handicapping themselves by choosing an OS where they need to do significant amounts of the scaling work themselves.
    - hyperknot 1y ago
      I'm sure the server hardware is not a problem. The full dataset is 150 GB and the server has 64 GB RAM, most of which will be never requested. So I'm sure that the used tiles would actually get served from OS cache. If not, it's on a RAID 0 NVME SSD, connected locally. What I've been referring to is the fact that even unlimited 1 Gbps connections can be quite expensive, now try to find a 2x40 gig connection for a reasonable money. That one user generated 200 TB in 24 hours! I have no idea about bandwidth pricing, but I bet it ain't cheap to serve that.
      - Sesse__ 1y ago
        Well, “bandwidth is expensive” is a true claim, but it's also a very different claim from “a [normal] caching server couldn't handle 56 Gbit/sec”…?
        hyperknot 1y ago
        You are correct. I was putting "a caching server on their side" in the context of their side being a single dev hobby project running on a VPS, exploding on the weekend. I agree that these servers do exist and some companies do pay for this bandwidth as part of their normal operations.
        Aeolun 1y ago
        56 Gbit/sec costs you about €590/day even on Hetzner.
- bigstrat2003 1y ago
  I realize that what constitutes "insane" is a subjective judgement. But, uh... I most certainly would call 56 Gbps insane. Which is not to say that hardware which handles it doesn't exist. It might not even be especially insane hardware. But that is a pretty insane data rate in my book.

wyager 1y ago

> or 56 x 1 Gbit servers 100% saturated Presumably a caching server would be 10GbE, 40GbE, or 100GbE 56Gbit/sec of pre-generated data is definitely something that you can handle from 1 or 2 decent servers, assuming each request doesn't generate a huge number of random disk reads or something

ndriscoll 1y ago

I'd be somewhat surprised if nginx couldn't saturate a 10Gbit link with an n150 serving static files, so I'd expect 6x $200 minipcs to handle it. I'd think the expensive part would be the hosting/connection.

Sesse__ 1y ago

> We are talking about an insane amount of data here. It was 56 Gbit/s. This is not something a "caching server" could handle. You are not talking about an insane amount of data if it's 56 Gbit/s. Of course a caching server could handle that. Source: Has written servers that saturated 40gig (with TLS) on an old quadcore.

hyperknot 1y ago
OK, technically there might exist such server, I guess Netflix and friends are using those. But we are talking about a community supported, free service here. Hetzner servers are my only options, because of their unmetered bandwidth.
- Sesse__ 1y ago
  It really depends on the size of the active set. If it fits into RAM of whatever server you are using, then it's not a problem at all, even with completely off-the-shelf hardware and software. Slap two 40gig NICs in it, install Varnish or whatever and you're good to go. (This is, of course, assuming that you have someone willing to pay for the bandwidth out to your users!) If you need to go to disk to serve large parts of it, it's a different beast. But then again, Netflix was doing 800gig already three years ago (in large part from disk) and they are handicapping themselves by choosing an OS where they need to do significant amounts of the scaling work themselves.
  - hyperknot 1y ago
    I'm sure the server hardware is not a problem. The full dataset is 150 GB and the server has 64 GB RAM, most of which will be never requested. So I'm sure that the used tiles would actually get served from OS cache. If not, it's on a RAID 0 NVME SSD, connected locally. What I've been referring to is the fact that even unlimited 1 Gbps connections can be quite expensive, now try to find a 2x40 gig connection for a reasonable money. That one user generated 200 TB in 24 hours! I have no idea about bandwidth pricing, but I bet it ain't cheap to serve that.
    - Sesse__ 1y ago
      Well, “bandwidth is expensive” is a true claim, but it's also a very different claim from “a [normal] caching server couldn't handle 56 Gbit/sec”…?
      - hyperknot 1y ago
        You are correct. I was putting "a caching server on their side" in the context of their side being a single dev hobby project running on a VPS, exploding on the weekend. I agree that these servers do exist and some companies do pay for this bandwidth as part of their normal operations.
      - Aeolun 1y ago
        56 Gbit/sec costs you about €590/day even on Hetzner.
bigstrat2003 1y ago
I realize that what constitutes "insane" is a subjective judgement. But, uh... I most certainly would call 56 Gbps insane. Which is not to say that hardware which handles it doesn't exist. It might not even be especially insane hardware. But that is a pretty insane data rate in my book.