5 ms·

Unpkg CDN down causing dependent website outages

40 points by ManWith2Plans 2y ago

taspeotis 2y ago
I assume the developers who use a free third party service for serving their dependencies are the same devs who :pikachu_face: when they are told they can’t statically link GPL libraries into their proprietary apps.
- afavour 2y ago
  Personally I’d say the overlap between web developers linking out to a JS CDN and native developers statically linking GPL libraries is probably pretty small.
jimaek 2y ago
I invite everyone affected to consider switching to a production focused CDN https://www.jsdelivr.com/ https://www.jsdelivr.com/ There is also a tool to simplify migration https://www.jsdelivr.com/unpkg https://www.jsdelivr.com/unpkg
- account42 2y ago
  I invite everyone (even those not affected in this instance) to consider switching away from relying on random third parties for your hosting. Like wtf how is did this ever become the norm even when there was a slight performance advantage from a shared cache, which doesn't even exist anymore these days.
  - arkh 2y ago
    A very specific problem with third party hosts : if you use noscript and only allow main domain script as a default, you have to either whitelist those CDN or authorize them on every site using it. Not the best experience. Especially with some sites having dozens of third party scripts so instead of trying to find the right one(s) to authorize you just close the tab and forget about it.
  - alzoid 2y ago
    I do both. If you are loading a JS lib then check for its existence on the client. If it's not found load the resource from your server. You could also achieve this on the server side.
    - afavour 2y ago
      Or you could simplify the whole thing and just host it yourself. It isn’t worth that level of complication.
      - alzoid 2y ago
        In this situation server bandwidth was an issue. Using an available CDN with the fall back was what we decided to do. It's not that complicated, from the client check if the object you expect to exist in the library exists, if not load it from your server.
        afavour 2y ago
        I’m curious, in what scenario was bandwidth for static resources an issue? It’s one of the most trivial things to serve cheaply.
        alzoid 2y ago
        When the server is behind a pipe of limited size. When most of your new visits happen all at once over a short period of time. When you can't rely on a mobile phones cache like a desktop because 90% of your visitors are using one. A CDN is a easy win to off load resources, even if they are small.
        afavour 2y ago
        > A CDN is an easy win to off load resources, even if they are small. Yes and setting up your own CDN config (Cloudfront, whatever) is minutes of work and brings all of the benefits you’ve outlined. And costs pennies.
- madeofpalk 2y ago
  Why? If you care about being even remotely reliable (hinted at with "production-focused"), why would you do this? Just use the same host as where you serve everything else from. Browsers have partitioned caches per origin for over 10 years, so there's no performance gains from getting cache hits from other websites.
grepfru_it 2y ago
“Works on my machine” Depend on external services, die by external services. If you can’t afford redundancy, then you must accept downtime. I build all of my dependent libraries locally and bundle with my app. For one application, I maintain and build the entire OS from source. I depend on aws,azure cloud storage for my own hosted cdn. I can fail back to my own hosts if necessary.
cagenut 2y ago
its a fascinating little tech-anthropology quirk that front-end/web developers use the term "CDN" to mean a different thing from how devops/sre/infra people use the term. in the strictest sense, unpkg is not a cdn. unpkg is a javascript library hosting service that uses cloudflare as a cdn. cloudflare is not down, therefore this headline "unpkg cdn down..." reads as a massive "oh shit!" to infra/devops type cloudflare customers who then have to immediately worry that their site is down, only to realize no, they're fine, this is just devs using the term to mean a different thing (roughly: "third party javascript host").
- re-thc 2y ago
  > its a fascinating little tech-anthropology quirk that front-end/web developers use the term "CDN" to mean a different thing from how devops/sre/infra people use the term. Do they? Same to me. > in the strictest sense, unpkg is not a cdn. unpkg is a javascript library hosting service that uses cloudflare as a cdn. This is just arguing over semantics. It even says on the website that unpkg CDN is powered by Cloudflare. By your analogy if a website hosted on AWS is down, is the whole of AWS down? If I provide managed kubernetes on GCP and my service is down, does that make GKE or GCP down? No. A CDN is a CDN just like a website is a website, whether it's backed by Wordpress, NextJs or any other thing.
CoastalCoder 2y ago
This is one thing I never understood about web development. I would have thought the risks from 3rd party production dependencies would make devs do all in their power to minimize them. But IIUC it's a somewhat common practice. Is there something about the market dynamics that make this a reasonable tradeoff? E.g., the problems are rare and this somehow allows faster time to market? (Asking as a systems programmer, very much not a web developer.)
- giancarlostoro 2y ago
  Personally if I'm going to rely on someone to host it, I check if my hosting provider does it for me already. I believe Microsoft pre-hosts some JS packages. Google hosts fonts and such as well. I also miss being able to host random CSS and assets on Tumblr. Otherwise, it should be on my own CDN. Especially for enterprise sites.
- afavour 2y ago
  > But IIUC it's a somewhat common practice. I don’t have any data but I really don’t think it actually is that common. I definitely heard it used a lot more in the 2000s jQuery era but not really today. Browsers even have isolated storage these days so the benefits really are infinitesimal.
  - evilduck 2y ago
    I think it's fairly common. I was reviewing an implementation of a phone number input component last month. It looked great, changing the country code automatically showed the correct country's flag or vice versa select your flag and the country code updated, plus a bunch of useful phone number validation logic was provided, you just add the NPM package and use the resulting component and magic happened. But open up dev tools and you see that all the flags are being brought in from a CDN. The only fix is to fork the library and change asset paths in their code to self host. Many libraries are like this, icon collections, design system web components, whatever. If you're not vigilant it's easy to lazily include CDN hosted assets without explicit intent.
    - 2y ago
pier25 2y ago
So what was the issue? Unpkg runs on Fly with Cloudflare as CDN.
apitman 2y ago
Unless you're running a site that you know is constantly getting hits from all over the globe, using a CDN will make your site load slower for some users. And you almost certainly don't need one for scaling purposes. I've handled frontpage traffic multiple times without. What I wish existed (or maybe does) is something like a CDN with ~8 global locations that promises to keep your static site in a warm cache for a small fee, without me having to set up a fly.io app manually. That would let my blog always load fast anywhere on the planet even for the first hit in a region.
- supriyo-biswas 2y ago
  You're looking for a CDN with an integrated storage service, bunny.net has that feature[1]. They also have a volume tier for cheaply serving files[2] which seems to meet your need for a low PoP count (with some discounts to go along with it). [1] https://support.bunny.net/hc/en-us/articles/360017093479-Understanding-Perma-Cache https://support.bunny.net/hc/en-us/articles/360017093479-Und... [2] https://bunny.net/network/ https://bunny.net/network/
- KomoD 2y ago
  > What I wish existed (or maybe does) is something like a CDN with ~8 global locations that promised to keep your static site in a warm cache for a small fee cloudflare cache reserve or bunny.net as someone else mentioned
- madeofpalk 2y ago
  > What I wish existed (or maybe does) is something like a CDN with ~8 global locations that promised to keep your static site in a warm cache for a small fee IIRC this is what Cloudflare Argo is supposed to do? https://www.cloudflare.com/en-gb/application-services/products/argo-smart-routing/ https://www.cloudflare.com/en-gb/application-services/produc... Rather than routing cache misses to origin, it'll route to other PoPs with the cached content?
deleted 2y ago
[deleted]

40 points by ManWith2Plans 2y ago

I assume the developers who use a free third party service for serving their dependencies are the same devs who :pikachu_face: when they are told they can’t statically link GPL libraries into their proprietary apps.

afavour 2y ago
Personally I’d say the overlap between web developers linking out to a JS CDN and native developers statically linking GPL libraries is probably pretty small.

jimaek 2y ago

I invite everyone affected to consider switching to a production focused CDN https://www.jsdelivr.com/ https://www.jsdelivr.com/ There is also a tool to simplify migration https://www.jsdelivr.com/unpkg https://www.jsdelivr.com/unpkg

account42 2y ago
I invite everyone (even those not affected in this instance) to consider switching away from relying on random third parties for your hosting. Like wtf how is did this ever become the norm even when there was a slight performance advantage from a shared cache, which doesn't even exist anymore these days.
- arkh 2y ago
  A very specific problem with third party hosts : if you use noscript and only allow main domain script as a default, you have to either whitelist those CDN or authorize them on every site using it. Not the best experience. Especially with some sites having dozens of third party scripts so instead of trying to find the right one(s) to authorize you just close the tab and forget about it.
- alzoid 2y ago
  I do both. If you are loading a JS lib then check for its existence on the client. If it's not found load the resource from your server. You could also achieve this on the server side.
  - afavour 2y ago
    Or you could simplify the whole thing and just host it yourself. It isn’t worth that level of complication.
    - alzoid 2y ago
      In this situation server bandwidth was an issue. Using an available CDN with the fall back was what we decided to do. It's not that complicated, from the client check if the object you expect to exist in the library exists, if not load it from your server.
      - afavour 2y ago
        I’m curious, in what scenario was bandwidth for static resources an issue? It’s one of the most trivial things to serve cheaply.
        alzoid 2y ago
        When the server is behind a pipe of limited size. When most of your new visits happen all at once over a short period of time. When you can't rely on a mobile phones cache like a desktop because 90% of your visitors are using one. A CDN is a easy win to off load resources, even if they are small.
        afavour 2y ago
        > A CDN is an easy win to off load resources, even if they are small. Yes and setting up your own CDN config (Cloudfront, whatever) is minutes of work and brings all of the benefits you’ve outlined. And costs pennies.
madeofpalk 2y ago
Why? If you care about being even remotely reliable (hinted at with "production-focused"), why would you do this? Just use the same host as where you serve everything else from. Browsers have partitioned caches per origin for over 10 years, so there's no performance gains from getting cache hits from other websites.

grepfru_it 2y ago

“Works on my machine” Depend on external services, die by external services. If you can’t afford redundancy, then you must accept downtime. I build all of my dependent libraries locally and bundle with my app. For one application, I maintain and build the entire OS from source. I depend on aws,azure cloud storage for my own hosted cdn. I can fail back to my own hosts if necessary.

cagenut 2y ago

its a fascinating little tech-anthropology quirk that front-end/web developers use the term "CDN" to mean a different thing from how devops/sre/infra people use the term. in the strictest sense, unpkg is not a cdn. unpkg is a javascript library hosting service that uses cloudflare as a cdn. cloudflare is not down, therefore this headline "unpkg cdn down..." reads as a massive "oh shit!" to infra/devops type cloudflare customers who then have to immediately worry that their site is down, only to realize no, they're fine, this is just devs using the term to mean a different thing (roughly: "third party javascript host").

re-thc 2y ago
> its a fascinating little tech-anthropology quirk that front-end/web developers use the term "CDN" to mean a different thing from how devops/sre/infra people use the term. Do they? Same to me. > in the strictest sense, unpkg is not a cdn. unpkg is a javascript library hosting service that uses cloudflare as a cdn. This is just arguing over semantics. It even says on the website that unpkg CDN is powered by Cloudflare. By your analogy if a website hosted on AWS is down, is the whole of AWS down? If I provide managed kubernetes on GCP and my service is down, does that make GKE or GCP down? No. A CDN is a CDN just like a website is a website, whether it's backed by Wordpress, NextJs or any other thing.

CoastalCoder 2y ago

This is one thing I never understood about web development. I would have thought the risks from 3rd party production dependencies would make devs do all in their power to minimize them. But IIUC it's a somewhat common practice. Is there something about the market dynamics that make this a reasonable tradeoff? E.g., the problems are rare and this somehow allows faster time to market? (Asking as a systems programmer, very much not a web developer.)

giancarlostoro 2y ago
Personally if I'm going to rely on someone to host it, I check if my hosting provider does it for me already. I believe Microsoft pre-hosts some JS packages. Google hosts fonts and such as well. I also miss being able to host random CSS and assets on Tumblr. Otherwise, it should be on my own CDN. Especially for enterprise sites.
afavour 2y ago
> But IIUC it's a somewhat common practice. I don’t have any data but I really don’t think it actually is that common. I definitely heard it used a lot more in the 2000s jQuery era but not really today. Browsers even have isolated storage these days so the benefits really are infinitesimal.
- evilduck 2y ago
  I think it's fairly common. I was reviewing an implementation of a phone number input component last month. It looked great, changing the country code automatically showed the correct country's flag or vice versa select your flag and the country code updated, plus a bunch of useful phone number validation logic was provided, you just add the NPM package and use the resulting component and magic happened. But open up dev tools and you see that all the flags are being brought in from a CDN. The only fix is to fork the library and change asset paths in their code to self host. Many libraries are like this, icon collections, design system web components, whatever. If you're not vigilant it's easy to lazily include CDN hosted assets without explicit intent.
  - 2y ago

pier25 2y ago

So what was the issue? Unpkg runs on Fly with Cloudflare as CDN.

apitman 2y ago

Unless you're running a site that you know is constantly getting hits from all over the globe, using a CDN will make your site load slower for some users. And you almost certainly don't need one for scaling purposes. I've handled frontpage traffic multiple times without. What I wish existed (or maybe does) is something like a CDN with ~8 global locations that promises to keep your static site in a warm cache for a small fee, without me having to set up a fly.io app manually. That would let my blog always load fast anywhere on the planet even for the first hit in a region.

supriyo-biswas 2y ago
You're looking for a CDN with an integrated storage service, bunny.net has that feature[1]. They also have a volume tier for cheaply serving files[2] which seems to meet your need for a low PoP count (with some discounts to go along with it). [1] https://support.bunny.net/hc/en-us/articles/360017093479-Understanding-Perma-Cache https://support.bunny.net/hc/en-us/articles/360017093479-Und... [2] https://bunny.net/network/ https://bunny.net/network/
KomoD 2y ago
> What I wish existed (or maybe does) is something like a CDN with ~8 global locations that promised to keep your static site in a warm cache for a small fee cloudflare cache reserve or bunny.net as someone else mentioned
madeofpalk 2y ago
> What I wish existed (or maybe does) is something like a CDN with ~8 global locations that promised to keep your static site in a warm cache for a small fee IIRC this is what Cloudflare Argo is supposed to do? https://www.cloudflare.com/en-gb/application-services/products/argo-smart-routing/ https://www.cloudflare.com/en-gb/application-services/produc... Rather than routing cache misses to origin, it'll route to other PoPs with the cached content?

deleted 2y ago

[deleted]