7 ms·

I'm just guessing, but... "developer gets a great idea - let's push an update to the API as a GET request so we can cache this on the CDN... forgetting that th

I'm just guessing, but... "developer gets a great idea - let's push an update to the API as a GET request so we can cache this on the CDN... forgetting that the JWT token is potentially returned in the call. Now, whoever makes the call first gets their JWT token stored for everyone else to load instead when the API call is made." Ta-da, Klarna.

miohtama 5y ago
I expect something exactly like this happened. I had a similar bug long time ago. Apache was somehow incorrectly caching the request and the session cookie in the request ended up in a cache. But it happened only about 1/10,000th of the time so it was impossible to figure out the root cause. However, one common source for this kind of bugs is to ”cache any URL ending .pdf as a static file” and then you are in fact serving logged in PDFs like customer invoices that come with the session cookie. I think CloudFlare used to come with a default rule to treat .pdf as a static content. The responses were cached when you hit their ”cache the good stuff” checkbox.
dminor 5y ago
Years ago I added varnish in front of a website to cache image requests, not realizing that if the response included 'set-cookie' that was also cached. We immediately started getting reports of random products appearing in our customers' shopping carts, as people's sessions got merged with random strangers.
- Puts 5y ago
  Just feel the urge to point out that Varnish by default do specifically not cache requests with a set-cookie header. :)
growt 5y ago
I introduced a similar bug into one of my products in the past (Be honest, who hasn't?). But I'm surprised here because Klarna is a quite mature product and something like this shouldn't really happen at that stage.
- yawaramin 5y ago
  Oh, it can definitely happen even in mature products. One I worked on had pretty much the same issue as Klarna (people seeing others' info) when someone updated a web client library we were using to a new version that subtly changed how it handled concurrency.
zitterbewegung 5y ago
I remember something similar when there was a load balancing issue with some website where it would randomly assigning a user with someone else's account.
iratewizard 5y ago
To get around this, one could include the request IP address in the JWT and required a refresh token to be sent when the user's IP switches.
- secureleaf 5y ago
  This is not a safe method for protecting against this type of cache vulnerability. IP addresses are regularly shared by multiple users, especially when behind NAT (even mobile ISPs are doing carrier grade NAT these days).
  - iratewizard 5y ago
    So there should be no fail safe since it can't be guaranteed to work in every scenario.
- remram 5y ago
  In this context, this would just prevent everybody from logging in. The JWT would correctly get rejected but people would still be getting the wrong token from the CDN over and over.
  - iratewizard 5y ago
    Which would you rather? The situation you just described or users accidentally spoofing each other's session?
irjustin 5y ago
I can 100% see this being the cause if this comes out as the root. But... API's really shouldn't be cached? At least not at the CDN level. The risk of serving up stale dashboard data alone makes users go ????... and we definitely don't want - not even mentioning the problem here, that's crazy.
- chrisrogers 5y ago
  Depends on the scope of the API of course, but it's a good rule of thumb for any API with private auth
- toredash 5y ago
  Of course you can cache it, but your assuming it should never. Nothing wrong with caching API calls on the CDN forever as long as your purge the cache once you need it. Event based purging.
  - cowmoo728 5y ago
    "There are only two hard things in Computer Science: cache invalidation and naming things." Cache invalidation is always a very tricky affair. It can work for a while but as complexity grows it gets very hard to maintain and debug. It's very much a "here be dragons" situation and you have to go into it with your guard up. I was at a small startup that had a quick and dirty contractor built API. It worked, but for our largest customers, 99th percentile latency started going over the API gateway timeout. The quick and dirty hack on top of it was aggressive caching with too-clever invalidation logic. It worked until new features were added and then it started failing dramatically and unpredictably. The bugs were an absolute nightmare. We ended up spending almost a year cleaning up the data model, sharding things by customer, and fixing a bunch of N+1 queries, all so that we could get rid of our API cache layer and kill the bugs for good.
- beejiu 5y ago
  100% agree with this. A database is, in some form, a cache of its own. If you have to add additional cache on top, it's an additional source of complexity and risk. If you are building a financial platform, you should DESIGN around this.
deleted 5y ago
[deleted]
akamia 5y ago
I worked with a team that owned a service that resizes images. An engineer was assigned a task to add support for auto rotating images. His solution involved saving the image to a file and then using a library to handle the rotation. He used a hardcoded value for the file name. In a local environment where requests are sparse this looked fine to him and other engineers on the team missed it in code reviews. It wasn't until it went out to prod that he realized the error in this. Users started seeing other users' images because the file's content was constantly being overwritten. When you test features like this or caching a response with a JWT it can be very easy to default to the happy path or ignore the impact of a large volume of concurrent users.
- auggierose 5y ago
  "An engineer was assigned" Nope. That definitely wasn't an engineer.
  - beckingz 5y ago
    Real software engineers don't make mistakes?
    - auggierose 5y ago
      Not mistakes like that.
  - _vertigo 5y ago
    No true Scottish engineer would have made that error!
    - auggierose 5y ago
      :-)
  - akamia 5y ago
    Mistakes happen. I've never met an engineer who has never made a mistake. However, I have met brilliant engineers who have written incredibly complex software and have also managed to make some silly mistakes along the way.
AtNightWeCode 5y ago
I doubt that Klarna, a bank, have OSI layer 7 proxies in the cloud, with TLS termination in their CDN solution, on AWS. I would assume this traffic is outside of that. But then again, I know they wasted 25M+ Euros on a garbage NodeJS platform. They also created an own cloud once. Yes, it is in the trash bin.
- piva00 5y ago
  What makes you doubt that?
- darthrupert 5y ago
  Surprisingly many IT companies tried to create their own clouds, or at least their own kubernetes.
  - jordanbeiber 5y ago
    Surprisingly many have saved boatloads of time automating processes pertaining to the tasks at hand. So, yeah, sound reasonable. :)
- mekkkkkk 5y ago
  I'd actually bet against you on that one. They are still stuck with one foot in the startup mindset.
- jordanbeiber 5y ago
  They didn’t “create” their own cloud - they wanted to host their own hardware using an api layer to provision resources. That stuff was not built in-house. Manhandled in-house though...
  - AtNightWeCode 5y ago
    Sebastian used the word cloud when I met him.
    - Hikikomori 5y ago
elamje 5y ago
This reminds me - A couple of years back, I was making https://lifeboxhq.com https://lifeboxhq.com which involved users uploading quite a bit of content. I was happily testing security with some url resource enumeration and for some reason, I could non-deterministically access user uploads via url, even on accounts I didn't own. I spent several days looking at my Flask code, javascript, etc. to debug.... I knew it wasn't my code, but I was getting more and more frustrated, then I remembered I set up Cloudflare.... Remember to exclude certain routes from Cloudflare if you want to avoid arbitrary user content from being cached without authentication.

miohtama 5y ago

I expect something exactly like this happened. I had a similar bug long time ago. Apache was somehow incorrectly caching the request and the session cookie in the request ended up in a cache. But it happened only about 1/10,000th of the time so it was impossible to figure out the root cause. However, one common source for this kind of bugs is to ”cache any URL ending .pdf as a static file” and then you are in fact serving logged in PDFs like customer invoices that come with the session cookie. I think CloudFlare used to come with a default rule to treat .pdf as a static content. The responses were cached when you hit their ”cache the good stuff” checkbox.

dminor 5y ago

Years ago I added varnish in front of a website to cache image requests, not realizing that if the response included 'set-cookie' that was also cached. We immediately started getting reports of random products appearing in our customers' shopping carts, as people's sessions got merged with random strangers.

Puts 5y ago
Just feel the urge to point out that Varnish by default do specifically not cache requests with a set-cookie header. :)

growt 5y ago

I introduced a similar bug into one of my products in the past (Be honest, who hasn't?). But I'm surprised here because Klarna is a quite mature product and something like this shouldn't really happen at that stage.

yawaramin 5y ago
Oh, it can definitely happen even in mature products. One I worked on had pretty much the same issue as Klarna (people seeing others' info) when someone updated a web client library we were using to a new version that subtly changed how it handled concurrency.

zitterbewegung 5y ago

I remember something similar when there was a load balancing issue with some website where it would randomly assigning a user with someone else's account.

iratewizard 5y ago

To get around this, one could include the request IP address in the JWT and required a refresh token to be sent when the user's IP switches.

secureleaf 5y ago
This is not a safe method for protecting against this type of cache vulnerability. IP addresses are regularly shared by multiple users, especially when behind NAT (even mobile ISPs are doing carrier grade NAT these days).
- iratewizard 5y ago
  So there should be no fail safe since it can't be guaranteed to work in every scenario.
remram 5y ago
In this context, this would just prevent everybody from logging in. The JWT would correctly get rejected but people would still be getting the wrong token from the CDN over and over.
- iratewizard 5y ago
  Which would you rather? The situation you just described or users accidentally spoofing each other's session?

irjustin 5y ago

I can 100% see this being the cause if this comes out as the root. But... API's really shouldn't be cached? At least not at the CDN level. The risk of serving up stale dashboard data alone makes users go ????... and we definitely don't want - not even mentioning the problem here, that's crazy.

chrisrogers 5y ago
Depends on the scope of the API of course, but it's a good rule of thumb for any API with private auth
toredash 5y ago
Of course you can cache it, but your assuming it should never. Nothing wrong with caching API calls on the CDN forever as long as your purge the cache once you need it. Event based purging.
- cowmoo728 5y ago
  "There are only two hard things in Computer Science: cache invalidation and naming things." Cache invalidation is always a very tricky affair. It can work for a while but as complexity grows it gets very hard to maintain and debug. It's very much a "here be dragons" situation and you have to go into it with your guard up. I was at a small startup that had a quick and dirty contractor built API. It worked, but for our largest customers, 99th percentile latency started going over the API gateway timeout. The quick and dirty hack on top of it was aggressive caching with too-clever invalidation logic. It worked until new features were added and then it started failing dramatically and unpredictably. The bugs were an absolute nightmare. We ended up spending almost a year cleaning up the data model, sharding things by customer, and fixing a bunch of N+1 queries, all so that we could get rid of our API cache layer and kill the bugs for good.
beejiu 5y ago
100% agree with this. A database is, in some form, a cache of its own. If you have to add additional cache on top, it's an additional source of complexity and risk. If you are building a financial platform, you should DESIGN around this.

deleted 5y ago

[deleted]

akamia 5y ago

I worked with a team that owned a service that resizes images. An engineer was assigned a task to add support for auto rotating images. His solution involved saving the image to a file and then using a library to handle the rotation. He used a hardcoded value for the file name. In a local environment where requests are sparse this looked fine to him and other engineers on the team missed it in code reviews. It wasn't until it went out to prod that he realized the error in this. Users started seeing other users' images because the file's content was constantly being overwritten. When you test features like this or caching a response with a JWT it can be very easy to default to the happy path or ignore the impact of a large volume of concurrent users.

auggierose 5y ago
"An engineer was assigned" Nope. That definitely wasn't an engineer.
- beckingz 5y ago
  Real software engineers don't make mistakes?
  - auggierose 5y ago
    Not mistakes like that.
- _vertigo 5y ago
  No true Scottish engineer would have made that error!
  - auggierose 5y ago
    :-)
- akamia 5y ago
  Mistakes happen. I've never met an engineer who has never made a mistake. However, I have met brilliant engineers who have written incredibly complex software and have also managed to make some silly mistakes along the way.

AtNightWeCode 5y ago

I doubt that Klarna, a bank, have OSI layer 7 proxies in the cloud, with TLS termination in their CDN solution, on AWS. I would assume this traffic is outside of that. But then again, I know they wasted 25M+ Euros on a garbage NodeJS platform. They also created an own cloud once. Yes, it is in the trash bin.

piva00 5y ago
What makes you doubt that?
darthrupert 5y ago
Surprisingly many IT companies tried to create their own clouds, or at least their own kubernetes.
- jordanbeiber 5y ago
  Surprisingly many have saved boatloads of time automating processes pertaining to the tasks at hand. So, yeah, sound reasonable. :)
mekkkkkk 5y ago
I'd actually bet against you on that one. They are still stuck with one foot in the startup mindset.
jordanbeiber 5y ago
They didn’t “create” their own cloud - they wanted to host their own hardware using an api layer to provision resources. That stuff was not built in-house. Manhandled in-house though...
- AtNightWeCode 5y ago
  Sebastian used the word cloud when I met him.
  - Hikikomori 5y ago

elamje 5y ago

This reminds me - A couple of years back, I was making https://lifeboxhq.com https://lifeboxhq.com which involved users uploading quite a bit of content. I was happily testing security with some url resource enumeration and for some reason, I could non-deterministically access user uploads via url, even on accounts I didn't own. I spent several days looking at my Flask code, javascript, etc. to debug.... I knew it wasn't my code, but I was getting more and more frustrated, then I remembered I set up Cloudflare.... Remember to exclude certain routes from Cloudflare if you want to avoid arbitrary user content from being cached without authentication.