7 ms·

The Pirate Bay Runs on 21 “Raid-Proof” Virtual Machines

139 points by tjaerv 12y ago

tjaerv 12y ago
"At the time of writing the site uses 21 virtual machines (VMs) hosted at different providers. [...] All virtual machines are hosted with commercial cloud hosting providers, who have no clue that The Pirate Bay is among their customers."
- crazy1van 12y ago
  They may "have no clue" but it seems like that's only because they don't care and haven't looked. I don't see anything in the article that would prevent the providers from figuring this out unless I'm missing something.
  - allegory 12y ago
    If someone is paying the bill, do they really care?
    - nols 12y ago
      The first time TPB was raided they took a bunch of unrelated servers too. Not sure how discriminating the police will be next time, regardless of VMs.
    - larrys 12y ago
      "If someone is paying the bill, do they really care?" So you could cross reference names of the people raided with payment information of the VPS providers (usual suspects or top "n" providers let's say). Of course that could be hidden as well. Other issue is how does anyone know this isn't misinformation anyway and that the VPS providers don't play a role or not as much of a role as is indicated. Just because someone is writing this or because they said it? What advantage does it have for anyone (like this) to reveal anything about how they are situated security wise if not to lead people off the beaten track even given some possible marketing advantage?
      - nacs 12y ago
        > payment information of the VPS providers I doubt this will be useful as they're probably using Bitcoin or prepaid cards or things like that for payments. > What advantage does it have for anyone (like this) to reveal anything It could teach others how to setup websites that are harder to censor or more resilient to raids (plus it gets them free PR/traffic).
    - bowlofpetunias 12y ago
      It's not about the money, it's about not being able to care and survive as a business. Because then they would also have to care about the thousands of other VM's that may run all kinds of stuff that is illegal somewhere, questionable, politically, socially, culturally or commercially sensitive etcetera. No ISP can afford to be proactive about this. They cannot afford to care. Or even know.
  - adwf 12y ago
    I think only the load balancer would be vulnerable to discovery. Everything behind the load balancer could be a secure connection to a completely different datacenter if needed.
  - Wilya 12y ago
    Apart from the external-facing proxy (which is the most exposed link in this setup), these VMs don't need any sort of public presence. Unless the provider inspects processes running on all their customers machines, all they can see is a VM with opaque VPN connections to a few external ips.
TomAnthony 12y ago
If the load balancer is the weak point that would be first to be discovered, then I imagine they must have some mechanism to stop it leaving evidence that leads to the other machines if it were to get raided (it isn't on their hardware, so they can't prevent the files being backed up). Is there a way the codebase could be entirely encrypted and not even accessible to the cloud provider (with some 'boot password' needed each time the server starts up)?
- TheLoneWolfling 12y ago
  It depends on what you mean by that. The only way to prevent a codebase from being seen by an adversary with physical access when the server is on is to not have the sensitive data on the server in the first place. Encryption (with the decryption key being gotten at boot from, say, a particular .onion address) would work against backups, but won't protect against an adversary with admin access to the server when the virtual server is on.
  - aianus 12y ago
    This would be a cool application for some kind of homomorphic encryption. Server gets encrypted search request and matches it against an encrypted index and returns the encrypted results.
    - TheLoneWolfling 12y ago
      Homomorphic encryption isn't fast enough... yet. But it's getting faster! It will be amazing if/when we get to the point where you can have a virtual server where you know that the person with physical access to the server cannot access your data. Even if it is two orders of magnitude slower than raw hardware that's still fast enough for some things. (For example, being able to have a username+password -> personal info database safely run on someone else's hardware.) And once it takes off there probably will start to be hardware support/accelerators for it - like vector intrinsics and AES instructions currently.
    - KMag 12y ago
      The problem with homomorphic encryption in this case is that you need access to the private key in order to interpret the results of the computation. Joe Public encrypts queries using a public key, and the query runs on a possibly compromised server without the attackers learning anything about the database or the query, but then the result of the query looks like absolute gibberish to Joe Public. Giving Joe Public access to the private key necessary for interpreting the query result allows attackers to inspect all of the intermediate states of the query finite state machine, which allows debugging and inspection just as if homomorphic encryption wasn't in use. I suppose the routing proxy could hold the private key and decrypt the query result for the general public. However, the location of the routing proxy is almost certainly going to be compromised before the locations of the servers executing the queries, so in the decrypting proxy scenario, the attackers will almost certainly have the secret keys before they get access to the boxes executing the queries. There's also the problem that the messages being decrypted are the final states of finite state machines that executed the queries, so the messages to be copied over the network add up in size to at least the size of the dataset being queried. (The data can be sharded into many smaller databases, and almost certainly would be in order to speed up the homomorphic computation steps, but this doesn't cut down on the amount of network traffic necessary to retrieve all search results for a single query. A simple query on 1 TB of data, split into 10,000 databases each of 100 MB would require copying and remotely decrypting 10,000 messages, each over 100 MB in size.) It might be possible to discover a homomorphic encryption scheme whereby knowledge of the private key allows one to devise a mapping from a higher dimensional finite state to a lower dimensional finite state machine, where the secret key for the smaller dimensional machine doesn't leak information about the secret key for the larger machine. In this case, it may be possible to perform some finishing operations on the query to prepare it for conversion to the smaller state machine and give the public the private key to the smaller state machine so that the query result could be read from the machine by the public without the public being able to observe intermediate states of the query computation. However, I believe this is far beyond our current mathematical understanding.
fooyc 12y ago
" This saved costs, guaranteed better uptime, and made the site more portable and thus harder to take down " Probably not true for " This saved costs ". From what i've seen, virtual machines usually cost more than twice the price of renting the equivalent "real" machine monthly. They could have used dedicated servers; there are more dedicated server providers than VM providers, thus achieving the same goal, less expensively. Probably not true for " better uptime " either; VMs are still hosted on real hardware, which fails, too. (Although distributing the work on more independent machines can improve uptime.)
- drsintoma 12y ago
  They are more expensive, but they are usually easy and immediate to acquire. Which makes provisioning much more efficient in case of fluctuating traffic. And overall sysadmins will have less tendency to over-provision, meaning getting more and beefier machines than it's needed "to be safe".
  - fooyc 12y ago
    Nothing prevents you from using bare dedicated servers for your usual traffic, and VMs for anything else.
    - TillE 12y ago
      This is clearly the ideal setup for most use cases, and I'm somewhat puzzled as to why it's not more common. I guess using only virtual servers is a tiny bit simpler, so companies will just eat the extra cost.
    - drsintoma 12y ago
      In an ideal world yes. Or if your software works already seamlessly cross-datacenter. But in the real world is rare that your hosting provider is good at both VMs and metal. At least that's the biggest problem I've always encountered, specially with budget providers.
  - RyanZAG 12y ago
Nanzikambe 12y ago
Interesting, so I'm presuming there's several VPNs involved between the load-balancer and all the discrete servers. I wonder if they use a VPN provider with a static IP and no-logs policy or if it's simply yet another VPS. I'd love to hear a little more about the architecture.
- icedchai 12y ago
  Why would they use a VPN provider? It's trivial to set up your own if you control the systems at both ends.
  - TheLoneWolfling 12y ago
    Assuming at least one side isn't under a NAT, true. Otherwise it gets... "fun".
    - the_mitsuhiko 12y ago
      A NAT does not stop you if you do it right. Console games have been hosting servers through NATs for the last decade.
      - TheLoneWolfling 12y ago
        As I said: it gets "fun", especially if you want to do it without a third server to set up the connection. Still doable, just "fun".
        tacotime 12y ago
        I would speculate that the massive advertising revenue generated by the 'bay probably renders any such 'fun' factor trivial.
    - dlgeek 12y ago
      Why would you be under a NAT? A public IP and a networking allotment is one of the core offerings of almost every cloud provider out there.
verroq 12y ago
Why can't people find where their servers? I understand they have their own IP allocation, thus they can use BGP tricks. But don't they need a sympathetic ISP or similar to help them get the routes in?
- tete 12y ago
  I guess if ISPs would really go sniffing on whether they host them they would probably be able to find out (probably!). But then when you have couple of hundred VPS customer and give a tiny bit about their privacy then as long as you get paid and receive no complaints why would you really go look for them?
- nkcmr 12y ago
  IIRC They have their load balancer hosted under a sovereign IP address (the IP block belongs to a political party). So attempting to mess with it could constitute infringement of free speech.
Theodores 12y ago
> In total the VMs use 182 GB of RAM and 94 CPU cores. The total storage capacity is 620 GB, but that’s not all used. That level of hardware/cores seems a bit over the top given what TPB does. When I was a boy we had this thing called 'Alta Vista'. It was the search engine before Bing! came along. Processors did not run at gigahertz speeds back then and a large disk was 2Gb. Nonetheless most offices had the internet and when people went searching 'Alta Vista' was the first port of call for many. TPB has an index of a selective part of the internets, i.e. movies, software, music, that sort of thing. Meanwhile, back in the 1990's, AltaVista indexed everything, as in the entire known internets, with everything stored away in less than the 620Gb used by TPB for their collection of 'stolen' material. From http://en.wikipedia.org/wiki/AltaVista http://en.wikipedia.org/wiki/AltaVista Alta Vista is a very large project, requiring the cooperation of at least 5 servers, configured for searching huge indices and handling a huge Internet traffic load. The initial hardware configuration for Alta Vista is as follows: Alta Vista -- AlphaStation 250 4/266 4 GB disk 196 MB memory Primary web server for gotcha.com Queries directed to WebIndexer or NewsIndexer NewsServer -- AlphaStation 400 4/233 24 GB of RAID disks 160 MB memory News spool from which news index is generated Serves articles (via http) to those without news server NewsIndexer -- AlphaStation 250 4/266 13 GB disk 196 MB memory Builds news index using articles from NewsServer Answers news index queries from Alta Vista Spider -- DEC 3000 Model 900 (replacement for Model 500) 30 GB of RAID disk 1GB memory Collects pages from the web for WebIndexer WebIndexer -- Alpha Server 8400 5/300 210 GB RAID disk (expandable) 4 GB memory (expandable) 4 processors (expandable) Builds the web index using pages sent by Spider. Answers web index queries from Alta Vista
- lmm 12y ago
  How many pages were there on "the entire internet" back then? How many reqs/second did AltaVista serve? How does that compare to the numbers for TPB?
  - Theodores 12y ago

139 points by tjaerv 12y ago

"At the time of writing the site uses 21 virtual machines (VMs) hosted at different providers. [...] All virtual machines are hosted with commercial cloud hosting providers, who have no clue that The Pirate Bay is among their customers."

crazy1van 12y ago
They may "have no clue" but it seems like that's only because they don't care and haven't looked. I don't see anything in the article that would prevent the providers from figuring this out unless I'm missing something.
- allegory 12y ago
  If someone is paying the bill, do they really care?
  - nols 12y ago
    The first time TPB was raided they took a bunch of unrelated servers too. Not sure how discriminating the police will be next time, regardless of VMs.
  - larrys 12y ago
    "If someone is paying the bill, do they really care?" So you could cross reference names of the people raided with payment information of the VPS providers (usual suspects or top "n" providers let's say). Of course that could be hidden as well. Other issue is how does anyone know this isn't misinformation anyway and that the VPS providers don't play a role or not as much of a role as is indicated. Just because someone is writing this or because they said it? What advantage does it have for anyone (like this) to reveal anything about how they are situated security wise if not to lead people off the beaten track even given some possible marketing advantage?
    - nacs 12y ago
      > payment information of the VPS providers I doubt this will be useful as they're probably using Bitcoin or prepaid cards or things like that for payments. > What advantage does it have for anyone (like this) to reveal anything It could teach others how to setup websites that are harder to censor or more resilient to raids (plus it gets them free PR/traffic).
  - bowlofpetunias 12y ago
    It's not about the money, it's about not being able to care and survive as a business. Because then they would also have to care about the thousands of other VM's that may run all kinds of stuff that is illegal somewhere, questionable, politically, socially, culturally or commercially sensitive etcetera. No ISP can afford to be proactive about this. They cannot afford to care. Or even know.
- adwf 12y ago
  I think only the load balancer would be vulnerable to discovery. Everything behind the load balancer could be a secure connection to a completely different datacenter if needed.
- Wilya 12y ago
  Apart from the external-facing proxy (which is the most exposed link in this setup), these VMs don't need any sort of public presence. Unless the provider inspects processes running on all their customers machines, all they can see is a VM with opaque VPN connections to a few external ips.

TomAnthony 12y ago

If the load balancer is the weak point that would be first to be discovered, then I imagine they must have some mechanism to stop it leaving evidence that leads to the other machines if it were to get raided (it isn't on their hardware, so they can't prevent the files being backed up). Is there a way the codebase could be entirely encrypted and not even accessible to the cloud provider (with some 'boot password' needed each time the server starts up)?

TheLoneWolfling 12y ago
It depends on what you mean by that. The only way to prevent a codebase from being seen by an adversary with physical access when the server is on is to not have the sensitive data on the server in the first place. Encryption (with the decryption key being gotten at boot from, say, a particular .onion address) would work against backups, but won't protect against an adversary with admin access to the server when the virtual server is on.
- aianus 12y ago
  This would be a cool application for some kind of homomorphic encryption. Server gets encrypted search request and matches it against an encrypted index and returns the encrypted results.
  - TheLoneWolfling 12y ago
    Homomorphic encryption isn't fast enough... yet. But it's getting faster! It will be amazing if/when we get to the point where you can have a virtual server where you know that the person with physical access to the server cannot access your data. Even if it is two orders of magnitude slower than raw hardware that's still fast enough for some things. (For example, being able to have a username+password -> personal info database safely run on someone else's hardware.) And once it takes off there probably will start to be hardware support/accelerators for it - like vector intrinsics and AES instructions currently.
  - KMag 12y ago
    The problem with homomorphic encryption in this case is that you need access to the private key in order to interpret the results of the computation. Joe Public encrypts queries using a public key, and the query runs on a possibly compromised server without the attackers learning anything about the database or the query, but then the result of the query looks like absolute gibberish to Joe Public. Giving Joe Public access to the private key necessary for interpreting the query result allows attackers to inspect all of the intermediate states of the query finite state machine, which allows debugging and inspection just as if homomorphic encryption wasn't in use. I suppose the routing proxy could hold the private key and decrypt the query result for the general public. However, the location of the routing proxy is almost certainly going to be compromised before the locations of the servers executing the queries, so in the decrypting proxy scenario, the attackers will almost certainly have the secret keys before they get access to the boxes executing the queries. There's also the problem that the messages being decrypted are the final states of finite state machines that executed the queries, so the messages to be copied over the network add up in size to at least the size of the dataset being queried. (The data can be sharded into many smaller databases, and almost certainly would be in order to speed up the homomorphic computation steps, but this doesn't cut down on the amount of network traffic necessary to retrieve all search results for a single query. A simple query on 1 TB of data, split into 10,000 databases each of 100 MB would require copying and remotely decrypting 10,000 messages, each over 100 MB in size.) It might be possible to discover a homomorphic encryption scheme whereby knowledge of the private key allows one to devise a mapping from a higher dimensional finite state to a lower dimensional finite state machine, where the secret key for the smaller dimensional machine doesn't leak information about the secret key for the larger machine. In this case, it may be possible to perform some finishing operations on the query to prepare it for conversion to the smaller state machine and give the public the private key to the smaller state machine so that the query result could be read from the machine by the public without the public being able to observe intermediate states of the query computation. However, I believe this is far beyond our current mathematical understanding.

fooyc 12y ago

" This saved costs, guaranteed better uptime, and made the site more portable and thus harder to take down " Probably not true for " This saved costs ". From what i've seen, virtual machines usually cost more than twice the price of renting the equivalent "real" machine monthly. They could have used dedicated servers; there are more dedicated server providers than VM providers, thus achieving the same goal, less expensively. Probably not true for " better uptime " either; VMs are still hosted on real hardware, which fails, too. (Although distributing the work on more independent machines can improve uptime.)

drsintoma 12y ago
They are more expensive, but they are usually easy and immediate to acquire. Which makes provisioning much more efficient in case of fluctuating traffic. And overall sysadmins will have less tendency to over-provision, meaning getting more and beefier machines than it's needed "to be safe".
- fooyc 12y ago
  Nothing prevents you from using bare dedicated servers for your usual traffic, and VMs for anything else.
  - TillE 12y ago
    This is clearly the ideal setup for most use cases, and I'm somewhat puzzled as to why it's not more common. I guess using only virtual servers is a tiny bit simpler, so companies will just eat the extra cost.
  - drsintoma 12y ago
    In an ideal world yes. Or if your software works already seamlessly cross-datacenter. But in the real world is rare that your hosting provider is good at both VMs and metal. At least that's the biggest problem I've always encountered, specially with budget providers.
- RyanZAG 12y ago

Nanzikambe 12y ago

Interesting, so I'm presuming there's several VPNs involved between the load-balancer and all the discrete servers. I wonder if they use a VPN provider with a static IP and no-logs policy or if it's simply yet another VPS. I'd love to hear a little more about the architecture.

icedchai 12y ago
Why would they use a VPN provider? It's trivial to set up your own if you control the systems at both ends.
- TheLoneWolfling 12y ago
  Assuming at least one side isn't under a NAT, true. Otherwise it gets... "fun".
  - the_mitsuhiko 12y ago
    A NAT does not stop you if you do it right. Console games have been hosting servers through NATs for the last decade.
    - TheLoneWolfling 12y ago
      As I said: it gets "fun", especially if you want to do it without a third server to set up the connection. Still doable, just "fun".
      - tacotime 12y ago
        I would speculate that the massive advertising revenue generated by the 'bay probably renders any such 'fun' factor trivial.
  - dlgeek 12y ago
    Why would you be under a NAT? A public IP and a networking allotment is one of the core offerings of almost every cloud provider out there.

verroq 12y ago

Why can't people find where their servers? I understand they have their own IP allocation, thus they can use BGP tricks. But don't they need a sympathetic ISP or similar to help them get the routes in?

tete 12y ago
I guess if ISPs would really go sniffing on whether they host them they would probably be able to find out (probably!). But then when you have couple of hundred VPS customer and give a tiny bit about their privacy then as long as you get paid and receive no complaints why would you really go look for them?
nkcmr 12y ago
IIRC They have their load balancer hosted under a sovereign IP address (the IP block belongs to a political party). So attempting to mess with it could constitute infringement of free speech.

Theodores 12y ago

> In total the VMs use 182 GB of RAM and 94 CPU cores. The total storage capacity is 620 GB, but that’s not all used. That level of hardware/cores seems a bit over the top given what TPB does. When I was a boy we had this thing called 'Alta Vista'. It was the search engine before Bing! came along. Processors did not run at gigahertz speeds back then and a large disk was 2Gb. Nonetheless most offices had the internet and when people went searching 'Alta Vista' was the first port of call for many. TPB has an index of a selective part of the internets, i.e. movies, software, music, that sort of thing. Meanwhile, back in the 1990's, AltaVista indexed everything, as in the entire known internets, with everything stored away in less than the 620Gb used by TPB for their collection of 'stolen' material. From http://en.wikipedia.org/wiki/AltaVista http://en.wikipedia.org/wiki/AltaVista Alta Vista is a very large project, requiring the cooperation of at least 5 servers, configured for searching huge indices and handling a huge Internet traffic load. The initial hardware configuration for Alta Vista is as follows: Alta Vista -- AlphaStation 250 4/266 4 GB disk 196 MB memory Primary web server for gotcha.com Queries directed to WebIndexer or NewsIndexer NewsServer -- AlphaStation 400 4/233 24 GB of RAID disks 160 MB memory News spool from which news index is generated Serves articles (via http) to those without news server NewsIndexer -- AlphaStation 250 4/266 13 GB disk 196 MB memory Builds news index using articles from NewsServer Answers news index queries from Alta Vista Spider -- DEC 3000 Model 900 (replacement for Model 500) 30 GB of RAID disk 1GB memory Collects pages from the web for WebIndexer WebIndexer -- Alpha Server 8400 5/300 210 GB RAID disk (expandable) 4 GB memory (expandable) 4 processors (expandable) Builds the web index using pages sent by Spider. Answers web index queries from Alta Vista

lmm 12y ago
How many pages were there on "the entire internet" back then? How many reqs/second did AltaVista serve? How does that compare to the numbers for TPB?
- Theodores 12y ago