8 ms·

1,000,000 daily users with no cache

114 points by Sato 15y ago

iconfinder 15y ago
Why did you go for Ruby instead of PHP? Would you have had the (roughly) same database issues if you didn't have that many writes to the db?
- jrirei 15y ago
  We did go for Ruby in order to increase developer productivity (having a very small team of just two developers), and good code quality/high test coverage. We were sure we needed to refactor a lot later on. So Ruby seems like a good choice. But Ruby is NOT good at waiting no a database / network latency. But I guess with PHP we would have had exactly the same problems.
  - iconfinder 15y ago
    Did you compare bare bones PHP with Ruby or PHP with a library such as Zend? I curious because I'm considering recoding a large part of a website and are trying to avoid scaling issues. Thank you for the answer.
- deleted 15y ago
  [deleted]
rmoriz 15y ago
In another presentation of Wooga they described why they went off-cloud with the later games using cheap dedicated servers from http://hetzner.de/ http://hetzner.de/ Since hetzner recently upgraded the hardware but also limited the different options it would be very interesting to see what Wooga takes out of this...
Udo 15y ago
That was an interesting read. I'd love to watch the actual presentation if it's available somewhere. One thing that struck me as odd from a software design point of view is the "tile table". The name and the absurdly high number of I/Os per second suggest that each tile was stored in a separate database record? The game looks like a Farmville clone, so wouldn't it have been more economical to store the entire farm of a player in one blob? Is there someone from Wooga here to shed some light on the architectural decisions that went into the game?
- bermanoid 15y ago
  I've got to agree - 1M daily users and 50k db hits per second indicates something is being done in a ridiculously inefficient manner somewhere, and I'm really surprised that the solution was to fiddle around on the server-side. Suppose your players are active for an average of, say, 20 minutes - that's a damn generous upper bound for a Facebook game. That would mean that 1M / (24*3) = 13,888 users are active (on average) at any time. Which means that they must each be generating db events at a rate of about 3.6 events / sec / user, which is ludicrous (at the very least, these should probably be somehow combined so that each client only hits the db once in a while, preferably in response to a user action). And in reality, the average play session is probably quite a bit shorter than 20 minutes, which means the event rate is even higher per active user. What the hell are they doing that is causing so much database traffic? Is the game actually doing something a lot more sophisticated than I would assume it is based on the looks of it (to be fair, I've never played it), or is the client-side design just really that messed up? I mean, it's cool that they can handle that much traffic, and all, and the server guys should be proud, but IMO they should really be able to do better on the client-side to prevent this level of scaling from being necessary...
  - joelhaasnoot 15y ago
    Have you played these games? Everything you do on your farm/plot/area is a click on a tile. You're essentially trying to click on as many tiles in as little time as possible for your own sanity: there's no "select all".
    - jules 15y ago
      You can preload/cache all or nearly all data on the client side, and batch updates.
      - jrirei 15y ago
        We do preload all data on client side as the client must never wait on server responses for better user experience. But we cannot batch on client side as we cannot foresee when the user will kill the client. And actually we have more than 40K parallel user sessions as players come back more than once each day.
Sato 15y ago
I just notified @jrirei of this discussion.
gibybo 15y ago
50,000 writes/sec seems insanely high for a flash game, would love to hear more about what those writes are doing.
- snorkel 15y ago
  It's a massive multiplayer game with 1million+ users so seems they're constantly transmitting the state of every active player to their server farm acting the game hub.
fleitz 15y ago
I really want to know what would have happened if they just bought a couple 24 drive arrays and stacked them full of SSDs. 50,000 IOPS per second sounds like it could be handled with a couple gigs of BBWC and a decent drive array. Typically, you want about 200 15K spindles per CPU spread over a couple controllers, jammed full of delicious battery backed RAM.
henrikschroder 15y ago
I'm amazed that some people still do services without using memcached or similar. It's not very difficult, and brings enormous benefits. So, why not?
- judofyr 15y ago
  They used Redis. Which I would say is pretty similar to memcached.
  - henrikschroder 15y ago
    I thought so too, but that means the slide title is wrong, they're not doing that much traffic without a cache.
    - snorkel 15y ago
      They're using Redis as a primary store rather than a cache.
- snorkel 15y ago
  Redis is like memcached on steroids.
tyler 15y ago
"with no cache" is a bit of a misleading statement, considering the entirety of their data set is stored in RAM. Turns out you don't really need memcached if you don't read anything from disk.
- Sato 15y ago
  Not really. Consider their scenario. 100,000 db operations and 50,000 updates per sec. In this case, simple cache costs more than without. Eviction is expensive. Also, it's no surprise replication doesn't scale because Updates get propagated. Not sure details, but they succeeded to relax the tight requirements of ACID transactions. So this is a good case when RDBMS(or traditional database) fails. I guess their design is more like MMO, hope to hear from the guy.
  - tyler 15y ago
    No. Memcached on a reasonable server will do millions of requests per second. 50,000 updates per second is nothing for any modern cache. Also, replication has nothing to do with whether or not "without a cache" is a meaningful statement. The point is that by holding their entire data set in RAM, they've nullified the need for a cache. Effectively, their database is their cache. And considering the data isn't even written to disk for about 15 minutes, it's really more cache than database anyway.
    - Sato 15y ago
      "with no cache" means no cache in app server layer. I guessed that's how the guy uses the words "no cache". Getting rid of cache out of app server layer has great benefit on a cloud. Having cache in app server layer needs synchronizations to keep data consistent. Scale out design was made before cloud era, when we have our own dedicated system on our site. LAN can afford expensive sync communications, but on a cloud? Still makes sense if a scenario is read intensive. But when update intensive? That's why I see this interesting. Increasing memory is a cheap option on a cloud. So it's great if it scales by letting database utilize more memory.
revorad 15y ago
Can any DB veterans offer any insight on how this would scale if they had used PostgreSQL instead of MySQL?
- henrikschroder 15y ago
  At the end of the day, all transactional RDBMSs need to do the same things, and if they're mature and optimized enough, they'll all reach the same physical limits of what the underlying disk can handle. There will be small differences, but all numbers will be in the same magnitude. Switching from MySQL to PostGRES in the above scenario wouldn't do much for performance, some operational tasks might change, complex queries might change in speed, but the baseline of simple updates/selects per second won't really change. To get magnitudes more performance, you need to use a different model. Redis is a NoSQL key-value-store which keeps the entire dataset in memory and occasionally flushes changes to disk. Of course that's going to be faster than a system which flushes all changes to disk individually.
  - teh 15y ago
    > Of course that's going to be faster than a system which flushes all changes to disk individually. Even that is tunable these days. You can have unlogged tables, turn off fsync or have a high commit delay etc.
fgielow 15y ago
How do you people think this would cope with using NoSQL approaches, such as MongoDB, instead of SQL based ones?
srgseg 15y ago
In this Wooga presentation they talk about how DB hosting in the cloud is 20x more expensive than on rented dedicated servers, or 5x more expensive per DAU across the entire variety of servers required. http://www.slideshare.net/wooga/games-for-the-masses-scaling-rails-to-the-extreme http://www.slideshare.net/wooga/games-for-the-masses-scaling...
- deleted 15y ago
  [deleted]

114 points by Sato 15y ago

Why did you go for Ruby instead of PHP? Would you have had the (roughly) same database issues if you didn't have that many writes to the db?

jrirei 15y ago
We did go for Ruby in order to increase developer productivity (having a very small team of just two developers), and good code quality/high test coverage. We were sure we needed to refactor a lot later on. So Ruby seems like a good choice. But Ruby is NOT good at waiting no a database / network latency. But I guess with PHP we would have had exactly the same problems.
- iconfinder 15y ago
  Did you compare bare bones PHP with Ruby or PHP with a library such as Zend? I curious because I'm considering recoding a large part of a website and are trying to avoid scaling issues. Thank you for the answer.
deleted 15y ago
[deleted]

rmoriz 15y ago

In another presentation of Wooga they described why they went off-cloud with the later games using cheap dedicated servers from http://hetzner.de/ http://hetzner.de/ Since hetzner recently upgraded the hardware but also limited the different options it would be very interesting to see what Wooga takes out of this...

Udo 15y ago

That was an interesting read. I'd love to watch the actual presentation if it's available somewhere. One thing that struck me as odd from a software design point of view is the "tile table". The name and the absurdly high number of I/Os per second suggest that each tile was stored in a separate database record? The game looks like a Farmville clone, so wouldn't it have been more economical to store the entire farm of a player in one blob? Is there someone from Wooga here to shed some light on the architectural decisions that went into the game?

bermanoid 15y ago
I've got to agree - 1M daily users and 50k db hits per second indicates something is being done in a ridiculously inefficient manner somewhere, and I'm really surprised that the solution was to fiddle around on the server-side. Suppose your players are active for an average of, say, 20 minutes - that's a damn generous upper bound for a Facebook game. That would mean that 1M / (24*3) = 13,888 users are active (on average) at any time. Which means that they must each be generating db events at a rate of about 3.6 events / sec / user, which is ludicrous (at the very least, these should probably be somehow combined so that each client only hits the db once in a while, preferably in response to a user action). And in reality, the average play session is probably quite a bit shorter than 20 minutes, which means the event rate is even higher per active user. What the hell are they doing that is causing so much database traffic? Is the game actually doing something a lot more sophisticated than I would assume it is based on the looks of it (to be fair, I've never played it), or is the client-side design just really that messed up? I mean, it's cool that they can handle that much traffic, and all, and the server guys should be proud, but IMO they should really be able to do better on the client-side to prevent this level of scaling from being necessary...
- joelhaasnoot 15y ago
  Have you played these games? Everything you do on your farm/plot/area is a click on a tile. You're essentially trying to click on as many tiles in as little time as possible for your own sanity: there's no "select all".
  - jules 15y ago
    You can preload/cache all or nearly all data on the client side, and batch updates.
    - jrirei 15y ago
      We do preload all data on client side as the client must never wait on server responses for better user experience. But we cannot batch on client side as we cannot foresee when the user will kill the client. And actually we have more than 40K parallel user sessions as players come back more than once each day.

Sato 15y ago

I just notified @jrirei of this discussion.

gibybo 15y ago

50,000 writes/sec seems insanely high for a flash game, would love to hear more about what those writes are doing.

snorkel 15y ago
It's a massive multiplayer game with 1million+ users so seems they're constantly transmitting the state of every active player to their server farm acting the game hub.

fleitz 15y ago

I really want to know what would have happened if they just bought a couple 24 drive arrays and stacked them full of SSDs. 50,000 IOPS per second sounds like it could be handled with a couple gigs of BBWC and a decent drive array. Typically, you want about 200 15K spindles per CPU spread over a couple controllers, jammed full of delicious battery backed RAM.

henrikschroder 15y ago

I'm amazed that some people still do services without using memcached or similar. It's not very difficult, and brings enormous benefits. So, why not?

judofyr 15y ago
They used Redis. Which I would say is pretty similar to memcached.
- henrikschroder 15y ago
  I thought so too, but that means the slide title is wrong, they're not doing that much traffic without a cache.
  - snorkel 15y ago
    They're using Redis as a primary store rather than a cache.
snorkel 15y ago
Redis is like memcached on steroids.

tyler 15y ago

"with no cache" is a bit of a misleading statement, considering the entirety of their data set is stored in RAM. Turns out you don't really need memcached if you don't read anything from disk.

Sato 15y ago
Not really. Consider their scenario. 100,000 db operations and 50,000 updates per sec. In this case, simple cache costs more than without. Eviction is expensive. Also, it's no surprise replication doesn't scale because Updates get propagated. Not sure details, but they succeeded to relax the tight requirements of ACID transactions. So this is a good case when RDBMS(or traditional database) fails. I guess their design is more like MMO, hope to hear from the guy.
- tyler 15y ago
  No. Memcached on a reasonable server will do millions of requests per second. 50,000 updates per second is nothing for any modern cache. Also, replication has nothing to do with whether or not "without a cache" is a meaningful statement. The point is that by holding their entire data set in RAM, they've nullified the need for a cache. Effectively, their database is their cache. And considering the data isn't even written to disk for about 15 minutes, it's really more cache than database anyway.
  - Sato 15y ago
    "with no cache" means no cache in app server layer. I guessed that's how the guy uses the words "no cache". Getting rid of cache out of app server layer has great benefit on a cloud. Having cache in app server layer needs synchronizations to keep data consistent. Scale out design was made before cloud era, when we have our own dedicated system on our site. LAN can afford expensive sync communications, but on a cloud? Still makes sense if a scenario is read intensive. But when update intensive? That's why I see this interesting. Increasing memory is a cheap option on a cloud. So it's great if it scales by letting database utilize more memory.

revorad 15y ago

Can any DB veterans offer any insight on how this would scale if they had used PostgreSQL instead of MySQL?

henrikschroder 15y ago
At the end of the day, all transactional RDBMSs need to do the same things, and if they're mature and optimized enough, they'll all reach the same physical limits of what the underlying disk can handle. There will be small differences, but all numbers will be in the same magnitude. Switching from MySQL to PostGRES in the above scenario wouldn't do much for performance, some operational tasks might change, complex queries might change in speed, but the baseline of simple updates/selects per second won't really change. To get magnitudes more performance, you need to use a different model. Redis is a NoSQL key-value-store which keeps the entire dataset in memory and occasionally flushes changes to disk. Of course that's going to be faster than a system which flushes all changes to disk individually.
- teh 15y ago
  > Of course that's going to be faster than a system which flushes all changes to disk individually. Even that is tunable these days. You can have unlogged tables, turn off fsync or have a high commit delay etc.

fgielow 15y ago

How do you people think this would cope with using NoSQL approaches, such as MongoDB, instead of SQL based ones?

srgseg 15y ago

In this Wooga presentation they talk about how DB hosting in the cloud is 20x more expensive than on rented dedicated servers, or 5x more expensive per DAU across the entire variety of servers required. http://www.slideshare.net/wooga/games-for-the-masses-scaling-rails-to-the-extreme http://www.slideshare.net/wooga/games-for-the-masses-scaling...

deleted 15y ago
[deleted]