8 ms·

How the EVE Online Servers Deal with a 3,000 Person Battle

331 points by JGM564 14y ago

stephengillie 14y ago
It's like each solar system is a VM, and they can't move their VMs to a new physical server without disconnecting all clients. And all of their physical servers are at 100% load all of the time? Oh, I guess it's 100% utilization, not 100% load. As in, they don't spin down servers to save power during off-hours I guess. The time dilation is a neat solution to the server load problem, but it's sooo annoying as a player. In beta, it was interesting to watch the entire game desync and grind to a halt, but we could still chat and look around. Interesting, but frustrating.
- incision 14y ago
  >It's like each solar system is a VM, and they can't move their VMs to a new physical server without disconnecting all clients. And all of their physical servers are at 100% load all of the time? I wasn't sure how to interpret that either. I've always been a bit surprised that the the backend systems for these MMOs aren't a bit more flexible. Though, EvE did launch nearly 10 years ago and it has never had a huge number of subscribers. I have no idea whether more recent games have solved these problems.
  - namidark 14y ago
    I can't think of any games that deal with PVP on this scale either.
  - mmanfrin 14y ago
    I wouldn't call over 400,000 players '[not] a huge number'. It's the 2nd or 3rd most popular paid-subscription MMO.
    - incision 14y ago
      The specific, relative number of subscribers is not the point. That said, I wouldn't call something huge when there are comparables (even if it's just one or two) which dwarf it and others which (at some point) exceeded it.
    - flexd 14y ago
      As you can see here http://eve-offline.net/?server=tranquility http://eve-offline.net/?server=tranquility , there is about 44k people online at this point, probably a bit less since most people have more than one account. That seems like a fairly substantial amount of users in a single world. Is there any other game world with this many active users in a single world?
  - Goronmon 14y ago
    I have no idea whether more recent games have solved these problems. Most games solve this problem with completely separate servers and population caps. I know in Guild Wars 2 they actually stop displaying players past a certain number to improve performance. This works fine in calm areas like cities, but in PvP it causes issues with being killed by "invisible" groups that the game fails to load in time. I may be remembering incorrectly, but last I remember reading CCP was basically in uncharted territory on the tech front as far as EVE Online is concerned. No other game developer has even tried to tackle this problem at the level they have, and I can't think of many applications in the world that could comparable in scale and complexity to what the EVE servers have to deal with. Edit: To summarize, even being 10 years old, no game has come close to matching it in this context.
    - xenophanes 14y ago
      Guild Wars 2 is stupid about this. At least on release, they stopped displaying ENEMIES FIRST. So if you're in a big enough group, a couple enemies can walk up and none of you can see them b/c you hit the limit from displaying allies, but they can still see some of you guys and kill you while invisible. If they prioritized displaying just party members (parties only go up to 5 people) and maybe people on your friends list, and then enemies, it would have worked better.
      - saraid216 14y ago
        They supposedly just fixed this in the patch yesterday. As a player who gets owned by unrendered thieves, I'm not convinced. -_- (OTOH, I'm also pretty bad at PvP anyways, so whatever.)
  - outworlder 14y ago
    400k+ active players IS a huge number. Specially considering they are in the same logical server. Other MMO's handle this by having copies of the world and splitting the population - which is the same approach used by Ultima Online back in 1997. So actually, EVE is way ahead of the rest, technology-wise.
    - incision 14y ago
      >So actually, EVE is way ahead of the rest, technology-wise. That doesn't seem like a sensible comparison considering how much the gameplay of EvE differs from the typical MMO. In any case, I'm not making a dig at EvE, rather acknowledging that the game has had a long life and has never been a multi-million subscriber behemoth. Therefore, they might make concessions, or allowed the persistence of previous limitations / decisions in design/infrastructure for lack of resources [1][2]. 1: http://news.ycombinator.com/item?id=5135873 http://news.ycombinator.com/item?id=5135873 2: http://massively.joystiq.com/2008/09/28/eve-evolved-eve-onlines-server-model/ http://massively.joystiq.com/2008/09/28/eve-evolved-eve-onli...
      - Goronmon 14y ago
        That doesn't seem like a sensible comparison considering how much the gameplay of EvE differs from the typical MMO. The "typical" MMO doesn't even try to handle large populations. The use of separate servers, population caps and login queues are how just about every one else deals with congestion problems. Even from your second link there is the quote "Working with IBM, the EVE server cluster is maintained in London and is currently the largest supercomputer employed in the gaming industry." Sure that is from 2008, but newer MMOs are built using the same overall server architecture you saw back in the days of Ultima Online.
      - vacri 14y ago
        400k subscribing users is a great big pile of cash to work with.
- iharris 14y ago
  If memory serves, I believe that they mean that the servers run at 100% capacity, not 100% load ie. CCP does not throttle back system resources during periods of light use.
- cdash 14y ago
  The real problem for them that forces them to use time dilation is the fact that a solar system can only run on one core. They have a few machines with a 4.4 Ghz or greater processor I believe for system that are high load but currently they are extremely limited in what they can do hardware wise.
- alexkus 14y ago
  Two main solutions (given that the article mentions that they have huge machines already):- A) Allow a single solar system to span multiple machines. Very hard, especially if the server software isn't architected for this. Retrofitting this can be nigh on impossible. B) Have a few huge machines that can be used to host scenarios like this and, more importantly, have a way of migrating users over to the huge machine seemlessly. The latter can be done but it's tricky, especially if transferring game state between instances of the server is not simple (I'm not talking about transferring the VM itself with something like vMotion). It comes down to:- 1) Being able to make the bigger machine act as a temporary proxy pushing connections data back to the smaller machine. 2) Having a way of telling clients to make a new connection to the bigger machine and, once that connection is made (and the data is being proxied to the smaller machine) cut the connection to the smaller machine. Users see no loss of service or reconnects at all. 3) Once all clients are now being proxied by the bigger machine; pause and transfer the game state from the smaller machine to the big machine and then continue. Obviously it works best if a chunk of state can be transferred in the background and then the final transfer (and pause) is as short as possible in order to transfer over the bang up to the minute state. Option (A) is always the proverbial "In v2 of the server we'll do it a completely different way..."
  - roc 14y ago
    Which never seems to come around, because the new hardware is so much faster that it can host what were previously problematic server loads without a problem, and you've got a million other things to write. Yet players have a tendency to figure out when places are too overcrowded to be fun. So your old problematic load is almost never representative of how many players wanted to be in that area, but merely how many players were willing to put up with that level of degraded performance. So upon release (or sufficiently close to it to start stress testing, which is conveniently when it's too late to really change architecture) the new limits are quickly hit.
  - saraid216 14y ago
    They've been struggling with the issue of multi-thousand player fights for a while now, and have moved towards both of these solutions but are obviously not quite there yet. For instance, the article actually talks about having said huge machines. There's a way in EVE to inform the GMs about anticipated big fights, at which point they'll do the reinforcement preemptively. In this case, there wasn't such a convenient warning.
rauljara 14y ago
The report of the battle they link to (http://www.pcgamer.com/2013/01/28/eve-online-battle-asakai/ http://www.pcgamer.com/2013/01/28/eve-online-battle-asakai/) is even more fascinating to me. You spend hours and hours of effort in order to get a ship, and one way of thinking about them is as the time spent to earn them compressed into 3d form. Unlike (most) other MMO's, battles actually destroy things. So in that youtube clip, you're watching years and years of effort getting evaporated because of a mis-click. In most of the games I play, the pay off of the grinding is getting to enjoy fighting more. But I could never play EVE, because battles would be so anxiety provoking. There would be no pleasure at all in earning a Titan, because I would be forever terrified of losing it. Of course, all games are kind of there to destroy your time. But for some reason it doesn't feel quite as wasted when you have a virtual thing there that is bad-ass in proportion to your time investment.
- fatbird 14y ago
  That feeling of "My ship! My beautiful, beautiful ship!" is offset by the fact that major alliances tend to have strong reimbursement programs. If your dreadnought goes boom in a fleet op, you get the cost of another hull and gear, and get to draw on corp/alliance stores as well. Otherwise, they'd never get their supercap pilots out of the hanger :)
  - baddox 14y ago
    What would be the point of having all this cool stuff if no one else even gets to see it and you don't get to use it?
    - fatbird 14y ago
      If you spent six months grinding the ISK to buy and outfit a carrier, how willing would you be to take it into a battle where any of a hundred things could destroy it, including server lag or a random disconnection?
      - deleted 14y ago
        [deleted]
dholowiski 14y ago
tl/dr: Sharding and slowing everything down.
- lflux 14y ago
  Eve isn't sharded in the common MMO sense.
  - dholowiski 14y ago
    No, but they talked about disconnecting the battle from the surrounding solar system, and the negative consequences. Essentially sharding.
    - lacksconfidence 14y ago
      Actually they do not disconnect the battle from the solar system the battle is in. What they said is they run many solar systems on one server. During a major battle they migrate those other solar systems to different servers so that the server only has to deal with the one system. So in a sense, yes, each solar system is its own shard.
ChuckMcM 14y ago
This is a nice article. A friend of mine did some research for DARPA/NSF on internet 'crowds.' His research was looking at the question of fractal gatherings, which was basically you were 'near' the people around you in the virtual 3D space and could hear what they were saying, you were adjacent to people who were one space away but could hear them if they 'shouted' and the crowd was at a gathering that was sharing an experience from a presenter who was 'projected'. Basically a virtual concert venue for example where process affinity scheduling took into account where you "were" in the 3D space. The questions are many and as far as I can tell the folks who research this are few and far between. Process migration continues to be a hard thing to do. VMware and others have made progress, virtualized interfaces and peripherals, self contained 'state'. Works 'ok' at the VM level but still doesn't work at all AFAIK at the 'thread' level. World of Warcraft addresses this somewhat by 'instances' which is that under certain circumstances (entering a dungeon for example) you and your party of 5, 10, 25, or 40 people all transition (through a loading screen) into a place with nobody else (except you). They can dedicate machines to host an instance and do process migration at that level. DARPA is interested of course because they would love to have a way of connecting warfighters into a virtual command and control center regardless of where they are physically. Essentially being in two places at once is a big force multiplier.
deleted 14y ago
[deleted]
- lmm 14y ago
  I might be. You don't have any contact information in your profile.
EarthLaunch 14y ago
This was particularly interesting for me. I've already built a scalable MMO server architecture in Erlang, which hosts all players on one world, using server nodes that can be moved between machines. I calculate that my server architecture can handle up to 10,000 moving players in a local area. That's characters, not EVE-style ships (characters are heavier). If anyone is interested in talking about this, helping, or funding me, please get in touch. My server is called Proxima and my game is First Earth (you can find it if you're interested). I've been slowly bootstrapping it for 3 years.
- outworlder 14y ago
  Have you got more details? Links?
  - EarthLaunch 14y ago
    I haven't been public about my process of building the server or its details (I just don't have time), but the game is at: http://firstearthgame.com http://firstearthgame.com Someday I'll make time to blog about the server.
    - bestes 14y ago
      Wow, those screenshots look impressive!
RickHull 14y ago
Fun facts: This fight was between the Honeybadger Coalition (HBC) and the Clusterfuck Coalition (CFC). The core of the Honeybadger Coalition is TEST Alliance, and the core of TEST Alliance is Dreddit, the "original" Reddit-based corp (there are / have been several, some of which predate Dreddit and have since folded). HBC routed the CFC, which has Goonswarm as its core, which is organized around the Something Awful forums. TEST Alliance was nurtured by the CFC for a long time, splitting off only in the past year to forge a more independent identity. As such, the hostilities between these old friends is hard to gauge. At this point, the presumption is that it's all in good fun. But that may change.
- alex_c 14y ago
  Admittedly I haven't followed Eve politics for over a year, but I would be very surprised to see Test and Goons get in a "real" fight (e.g. over sovereignty or revenue-generating moons). They both treat the game in the same way (a strange mix of competence, incompetence, and irreverence) and have too much shared history. I suspect the only reason they're even hostile right now is because they have absolutely no external threat and are bored.
  - Semaphor 14y ago
    Fits with what I read on Reddit when it came up. Was a fun fight and TEST won but no one is really mad at anyone else:)
- creamyhorror 14y ago
  I was looking for information on this too, because last I heard, TEST and Goons were closely-allied buddies, and here they are fighting each other in a relatively large way. Then again, I think Pandemic Legion and Goons used to work together against BoB, but then seemingly broke ties (and now PL+TEST, as part of HBC, have clashed with Goons). This article clarified things for me somewhat: http://themittani.com/features/why-didnt-hbc-and-cfc-go-war http://themittani.com/features/why-didnt-hbc-and-cfc-go-war As far as I understand the article, the CFC and HBC are the two biggest coalitions in the galaxy, and war between them is a risky bet for either in the current conditions. There's not enough incentive for them to do it just yet, and old ties come into play. (Some commenters mentioned technetium moons as an underlying factor - very profitable sources of income which PL stands to lose in a war.) From my outsider point of view, the political situation in the EVE galaxy has matured: the Allies (CFC) beat the Axis (BoB), some residual fighting has gone on, and everyone's trending either towards a Cold War or a United Nations. The ethos of EVE predisposes alliances to conflict, but there's a lot of territory and months of sleepness nights at stake when supercoalitions decide to go to war. And these supercoalitions used to be (at least in part) allied to each other against older enemies. The pendulum might swing again, but this time it won't end as easily as BoB crushed ASCN in 3 months.
plasma 14y ago
Second Life (I may be wrong on the game itself) talked about how it manages game scale across many servers. The entire game is a virtual machine. When a player leaves a server boundary, their player VM state is packaged up and sent to the next world server and then resumed on that server. Players don't even notice they crossed server boundaries, and this was possible because the game can be paused mid flight for that player - pretty neat.
jere 14y ago
[Edit: this comment turned out not to really be about Eve and more about the difficulty of real time MMOs] This blows my mind. I've been thinking about writing a small multiplayer game for a few weeks. And today I was actually doing back of the napkin calculations. The thing that quickly became clear is bandwidth is O(n^2) where n is the number of players in the same location, since you have to share the location and velocity of each player with each other player. Here's an example of a calculation I was using. It's the monthly bandwidth in TB of having a certain amount of players in a shared space (3000 in this case), assuming that the location/orientation/velocity of each player's ship is contained in a measly 48 bytes and you attempt 30fps. (48 * 30 * 3000 * 3000 * 3600 * 24 * 30) / 1e12 = 33592 TB = 33 PB THIRTY THREE PETA BYTES to simulate a single battle for a month. I'm looking around at various VPS providers and the bandwidth they offer. Using the naive calculation above and the cheapest Linode offering and it'd take $3 million/month to support a single one of these ongoing battles. Now, I'm sure the numbers above aren't used in reality. Obviously, you can reduce the updates per second... but not by much. You can't shave much off how much you send. It at least gives you a sense of the orders of magnitude required to support such a thing. Even contemplating supporting 100-1000 people at once is looking very difficult for me to pull off.
- gnarbarian 14y ago
  I think one thing Eve has going for it (and this is only an assumption) is that it wouldn't need to sync each player's position on each frame. It could reduce the load dramatically by only sending the acceleration / orientation data for a ship. So if a ship has set a course and has stopped accelerating it doesn't need to send anything else unless that course is changed.
  - jere 14y ago
    I haven't really played Eve, so I can't say. I'm imagining ships flying around and changing course constantly in a battle, but if someone can give a better description, I'd appreciate it. It'd be further reduced by several players sharing a ship... no idea how common that is. What I'm describing is definitely leaning more towards players that are constantly moving around though. It helps explain to me why in Asheron's Call, for instance, when you had too many people in the same city a "portal storm" arose and people started getting teleported out of the city randomly.
DigitalSea 14y ago
I always wondered how Eve coped with what seems like a massively data heavy game, keeping track of battles and logging data especially for epic battles where alliances are pitting their expensive ships against each other. Pretty insightful article, would love to know a bit more about their development processes and how they go about pushing new changes up. It's such a sensitive game, changing just one thing could throw the whole game into disarray. It's crazy when you think that they not only have to track battles, but they also have to capture every other action and movement within the game; mining, agent missions, the marketplace and all of the other components that rely on accurate and up-to-date data capture and realtime reporting. What makes this all even more crazier is the game has been around for over 10 years and counting. I still play Eve everyday, once you get past the menial tasks, you can have some real fun with this game being a drug runner, miner or just straight up renegade roaming the infinite environment.
Draiken 14y ago
The environment EVE creates in almost every aspect of the game is just fascinating. It is the only game I ever played I actually felt an enormous thrill and adrenaline rush when playing pvp. It doesn't matter if you're pvping alone or with a fleet. The fact that you actually lose/gain on a fight differentiates the game from any other MMO. Events that you may have experienced in-game can be recorded forever and actually impacts the whole game universe. I really hope more companies invest into having one persistent world in their MMOs. Instead we see this segregated servers in which great players/achievements/events are limited to this little server. It ends up lessening the interest of many players, specially older ones. Funny thing is that, normally, when the hit MMO stops being a hit they end up merging lots of servers anyways. Which IMHO end up lessening even more the time people have spent on that game.
- Scramblejams 14y ago
  "The fact that you actually lose/gain on a fight differentiates the game from any other MMO." Could you please expand on this? I'm interested in this point, but I haven't played Eve or other MMOs.
  - guizzy 14y ago
    For instance; I play Guild Wars 2. If I die, I lose a little bit of in-game cash from having to repair my armor after restarting (less cash than I can make in 5 minutes), and will possibly have to have to trek a couple of minutes back to where I was. But back when I played EVE (that's a very long time ago, the game's changed since)... I was playing as a trucker in low-sec, essentially carrying goods in somewhat dangerous territory. I was really lusting for T2 transports (blinged out trucks), and it was my goal to get one. So I trained and saved for months, until I finally could afford it. I'm not a very patient person, so I took it for a ride before it was fully fitted with the modules I needed for my protection. Some pirate jerk blew it up. It was insured, but I wasn't quite able to easily replace the modules I had lost. In one fight, weeks of my in-game time were lost. In other MMOs, winning and losing can mean that you've wasted a few minutes of your time. In EVE, it can mean months of wasted effort.
OGC 14y ago
This alliance has some guy who does simultaneous translation to russian?
clarky07 14y ago
I haven't played EVE in years, but I always love reading these reports to reminisce. It took a huge time investment but it was a lot of fun. Reading that CFC apparently has hundreds of titans and lost 3 just in this battle is crazy. I was playing when the first Titan was built, and I remember distinctly when the first one was destroyed.
warmwaffles 14y ago
And of course it's a video of the former Goonswarm[GOONS] now [GEWNS]
coldarchon 14y ago
Someone who plays EVE explain this to me. The game becomes laggy and the time gets dilated the more people "warp" in. But when it's so laggy that you disconnect you have to "warp" in again to get back into the fight. Isn't that a vicious circle?
mseebach 14y ago
On moving the game to a beefier server: “This is the machine that systems get reinforced on when players request that. Unfortunately, the same thing above applies – anyone in the system when the move happens gets disconnected. Because of this, it’s basically never done,” It seems like a fairly small addition to the game logic to allow the game to freeze for a few seconds and be forcibly moved to another server. Is this kind of scenario so rare that it's not worth the trouble, or am I missing something else?
- CJefferson 14y ago
  The problem (I believe) is that they don't have a way to 'freeze' and transfer a running game. Think of it like any program -- it is easy to "save your work" and load the file on another computer. It's much harder to exactly save the state of all parts of a running program and transfer that to another computer. Each user is represented by a stackless python program, these are hard to shut down and move. As you say, it could probably be done, but it would be a lot of work, which wouldn't get used very often.
  - archon 14y ago
    The other problem is that they would have to 'pause' the entire game while the physical move was in progress, otherwise players would still be able to move around in the rest of the game world (other than the cluster of stars they took offline to move them). As I understand it, this is already sort of a problem with the Time Dilation mechanic they currently use; time moves at 10% normal within the fight, giving people outside the slowed-down star systems time to deploy forces at 10x speed relative to the fight.
Syssiphus 14y ago
Aww, and now I want to start playing EVE again :). Great article, great battle.

331 points by JGM564 14y ago

It's like each solar system is a VM, and they can't move their VMs to a new physical server without disconnecting all clients. And all of their physical servers are at 100% load all of the time? Oh, I guess it's 100% utilization, not 100% load. As in, they don't spin down servers to save power during off-hours I guess. The time dilation is a neat solution to the server load problem, but it's sooo annoying as a player. In beta, it was interesting to watch the entire game desync and grind to a halt, but we could still chat and look around. Interesting, but frustrating.

incision 14y ago
>It's like each solar system is a VM, and they can't move their VMs to a new physical server without disconnecting all clients. And all of their physical servers are at 100% load all of the time? I wasn't sure how to interpret that either. I've always been a bit surprised that the the backend systems for these MMOs aren't a bit more flexible. Though, EvE did launch nearly 10 years ago and it has never had a huge number of subscribers. I have no idea whether more recent games have solved these problems.
- namidark 14y ago
  I can't think of any games that deal with PVP on this scale either.
- mmanfrin 14y ago
  I wouldn't call over 400,000 players '[not] a huge number'. It's the 2nd or 3rd most popular paid-subscription MMO.
  - incision 14y ago
    The specific, relative number of subscribers is not the point. That said, I wouldn't call something huge when there are comparables (even if it's just one or two) which dwarf it and others which (at some point) exceeded it.
  - flexd 14y ago
    As you can see here http://eve-offline.net/?server=tranquility http://eve-offline.net/?server=tranquility , there is about 44k people online at this point, probably a bit less since most people have more than one account. That seems like a fairly substantial amount of users in a single world. Is there any other game world with this many active users in a single world?
- Goronmon 14y ago
  I have no idea whether more recent games have solved these problems. Most games solve this problem with completely separate servers and population caps. I know in Guild Wars 2 they actually stop displaying players past a certain number to improve performance. This works fine in calm areas like cities, but in PvP it causes issues with being killed by "invisible" groups that the game fails to load in time. I may be remembering incorrectly, but last I remember reading CCP was basically in uncharted territory on the tech front as far as EVE Online is concerned. No other game developer has even tried to tackle this problem at the level they have, and I can't think of many applications in the world that could comparable in scale and complexity to what the EVE servers have to deal with. Edit: To summarize, even being 10 years old, no game has come close to matching it in this context.
  - xenophanes 14y ago
    Guild Wars 2 is stupid about this. At least on release, they stopped displaying ENEMIES FIRST. So if you're in a big enough group, a couple enemies can walk up and none of you can see them b/c you hit the limit from displaying allies, but they can still see some of you guys and kill you while invisible. If they prioritized displaying just party members (parties only go up to 5 people) and maybe people on your friends list, and then enemies, it would have worked better.
    - saraid216 14y ago
      They supposedly just fixed this in the patch yesterday. As a player who gets owned by unrendered thieves, I'm not convinced. -_- (OTOH, I'm also pretty bad at PvP anyways, so whatever.)
- outworlder 14y ago
  400k+ active players IS a huge number. Specially considering they are in the same logical server. Other MMO's handle this by having copies of the world and splitting the population - which is the same approach used by Ultima Online back in 1997. So actually, EVE is way ahead of the rest, technology-wise.
  - incision 14y ago
    >So actually, EVE is way ahead of the rest, technology-wise. That doesn't seem like a sensible comparison considering how much the gameplay of EvE differs from the typical MMO. In any case, I'm not making a dig at EvE, rather acknowledging that the game has had a long life and has never been a multi-million subscriber behemoth. Therefore, they might make concessions, or allowed the persistence of previous limitations / decisions in design/infrastructure for lack of resources [1][2]. 1: http://news.ycombinator.com/item?id=5135873 http://news.ycombinator.com/item?id=5135873 2: http://massively.joystiq.com/2008/09/28/eve-evolved-eve-onlines-server-model/ http://massively.joystiq.com/2008/09/28/eve-evolved-eve-onli...
    - Goronmon 14y ago
      That doesn't seem like a sensible comparison considering how much the gameplay of EvE differs from the typical MMO. The "typical" MMO doesn't even try to handle large populations. The use of separate servers, population caps and login queues are how just about every one else deals with congestion problems. Even from your second link there is the quote "Working with IBM, the EVE server cluster is maintained in London and is currently the largest supercomputer employed in the gaming industry." Sure that is from 2008, but newer MMOs are built using the same overall server architecture you saw back in the days of Ultima Online.
    - vacri 14y ago
      400k subscribing users is a great big pile of cash to work with.
iharris 14y ago
If memory serves, I believe that they mean that the servers run at 100% capacity, not 100% load ie. CCP does not throttle back system resources during periods of light use.
cdash 14y ago
The real problem for them that forces them to use time dilation is the fact that a solar system can only run on one core. They have a few machines with a 4.4 Ghz or greater processor I believe for system that are high load but currently they are extremely limited in what they can do hardware wise.
alexkus 14y ago
Two main solutions (given that the article mentions that they have huge machines already):- A) Allow a single solar system to span multiple machines. Very hard, especially if the server software isn't architected for this. Retrofitting this can be nigh on impossible. B) Have a few huge machines that can be used to host scenarios like this and, more importantly, have a way of migrating users over to the huge machine seemlessly. The latter can be done but it's tricky, especially if transferring game state between instances of the server is not simple (I'm not talking about transferring the VM itself with something like vMotion). It comes down to:- 1) Being able to make the bigger machine act as a temporary proxy pushing connections data back to the smaller machine. 2) Having a way of telling clients to make a new connection to the bigger machine and, once that connection is made (and the data is being proxied to the smaller machine) cut the connection to the smaller machine. Users see no loss of service or reconnects at all. 3) Once all clients are now being proxied by the bigger machine; pause and transfer the game state from the smaller machine to the big machine and then continue. Obviously it works best if a chunk of state can be transferred in the background and then the final transfer (and pause) is as short as possible in order to transfer over the bang up to the minute state. Option (A) is always the proverbial "In v2 of the server we'll do it a completely different way..."
- roc 14y ago
  Which never seems to come around, because the new hardware is so much faster that it can host what were previously problematic server loads without a problem, and you've got a million other things to write. Yet players have a tendency to figure out when places are too overcrowded to be fun. So your old problematic load is almost never representative of how many players wanted to be in that area, but merely how many players were willing to put up with that level of degraded performance. So upon release (or sufficiently close to it to start stress testing, which is conveniently when it's too late to really change architecture) the new limits are quickly hit.
- saraid216 14y ago
  They've been struggling with the issue of multi-thousand player fights for a while now, and have moved towards both of these solutions but are obviously not quite there yet. For instance, the article actually talks about having said huge machines. There's a way in EVE to inform the GMs about anticipated big fights, at which point they'll do the reinforcement preemptively. In this case, there wasn't such a convenient warning.

rauljara 14y ago

The report of the battle they link to (http://www.pcgamer.com/2013/01/28/eve-online-battle-asakai/ http://www.pcgamer.com/2013/01/28/eve-online-battle-asakai/) is even more fascinating to me. You spend hours and hours of effort in order to get a ship, and one way of thinking about them is as the time spent to earn them compressed into 3d form. Unlike (most) other MMO's, battles actually destroy things. So in that youtube clip, you're watching years and years of effort getting evaporated because of a mis-click. In most of the games I play, the pay off of the grinding is getting to enjoy fighting more. But I could never play EVE, because battles would be so anxiety provoking. There would be no pleasure at all in earning a Titan, because I would be forever terrified of losing it. Of course, all games are kind of there to destroy your time. But for some reason it doesn't feel quite as wasted when you have a virtual thing there that is bad-ass in proportion to your time investment.

fatbird 14y ago
That feeling of "My ship! My beautiful, beautiful ship!" is offset by the fact that major alliances tend to have strong reimbursement programs. If your dreadnought goes boom in a fleet op, you get the cost of another hull and gear, and get to draw on corp/alliance stores as well. Otherwise, they'd never get their supercap pilots out of the hanger :)
- baddox 14y ago
  What would be the point of having all this cool stuff if no one else even gets to see it and you don't get to use it?
  - fatbird 14y ago
    If you spent six months grinding the ISK to buy and outfit a carrier, how willing would you be to take it into a battle where any of a hundred things could destroy it, including server lag or a random disconnection?
    - deleted 14y ago
      [deleted]

dholowiski 14y ago

tl/dr: Sharding and slowing everything down.

lflux 14y ago
Eve isn't sharded in the common MMO sense.
- dholowiski 14y ago
  No, but they talked about disconnecting the battle from the surrounding solar system, and the negative consequences. Essentially sharding.
  - lacksconfidence 14y ago
    Actually they do not disconnect the battle from the solar system the battle is in. What they said is they run many solar systems on one server. During a major battle they migrate those other solar systems to different servers so that the server only has to deal with the one system. So in a sense, yes, each solar system is its own shard.

ChuckMcM 14y ago

This is a nice article. A friend of mine did some research for DARPA/NSF on internet 'crowds.' His research was looking at the question of fractal gatherings, which was basically you were 'near' the people around you in the virtual 3D space and could hear what they were saying, you were adjacent to people who were one space away but could hear them if they 'shouted' and the crowd was at a gathering that was sharing an experience from a presenter who was 'projected'. Basically a virtual concert venue for example where process affinity scheduling took into account where you "were" in the 3D space. The questions are many and as far as I can tell the folks who research this are few and far between. Process migration continues to be a hard thing to do. VMware and others have made progress, virtualized interfaces and peripherals, self contained 'state'. Works 'ok' at the VM level but still doesn't work at all AFAIK at the 'thread' level. World of Warcraft addresses this somewhat by 'instances' which is that under certain circumstances (entering a dungeon for example) you and your party of 5, 10, 25, or 40 people all transition (through a loading screen) into a place with nobody else (except you). They can dedicate machines to host an instance and do process migration at that level. DARPA is interested of course because they would love to have a way of connecting warfighters into a virtual command and control center regardless of where they are physically. Essentially being in two places at once is a big force multiplier.

deleted 14y ago

[deleted]

lmm 14y ago
I might be. You don't have any contact information in your profile.

EarthLaunch 14y ago

This was particularly interesting for me. I've already built a scalable MMO server architecture in Erlang, which hosts all players on one world, using server nodes that can be moved between machines. I calculate that my server architecture can handle up to 10,000 moving players in a local area. That's characters, not EVE-style ships (characters are heavier). If anyone is interested in talking about this, helping, or funding me, please get in touch. My server is called Proxima and my game is First Earth (you can find it if you're interested). I've been slowly bootstrapping it for 3 years.

outworlder 14y ago
Have you got more details? Links?
- EarthLaunch 14y ago
  I haven't been public about my process of building the server or its details (I just don't have time), but the game is at: http://firstearthgame.com http://firstearthgame.com Someday I'll make time to blog about the server.
  - bestes 14y ago
    Wow, those screenshots look impressive!

RickHull 14y ago

Fun facts: This fight was between the Honeybadger Coalition (HBC) and the Clusterfuck Coalition (CFC). The core of the Honeybadger Coalition is TEST Alliance, and the core of TEST Alliance is Dreddit, the "original" Reddit-based corp (there are / have been several, some of which predate Dreddit and have since folded). HBC routed the CFC, which has Goonswarm as its core, which is organized around the Something Awful forums. TEST Alliance was nurtured by the CFC for a long time, splitting off only in the past year to forge a more independent identity. As such, the hostilities between these old friends is hard to gauge. At this point, the presumption is that it's all in good fun. But that may change.

alex_c 14y ago
Admittedly I haven't followed Eve politics for over a year, but I would be very surprised to see Test and Goons get in a "real" fight (e.g. over sovereignty or revenue-generating moons). They both treat the game in the same way (a strange mix of competence, incompetence, and irreverence) and have too much shared history. I suspect the only reason they're even hostile right now is because they have absolutely no external threat and are bored.
- Semaphor 14y ago
  Fits with what I read on Reddit when it came up. Was a fun fight and TEST won but no one is really mad at anyone else:)
creamyhorror 14y ago
I was looking for information on this too, because last I heard, TEST and Goons were closely-allied buddies, and here they are fighting each other in a relatively large way. Then again, I think Pandemic Legion and Goons used to work together against BoB, but then seemingly broke ties (and now PL+TEST, as part of HBC, have clashed with Goons). This article clarified things for me somewhat: http://themittani.com/features/why-didnt-hbc-and-cfc-go-war http://themittani.com/features/why-didnt-hbc-and-cfc-go-war As far as I understand the article, the CFC and HBC are the two biggest coalitions in the galaxy, and war between them is a risky bet for either in the current conditions. There's not enough incentive for them to do it just yet, and old ties come into play. (Some commenters mentioned technetium moons as an underlying factor - very profitable sources of income which PL stands to lose in a war.) From my outsider point of view, the political situation in the EVE galaxy has matured: the Allies (CFC) beat the Axis (BoB), some residual fighting has gone on, and everyone's trending either towards a Cold War or a United Nations. The ethos of EVE predisposes alliances to conflict, but there's a lot of territory and months of sleepness nights at stake when supercoalitions decide to go to war. And these supercoalitions used to be (at least in part) allied to each other against older enemies. The pendulum might swing again, but this time it won't end as easily as BoB crushed ASCN in 3 months.

plasma 14y ago

Second Life (I may be wrong on the game itself) talked about how it manages game scale across many servers. The entire game is a virtual machine. When a player leaves a server boundary, their player VM state is packaged up and sent to the next world server and then resumed on that server. Players don't even notice they crossed server boundaries, and this was possible because the game can be paused mid flight for that player - pretty neat.

jere 14y ago

[Edit: this comment turned out not to really be about Eve and more about the difficulty of real time MMOs] This blows my mind. I've been thinking about writing a small multiplayer game for a few weeks. And today I was actually doing back of the napkin calculations. The thing that quickly became clear is bandwidth is O(n^2) where n is the number of players in the same location, since you have to share the location and velocity of each player with each other player. Here's an example of a calculation I was using. It's the monthly bandwidth in TB of having a certain amount of players in a shared space (3000 in this case), assuming that the location/orientation/velocity of each player's ship is contained in a measly 48 bytes and you attempt 30fps. (48 * 30 * 3000 * 3000 * 3600 * 24 * 30) / 1e12 = 33592 TB = 33 PB THIRTY THREE PETA BYTES to simulate a single battle for a month. I'm looking around at various VPS providers and the bandwidth they offer. Using the naive calculation above and the cheapest Linode offering and it'd take $3 million/month to support a single one of these ongoing battles. Now, I'm sure the numbers above aren't used in reality. Obviously, you can reduce the updates per second... but not by much. You can't shave much off how much you send. It at least gives you a sense of the orders of magnitude required to support such a thing. Even contemplating supporting 100-1000 people at once is looking very difficult for me to pull off.

gnarbarian 14y ago
I think one thing Eve has going for it (and this is only an assumption) is that it wouldn't need to sync each player's position on each frame. It could reduce the load dramatically by only sending the acceleration / orientation data for a ship. So if a ship has set a course and has stopped accelerating it doesn't need to send anything else unless that course is changed.
- jere 14y ago
  I haven't really played Eve, so I can't say. I'm imagining ships flying around and changing course constantly in a battle, but if someone can give a better description, I'd appreciate it. It'd be further reduced by several players sharing a ship... no idea how common that is. What I'm describing is definitely leaning more towards players that are constantly moving around though. It helps explain to me why in Asheron's Call, for instance, when you had too many people in the same city a "portal storm" arose and people started getting teleported out of the city randomly.

DigitalSea 14y ago

I always wondered how Eve coped with what seems like a massively data heavy game, keeping track of battles and logging data especially for epic battles where alliances are pitting their expensive ships against each other. Pretty insightful article, would love to know a bit more about their development processes and how they go about pushing new changes up. It's such a sensitive game, changing just one thing could throw the whole game into disarray. It's crazy when you think that they not only have to track battles, but they also have to capture every other action and movement within the game; mining, agent missions, the marketplace and all of the other components that rely on accurate and up-to-date data capture and realtime reporting. What makes this all even more crazier is the game has been around for over 10 years and counting. I still play Eve everyday, once you get past the menial tasks, you can have some real fun with this game being a drug runner, miner or just straight up renegade roaming the infinite environment.

Draiken 14y ago

The environment EVE creates in almost every aspect of the game is just fascinating. It is the only game I ever played I actually felt an enormous thrill and adrenaline rush when playing pvp. It doesn't matter if you're pvping alone or with a fleet. The fact that you actually lose/gain on a fight differentiates the game from any other MMO. Events that you may have experienced in-game can be recorded forever and actually impacts the whole game universe. I really hope more companies invest into having one persistent world in their MMOs. Instead we see this segregated servers in which great players/achievements/events are limited to this little server. It ends up lessening the interest of many players, specially older ones. Funny thing is that, normally, when the hit MMO stops being a hit they end up merging lots of servers anyways. Which IMHO end up lessening even more the time people have spent on that game.

Scramblejams 14y ago
"The fact that you actually lose/gain on a fight differentiates the game from any other MMO." Could you please expand on this? I'm interested in this point, but I haven't played Eve or other MMOs.
- guizzy 14y ago
  For instance; I play Guild Wars 2. If I die, I lose a little bit of in-game cash from having to repair my armor after restarting (less cash than I can make in 5 minutes), and will possibly have to have to trek a couple of minutes back to where I was. But back when I played EVE (that's a very long time ago, the game's changed since)... I was playing as a trucker in low-sec, essentially carrying goods in somewhat dangerous territory. I was really lusting for T2 transports (blinged out trucks), and it was my goal to get one. So I trained and saved for months, until I finally could afford it. I'm not a very patient person, so I took it for a ride before it was fully fitted with the modules I needed for my protection. Some pirate jerk blew it up. It was insured, but I wasn't quite able to easily replace the modules I had lost. In one fight, weeks of my in-game time were lost. In other MMOs, winning and losing can mean that you've wasted a few minutes of your time. In EVE, it can mean months of wasted effort.

OGC 14y ago

This alliance has some guy who does simultaneous translation to russian?

clarky07 14y ago

I haven't played EVE in years, but I always love reading these reports to reminisce. It took a huge time investment but it was a lot of fun. Reading that CFC apparently has hundreds of titans and lost 3 just in this battle is crazy. I was playing when the first Titan was built, and I remember distinctly when the first one was destroyed.

warmwaffles 14y ago

And of course it's a video of the former Goonswarm[GOONS] now [GEWNS]

coldarchon 14y ago

Someone who plays EVE explain this to me. The game becomes laggy and the time gets dilated the more people "warp" in. But when it's so laggy that you disconnect you have to "warp" in again to get back into the fight. Isn't that a vicious circle?

mseebach 14y ago

On moving the game to a beefier server: “This is the machine that systems get reinforced on when players request that. Unfortunately, the same thing above applies – anyone in the system when the move happens gets disconnected. Because of this, it’s basically never done,” It seems like a fairly small addition to the game logic to allow the game to freeze for a few seconds and be forcibly moved to another server. Is this kind of scenario so rare that it's not worth the trouble, or am I missing something else?

CJefferson 14y ago
The problem (I believe) is that they don't have a way to 'freeze' and transfer a running game. Think of it like any program -- it is easy to "save your work" and load the file on another computer. It's much harder to exactly save the state of all parts of a running program and transfer that to another computer. Each user is represented by a stackless python program, these are hard to shut down and move. As you say, it could probably be done, but it would be a lot of work, which wouldn't get used very often.
- archon 14y ago
  The other problem is that they would have to 'pause' the entire game while the physical move was in progress, otherwise players would still be able to move around in the rest of the game world (other than the cluster of stars they took offline to move them). As I understand it, this is already sort of a problem with the Time Dilation mechanic they currently use; time moves at 10% normal within the fight, giving people outside the slowed-down star systems time to deploy forces at 10x speed relative to the fight.

Syssiphus 14y ago

Aww, and now I want to start playing EVE again :). Great article, great battle.