125 ms·

CPU Usage Differences After Applying Meltdown Patch at Epic Games

• contrarian_ 9y ago
  Pretty much what I predicted here: https://news.ycombinator.com/item?id=16054674 https://news.ycombinator.com/item?id=16054674 > Sounds like servers handling lots of small UDP packets would be hit pretty hard.
  • _wmd 9y ago
    There's potential for a little rearchitecting to help, at least in the case of UDP: NAME sendmmsg - send multiple messages on a socket SYNOPSIS #define _GNU_SOURCE /* See feature_test_macros(7) */ #include <sys/socket.h> int sendmmsg(int sockfd, struct mmsghdr *msgvec, unsigned int vlen, unsigned int flags);
    • contrarian_ 9y ago
      Yep, I was actually debating whether I'd mention sendmmsg/recvmmsg in my original post but I left it out. Definitely an option for UDP, but you're out of luck if your game server uses TCP (surprisingly many do) because you have to recv from each socket separately.
      • _wmd 9y ago
        There may still be some options depending on the structure of your server and where the added CPU load is hurting most, for example, shunting IO to a thread/threadpool where futex() calls (if necessary) only occur for every N IO requests, rather than pay the syscall price for every IO on the main thread. But that might introduce new latency/ordering problems all of its own
      • dogma1138 9y ago
        The majority of online games use TCP these days with good prediction it’s more reliable than UDP.
        • Demiurge 9y ago
          Latency is more important than reliability for online gaming because the world state instantly gets stale. Instead of retransmit you want latest snapshot. I'd be curious to find out in what online games that is not the case.
          • dogma1138 9y ago
            Latency isn’t an issue today what so ever, it’s not like UDP also has a magical lower latency it had in the past because of smaller packets and slower computers but today? Online games work today with fixed ticks and polling usually at half of full frame rate which means that the server updates and polls the client 30 or 60 times a second or any other even multiplier of the expected synced frame rate.
            • bschwindHN 9y ago
              I'm calling bullshit. TCP has to retransmit lost packets whereas UDP can keep on going without waiting for transmission. Ephemeral data like controller input or past game states can be ignored because that time has passed, while TCP is still trying its best to get the packets there in order and reliably. Latency is still absolutely an issue. I play games from Japan with my friends in the states and I often have a ping of 140 ms or so. That is latency, and properly implemented games (Rocket League, for example) will deal with it using UDP among other techniques. Slower paced games can still use TCP though because latency issues are less sensitive.
            • Matheus28 9y ago
              >This fixed and predictable rate pretty much means that UDP is near pointless and games that still don’t have a predictable tick rate and use UDP tend to be a rubber banding lag fest. You're wrong. Head of line blocking is a real thing that happens very often in TCP. Edit: parent poster removed that part of the comment between me reading and submitting a reply
    • jacquesm 9y ago
      That only works if messages are independent of answers received and are all known at the same point in time. In most games this typically would not be the case, you'd use a message to cram as much state change into it as is known to keep the game moving fluidly. Packing more than one such message together would serve no purpose.
      • _wmd 9y ago
        I'd have presumed otherwise, but I'm not sure if you're understanding the API correctly.. it's not about sending multiple messages to the same destination, but to multiple destinations in a single call. The msg_hdr struct has room for specifying the target address. From userspace' perspective, even if the same data isn't being broadcast at every client, just building up a big array (perhaps while looping over the input from recvmmsg()!) and spitting it out once would have the same semantics as just calling sendmsg() immediately on each, etc
        • jacquesm 9y ago
          Yes, I understand the API correctly. Having implemented it once I think I have the basics down ;) But that said I was assuming that this would be in the context of multiple UDP messages sent from a game client to a game server.
          • Doxin 9y ago
            The bottleneck generally isn't at the client side for games, The server has much more network traffic to handle. So even if this performance fix only works server-side that might be enough.
        • deleted 9y ago
          [deleted]
      • EgoIncarnate 9y ago
        It's pretty common for FPS server game loops to read all the network packets, update player state, run one tick of game logic and physics for all users in a single game, and then send out updates to everyone.
    • alfanick 9y ago
      Or use userspace networking stack (with cards which allow this) to reduce number of syscalls.
      • gpderetta 9y ago
        Especially for UDP this would work very well (as there isn't much of a network stack anyway).
      • ckocagil 9y ago
        And solve the C10M problem at the same time. Two birds with one stone.
    • zzzcpan 9y ago
      If they are using socket API for UDP, performance is not critical for them. Otherwise porting UDP servers to DPDK/netmap is not rocket science and gets you like an order of magnitude better performance.
      • snuxoll 9y ago
        A shooter running at 30Hz (high side for free developer sponsored servers) with even 100 players is only going to be processing at most 10-15K packets per second, and that’s assuming 5 packets per tick per player. Server update rates are usually lower than the internal physics and game logic tick rate as well, so it’s doubtful to even be that high. These aren’t stats of the gameplay servers, these are the backend servers that handle matchmaking, player stats, inventories and progression.
    • forgotpw2018 9y ago
      You could use it, but you would probably have to reachitect the server to use green threads to avoid the overhead. Frequent syscall sends are done with games to keep the latency as low as possible. Any batching would increase delay
  • tgb 9y ago
    Since they describe this as log-in issues, should we expect that to be a server using lots of UDP? Or do you think that the log-in service is hemmed in by the load on the game servers?
    • dogma1138 9y ago
      No, login in Fortnite is over 443 and it uses normal HTTPS. It also uses HTTPs to load a lot of other data such as server data, friends list, chat etc. Unreal Engine comes with a version of chromium built in which is used for many in game things like social tabs, news, and in game purchases these all work over HTTP/S. Game data is sent over 5222 TCP.
      • jbg_ 9y ago
        Out of interest, do you know if the game data uses XMPP or if it just uses the same port?
        • one_thawt 9y ago
          It does not. It's all based on the UE4 network stack.
        • dogma1138 9y ago
          5222 is pretty much used for all Epyc games I don’t think that I has anything to do with XMMP tho.
      • tgb 9y ago
        Right, that's what I was guessing and am not sure why I was downvoted when everyone is confirming that contrarian's post doesn't really make sense since UDP isn't very relevant here.
  • dogma1138 9y ago
    Fortnite doesn’t use UDP at all (80, 443 and 5222 all TCP), UE4 uses TCP for its network stack by default.
    • kevindqc 9y ago
      Are you sure? I would be extremely surprised if it wouldn't use UDP. For things like getting stats, probably from a HTTP endpoint, sure, but for gameplay? The lag would be very bad, no? Lose a packet and everything is slowed down I see this, which indicates it uses UDP: https://imgur.com/al6KTwT https://imgur.com/al6KTwT and according to wireshark it's used heavily when in a game, so I assume that's the gameplay protocol. Also when I left my game (but stayed in the lobby), immediately the port 61879 stopped listening. I'm not sure about UE4, but previous versions of the unreal engine used UDP for replication and RPC.
      • dijit 9y ago
        It's not entirely uncommon to have TCP on the backend even for AAA games. I worked on a huge open world third-person shooter always-online AAA game and it uses TCP for everything.
        • kevindqc 9y ago
          Interesting. I know World of Warcraft uses TCP, but I imagine it's less "real-time" than shooters (ie: no hitscan) so a few dropped packets wouldn't have a huge impact (ie. if you're standing casting a spell for 2sec, the game can recover the lag easily). Didn't know some shooters used TCP
    • vvanders 9y ago
      Nope, it might use TCP for negotiation of things that aren't time sensitive but it uses UDP for replication[1] as has pretty much every Unreal or Quake based engine since they were first developed. I've worked on a variety of engines which were either UE or Quake based. All of them use UDP for temporal game state updates to avoid head of line blocking[2][3]. [1] https://answers.unrealengine.com/questions/197713/is-replication-and-reliability-made-with-udp-or-tc.html https://answers.unrealengine.com/questions/197713/is-replica... [2] https://stackoverflow.com/questions/39323556/why-do-game-developers-avoid-tcp-and-make-udp-reliable-in-the-application-level https://stackoverflow.com/questions/39323556/why-do-game-dev... [3] https://www.gamasutra.com/view/feature/131781/the_internet_sucks_or_what_i_.php?print=1 https://www.gamasutra.com/view/feature/131781/the_internet_s...
  • mtgx 9y ago
    I wonder if this means Google's QUIC protocol is dead in the water, since it works over UDP.
    • dsign 9y ago
      You can basically have bigger UDP packets, that will get fragmented and may not make it to their end. As far as I know, in Linux there is no support for multi-packet UDP vectored I/O. I wonder if it would be possible to "simulate" that with a raw socket....
  • Thaxll 9y ago
    Gameservers don't handle that many packets because they're limited in the number of player they host. A game a 60hz with 64 players will only receive 3840 packets/sec.
    • johnbellone 9y ago
      I don’t know the game or the engine, but if this is a login service, this is likely traffic from all of the players.
    • abstractbeliefs 9y ago
      And then it has to communicate new state to those 64 players 60 times a second for another 3840 reaching 7680 packets per second.
    • dxhdr 9y ago
      And then how many game servers can you run on a machine? Hopefully a lot more than just one.
• aceoflolo 9y ago
  >We wanted to provide a bit more context for the most recent login issues and service instability. All of our cloud services are affected by updates required to mitigate the Meltdown vulnerability. We heavily rely on cloud services to run our back-end and we may experience further service issues due to ongoing updates. So they are saying "you can't log in because we're too cheap to rent more servers now that this patch has increased CPU usage"
  • transpostmeta 9y ago
    Resolving scalability issues is not as simple as "rent more servers".
    • TrickyRick 9y ago
      Renting beefier servers may also be an option if the performance impact scales linearly with pre-patch performance. However it's not clear in this case how that works.
      • bhouston 9y ago
        Depends if the bottleneck is a single CPU or whether it is a single machine. Those beefier machines are usually slower per CPU. I've seen some crazy game server designs before in the quest to have fast response times. But UE4 is pretty professional so even if they are tied to a single machine, they probably are not tied to a single core for some of their critical algorithms I would hope.
    • aceoflolo 9y ago
      For a CPU-bound problem it mostly is as simple as that, yes
      • bhouston 9y ago
        Only if the algorithm can be split across multiple machines easily. Sometimes people assume that you can use local shared memory or something between threads in order to synchronize state. You figure out how many individuals can be on a server at once and then ensure that you can handle that load on a specific machine. I've seen this type of stuff for game state before because they need to keep everyone in a specific game domain (level or city depending on the type of game) synchronized and be nearly real-time. It can be hard to pull this across different machines without introducing significantly latency, redis or DBs is slow for a FPS shootter. Not saying this is the case but I can see it could be something like that.
• simooooo 9y ago
  Wow that's a huge jump. Scared for my web servers now
  • lsd5you 9y ago
    Not entirely sure why we need to update/protect most servers, since generally they won't be running untrusted code, right?
    • ZenoArrow 9y ago
      The reason for doing so is minimising the risk if an attacker breaks into a server. This is also why systems like ATMs should be patched. Untrusted code execution on servers and ATMs may not be the norm, but it's far from impossible.
      • lsd5you 9y ago
        Ok, sure, but we have to consider whether that is worth it, and if they can run unpriveleged code, there is a good chance there is a software privelege escalation available anyway. Of course on shared servers this is going to be a nightmare.
      • randyrand 9y ago
        couldn't they just use a myriad of other priveledge escalation bugs? there have been ~4600 privledge escalation bugs found since 1999. 250 a year. Almost one every day. https://www.cvedetails.com/vulnerabilities-by-types.php https://www.cvedetails.com/vulnerabilities-by-types.php At this point we still won't have security and now we won't have performance either. It's putting the cart before the horse.
        • ZenoArrow 9y ago
          > "couldn't they just use a myriad of other priveledge escalation bugs?" The idea is to patch all known privilege escalation bugs. It's rare for an attacker to rely on a single attack vector. They may get to the point where they only have limited access to running code, and want to break out of that sandbox. Spectre/Meltdown may be the route they take to do so.
• dannyw 9y ago
  Surprise, surprise, Intel’s Meltdown patch has significant and serious performance impacts for specific kinds of workloads. All because Intel decided to “optimise” by checking for permissions after speculative retirement, instead of before like AMD.
  • imjasonmiller 9y ago
    If, as you say, AMD is not affected by Meltdown unlike Intel, will this significantly change the server market? Excuse the pun, but would this make e.g. AMD EPYC a lot more attractive for such data centers?
    • zeusk 9y ago
      AMD has confirmed it isn't affected by Meltdown. https://lkml.org/lkml/2017/12/27/2 https://lkml.org/lkml/2017/12/27/2
    • arien 9y ago
      Aren't they still vulnerable to Spectre?
      • Choco31415 9y ago
        Correct, and so are Samsung and Qualcomm CPU’s, according to the original paper: https://spectreattack.com/spectre.pdf https://spectreattack.com/spectre.pdf
      • naasking 9y ago
        Pretty much every fast CPU is vulnerable to some subset of Spectre.
      • Tuna-Fish 9y ago
        Yes to the bounds check violation version, so far no to the BTB poisoning version. The bounds check version only works inside a single process, so is only relevant when you run untrusted code within the same process as some private data (such as JS in a web browser). AMD claims that they believe that the way their branch predictor works effectively makes the BTB poisoning unusable, but there is no actual proof, and their statement regarding it was much more wishy-washy than they were with meltdown. (Which they specifically state they are completely immune to.)
• ZenoArrow 9y ago
  Considering the performance impact, I wonder how console manufacturers are going to handle this (assuming that the processors they used are vulnerable to Spectre/Meltdown).
  • rcarmo 9y ago
    Console manufacturers don't run unsigned code, so I expect they'll just sit still until the next hardware refresh.
    • ZenoArrow 9y ago
      Modern consoles now come with web browsers, and the researchers proved that the attacks could be performed via web browsers, did they not?
      • deleted 9y ago
        [deleted]
      • lrem 9y ago
        Next patch: we enhanced your security by disabling execution of JavaScript from untrusted domains. In unrelated news, we now block ads!
        • ZenoArrow 9y ago
          If they are going to block JavaScript, they need to do it for all domains, not just untrusted domains. For example, if I can MITM my own website activity (which I can, by having a device that sits between a router and a console), then I can change the JavaScript coming from trusted domains. In other words, if I visit a site like Google or Facebook on an affected device, I can change the JavaScript that is run, and make it still appear like it came from a trusted domain.
          • yorwba 9y ago
            You can be prevented from a successful MITM by enforcing HTTPS using HSTS and pinning the keys of trusted sites. That's not a viable solution for accessing the internet at large, but for a console that is only incidentally used for web browsing, it's absolutely an option.
• lathiat 9y ago
  I’d encourage anyway seeing super huge hits to make sure they are not using paravirt (particularly on Amazon). As that needs mitigation the the virt level the impact seems very large.
  • deleted 9y ago
    [deleted]
• mmaunder 9y ago
  Post. More. Benchmarks. I'll do the same once I have them. More data on this across platforms and apps is incredibly helpful for all.
  • chrisseaton 9y ago
    Real data like this post are more useful than more benchmarks.
    • mmaunder 9y ago
      Guess that's what I meant, but I'll take either!
      • deleted 9y ago
        [deleted]
    • jacquesm 9y ago
      They are benchmarks, just not microbenchmarks. Actual application performance is the gold standard for benchmarks. https://en.wikipedia.org/wiki/Benchmark_(computing)#Types_of_benchmark https://en.wikipedia.org/wiki/Benchmark_(computing)#Types_of...
      • wolfgke 9y ago
        > They are benchmarks, just not microbenchmarks. Actual application performance is the gold standard for benchmarks. This depends on your intended purpose of the benchmark. If you want to evaluate performance of a whole system, sure. But one often does benchmarks to find parts of the system that are critical to performance so that you can do further optimizations or do change in the software architecture to avoid these critical performance hotsports. For this case other benchmarks that allow an easy fine-grained interpretation of the data are surely better.
• Abishek_Muthian 9y ago
  If it's helpful, Our Node.js, MongoDB, Python servers all with significant network traffic didn't have any measurable impact after KPTI patches on Amazon Linux on T2.medium(burst), M4.large, T2.large(burst) respectively. Our impact is lesser than the figures suggested by redhat's advisory - https://access.redhat.com/articles/3307751 https://access.redhat.com/articles/3307751
  • urbanxs 9y ago
    That’s most likely because neither node, mongo nor python actually use the optimisation features of cpus. Says a lot about their performance quality.
  • bonzini 9y ago
    Are you using PV or HVM instances?
    • Abishek_Muthian 9y ago
      All of them on HVM.
      • falcolas 9y ago
        IIRC, XEN said that 32bit HVM VMs are not affected by Meltdown, and so probably don't get impacted by AWS' patches. They still require Linux kernel updates to protect the kernel space, so changes might still be seen there.
        • Abishek_Muthian 9y ago
          Yes, but XEN said 64bit PV aren't affected either because they already run in KPTI like environment. So I assume 64bit HVM like ours aren't impacted for our work load.
        • bonzini 9y ago
          They are affected but there is no fix yet. 64-bit PV is unaffected and won't suffer a performance penalty (more precisely, it was already suffering it!!!), hence my original question, but a 64-bit PV guest can use Meltdown to attack the hypervisor. The fix is to update Xen, though I am not sure if fixes are already publicly available.
• viraptor 9y ago
  I know they don't have to share the details, but the "patched" part is not really clear. Did they update to a new image / more recent kernel / anything else? Much like the redis post linked in HN before, we don't know if the impact is because of the "pti turned off/on" change, or are there more moving parts involved.
  • arien 9y ago
    If I recall correctly from older posts (I played Fortnite for a while), they're using AWS.
    • viraptor 9y ago
      That's still not answering many questions. I hope they publish a full analysis at some point.
      • lawrenceyan 9y ago
        Since they're using default provisioned EC2 instances, it's likely that the developers don't necessarily even fully understand their performance degradation. They just expect the service that they pay for to work properly.
        • viraptor 9y ago
          It's true, but it's not what I meant. They wrote "after a host was patched". This is ambiguous. So they mean the host as in instance, or host as in AWS host machine? Did they just reboot it get on the new/updated VM host, or did they rebuild to include the PTI fixes as well. Did they upgrade anything, or did everything else stay on the same version.
      • teej 9y ago
        From what I can tell, Amazon isn’t giving people deep technical information. They’ll just send you an email telling you which instances are being forced to restart and when.
  • 9y ago
• jedisct1 9y ago
  The Meltdown patch also introduces a serious performance hit for DNS servers and resolvers. Before giving figures, I want to run the same tests on a more recent CPU, but my current benchmarks are not great to say the least.
  • pixl97 9y ago
    Is this a Program -> CPU interaction that is slowing things down, or is this a CPU -> network interaction. I'm wondering if there are classes of network drivers that are having a much larger effect in performance. Network cards these days can do many things to improve performance like TCP/UDP offloading and because of that their drivers are very complex and I'm going to assume that there well be Meltdown fallout because of this.
• Darthy 9y ago
  The Meltdown attack requires an attacker to have a piece of code executed on your server. Epic's servers are used for login, where people send you data, and for game logic, where people also just send you data like "player x moved his avatar here, player y shoots etc". If all the server does is execute the code which Epic wrote themselves and already trust, why would it need to apply the Meltdown patch?
  • em3rgent0rdr 9y ago
    Your argument also applied to other circumstances where the computer only runs trusted code. For example a fully FLOSS software stack on home computer with JavaScript disabled in the browser.
  • akira2501 9y ago
    > why would it need to apply the Meltdown patch? They're using a "monolithic" cloud provider and don't have a choice in their current deployment?
  • drawnwren 9y ago
    This is a horrible approach to security. If you only secure against attacks you expect, you're gonna have a bad time.
    • deleted 9y ago
      [deleted]
    • toomuchtodo 9y ago
      Security is about risk mitigation. You cannot derisk entirely, so you make tradeoffs. Without knowing all of the parameters, it’s disingenuous to say it’s a horrible approach to security. The most secure computer is powered off, enclosed in concrete 6 feet below the surface of the earth. It is not very useful though.
      • em3rgent0rdr 9y ago
        Reminds me of Battlestar Galactica. The humans were so (wisely) fearful of the Cylons that their ship computers were not networked in anyway. For their risk/reward trade-off curve, the benefit from having computer networks were not worth the risk of the Cylons being able to compromise the entire ship. All communication was either verbal or done via fax printed to paper and so had to go through human intermediaries.
• ChildOfChaos 9y ago
  I have turned windows updates off so I don’t have to deal with this crap. Screw you intel + Microsoft.
  • bartread 9y ago
    Whilst I sympathise with your frustration, at least with Intel, I can't help feeling like you might be storing up bigger problems for yourself with this course of action.
    • ChildOfChaos 9y ago
      I get what you are saying but right now there are no known exploits and with so much patching happening will there ever? These things are always over blown in the media and the reality is very little damage happens to the average user. It’s servers perhaps that are most at risk. Also I don’t do much on my Windows, mostly gaming, I run an iMac and boot into it only for certain tasks, so it’s extremely unlikely I will ever have an issue. Any performance hit, even if small is just not worth updating for to me.
      • kazagistar 9y ago
        What do you mean "no known exploits"? The authors of the paper have an exploit that reads arbitrary system memory from a browser. And even after the meltdown patches, spectre "fixes" we have seen are only partial mitigation, and still potentially allow reading of passwords and third party cookies. But I guess if you want to wait til it's too late...
      • nemothekid 9y ago
        What do you mean by “no known exploits”? There are several PoCs out, one of which is in JavaScript for meltdown.
      • pixie_ 9y ago
        Yea sounds like you need all the CPU you can get. Good decision.
• jacksmith21006 9y ago
  Really like to see similar done on the Google Cloud infrastructure and see if there is any difference.
• j1vms 9y ago
  Maybe off-topic: Is formal verification viable anywhere in CPU logic design? Also, could any existing "CPU static analyzers" have caught the issue that caused Meltdown? Edit: It looks like the answer to the first is a definite yes.
  • Sephr 9y ago
    Not until we have a fully accurate theory of physics and perfect models for the CPUs.
  • andrewaylett 9y ago
    Formal verification of what? You can only verify properties you've thought of, and no-one conceived of this particular 'feature' causing issues like this until now. So I don't think formal verification would have helped: if anyone was in a position to realise the issue was worth verifying, they'd have been able to raise it without formal verification too.
    • zzzcpan 9y ago
      Presumably of isolation features. I'm sure everyone knew what side channel attacks were and knew that formal verification could've helped find bugs in isolation. They just chose not to do it, because it's only important for high-assurance, not for your regular insecure linux/bsd/windows. And I'm sure they are not going to bother with verification even after everything.
    • serf 9y ago
      >You can only verify properties you've thought of, and no-one conceived of this particular 'feature' causing issues like this until now. I've read for years about supposed insecurities with branch prediction -- it just wasn't shown practically. To say that it wasn't conceived of is a little off.
  • sehugg 9y ago
    Spitballing, but you'd have to prove that either no side channels exist, or that the information leaked by side channel(s) is indistinguishable from random noise, which I believe is an area of ongoing research.
• jacksmith21006 9y ago
  Really like to seem some benchmarks from different cloud providers before and after patches are applied to see if there is a material difference. The one most interested in is Google versus Amazon.
• erdbeerkuchen 9y ago
  The image didn't show up for me, here is a direct link for anyone with the same problem: https://lh6.googleusercontent.com/MwzsHRXQLVbmJ3pusNuGwn0ZQVjo9h8nRJHJhIo4d3XFqbvUYCj8EPq5jV7zeVEEcHAkraNBesbbNDW_UAlIjvw-hZBd80rKt7ZYl35nBIcfCCVyRvW5V7M7KVejv9tvVBHfgSKr https://lh6.googleusercontent.com/MwzsHRXQLVbmJ3pusNuGwn0ZQV...
  • snowpanda 9y ago
    Sorry to ask, but does anyone have a third mirror? Neither are working for me and I really am interested in seeing this image.
    • contrarian_ 9y ago
      https://i.imgur.com/pA7w4Tf.png https://i.imgur.com/pA7w4Tf.png
    • zzzcpan 9y ago
      http://anonymouse.org/cgi-bin/anon-www.cgi/https://lh6.googleusercontent.com/MwzsHRXQLVbmJ3pusNuGwn0ZQVjo9h8nRJHJhIo4d3XFqbvUYCj8EPq5jV7zeVEEcHAkraNBesbbNDW_UAlIjvw-hZBd80rKt7ZYl35nBIcfCCVyRvW5V7M7KVejv9tvVBHfgSKr http://anonymouse.org/cgi-bin/anon-www.cgi/https://lh6.googl...
  • fermienrico 9y ago
    Can someone explain what 1, 2 and 3 series in the plot means? What are we looking at? Why did one of them all of a sudden spiked?
    • johnbellone 9y ago
      Perhaps it’s multiple instances?
    • djsumdog 9y ago
      Yea I hated that too. They put up a graph but with no key! Arg! The article said they updated one host, so I'm assuming the spike is from that one VM that got patched.
• herf 9y ago
  Graph says pretty clearly the patch is 60-70% slower! or 2.5-3x hardware costs. I wonder which CPU family this is?
  • jug 9y ago
    This surprised me a lot. I thought 30-50% would be worst case and then with additive effects from both Spectre and Meltdown fixes. Not sure how it can be this bad. I imagine it could get even worse if you are running in a virtualized environment on top where the server is affected in turn, but I figure thst wouldn’t show in a CPU graph like this..
    • Tuna-Fish 9y ago
      The kernel address space isolation makes syscalls much more expensive than they were. A toy program that just repeatedly calls the cheapest syscall in the kernel would lose much more than half it's speed, but that's not what's reported because no-one actually needs to run that workload. Epic seems to have been particularly unlucky in how KPTI impacts them.
• mtgx 9y ago
  Looks like it may be time for Epic Games to consider switching to AMD Ryzen/EPYC.
  • dijit 9y ago
    I have been looking at AMD/Epyc CPUs recently. (Because making NUMA aware C++ code is hard, and AMD Epyc is a single socket on a server with 4 very closely knit NUMA zones so non-NUMA code will run better on that vs Intel) But unfortunately there's no comoddity server from HP/Dell available yet. But I hear one is on the way on the Dell side.
    • snuxoll 9y ago
      HP has paper launched the DL385 G10 with EPYC, I'm not sure if it's fully available through sales channels or not. Supermicro also has had EPYC systems available for a while - though that won't do you any good at all if you need a big name OEM for "nobody got fired for buying Cisco" reasons.
• forgotpw2018 9y ago
  Huge real world performance impact, they didn't say how much but it looks like close to 100%. I smell a class action lawsuit coming.
  • pitaj 9y ago
    Why? This wasn't gross incompetence in processor design, this kind of attack is completely new. I don't see how a class action could apply here, by IANAL
    • Filligree 9y ago
      Doesn't stop people from filing one. They already have, in fact. I agree that it shouldn't get anywhere, but I'm not as sanguine about whether or not it actually will.
    • jjeaff 9y ago
      It doesn't have to be gross incompetence. If you pay for something and it doesn't deliver as promised, you may be entitled to a partial or full refund.
    • chrisper 9y ago
      If you know that your product is flawed, do you keep selling it or will you pull it from the shelves? To this date I can still buy broken Intel CPUs.... They knew about the flaw in June. Yet they still kept selling Coffee Lake CPUs. If my 8700k wouldn't still be significantly faster than Amd Ryzen (or I wouldn't have to also return the MB), I would have switched to an Amd in a heart beat.
      • djsumdog 9y ago
        To be fair to Intel, mitigating this type of issue isn't at all trivial. If it were, it could have been fixed in a microcode update. You can't rush a chip design for something as complicated as x86_64. There are long multi-year development cycles and tons of regression tests. With this they need to add even more tests before they can start on attacking the issues with the design.
• lazyjones 9y ago
  Those who have looked at the patches in detail: does it treat older generation CPUs differently than 7th gen? The Paper authors who wrote the KAISER patch expected much worse performance on older CPUs due to implementation issues... Unfortunately, Epic Games don't provide any details about their CPUs AFAICT.
  • simfoo 9y ago
    Really interested in this as well. Details matter here
  • johnbellone 9y ago
    They did mention that they were using a cloud provider. But that’s about it.
  • cthalupa 9y ago
    I believe PCID is supposed to help mitigate the performance impact from the KPTI changes. Both the hardware and kernel have to support PCID
• merb 9y ago
  if people would still use good old owned hardware without any virtualizsation, spectre and meltdown would not be as scary as it really is on server hardware. which means that only clients would be affected. but since everything runs on the cloud, we basically need to update the whole world.
  • ebbv 9y ago
    First off, having dedicated hardware doesn't make these a non-issue, it just makes them less of an issue. If you have dedicated hardware they are concerns. Applications still need to be isolated from each other, and it's a path for someone who gets in as a user to potentially leverage it to root access. Second, saying everyone should have dedicated hardware is just nonsense. It's like saying everyone should run their own datacenter. For most people neither of those makes financial sense.
  • kazagistar 9y ago
    It's still pretty scary to have computers where all of memory is readabe from every process, even if you own them.
    • merb 9y ago
      well 50:50 chances are high that if a process goes bogus, that you have more problems than just "memory is readable from every process". (of course that does not apply to clients where code can run in jit (javascript), software that communicates with the internet, etc and runs on a remote machine). most servers probably should only run trusted code (of course that is mostly never the case because no company evaluates every process they running) but chances are high that if some uses linux and gnu stuff that most shady stuff gets catched. (I mean if not, some people could do a lot of bad stuff, consider a misbehaving systemd, nginx/apache, databases, which could basically do a lot of harm.)
• stefantalpalaru 9y ago
  It's time to move to userspace network drivers like https://github.com/snabbco/snabb/ https://github.com/snabbco/snabb/
• deleted 9y ago
  [deleted]
• brendangregg 9y ago
  What's the syscall/sec per CPU rate for this workload? My guess: over 1M.
• jitbit 9y ago
  Thought I'd share our graph too: https://www.jitbit.com/alexblog/270-cpu-usage-stats-after-patching-for-meltdown/ https://www.jitbit.com/alexblog/270-cpu-usage-stats-after-pa... Seeing almost 40% increase in CPU load
  • simik 9y ago
    Do you mean 40 percentage points? In other words, tripling?
    • lucb1e 9y ago
      Looking at the graph: probably. It was already increasing (from 16-19%, one outlier to 27%, first 8 datapoints) to around 25-30% (the last 6 before the big jump). Then a big jump to 62-85% (N=20) with most around 75% (N≃13). The last bit is around 60-72% (N=12) and the very last datapoint is below 40% again but that is half outside the graph (cherry-picking datapoints?). So to summarize, from 25% to 65%, or 2.6× the original. Note that this is all from eyeballing that graph and estimating what percentage a datapoint is at, since the scale has an interval of 20 percent points and no minor grid or anything.