9 ms·

Zen 5's Leaked Slides

210 points by treesciencebot 3y ago

dist-epoch 3y ago
One big difference between CPUs and GPUs is memory bandwidth. Dumb question, but why don't Intel/AMD massively increase memory bandwidth by adding more memory lanes? For example instead of the usual 4 slots on a desktop motherboard you would have 16 slots surrounding the CPU on all sides. Is it that the CPU would be unable to effectively use all that bandwidth due to less parallelism compared with a GPU?
- ksec 3y ago
  >Is it that the CPU would be unable to effectively use all that bandwidth due to less parallelism compared with a GPU? Yes. That is why you only see more memory channels for higher CPU count.
- sapiogram 3y ago
  The standard answer is cost, but I'm not sure how significant that actually is in this day and age. Maybe it's just artificial market segmentation at this point.
  - wtallis 3y ago
    There's nothing artificial or imaginary about the extra costs of using a CPU socket with a 512-bit memory bus rather than a 128-bit bus. Bigger sockets and more memory slots make motherboards significantly more expensive, and the wider memory controller and faster interconnects makes a CPU more expensive and power-hungry. Those costs affect the whole platform, including entry-level systems that don't have enough memory to populate all those channels and ones that don't run workloads which would benefit from the extra bandwidth.
- pitaj 3y ago
  There's also the fact that CPUs and DRAM are socketed rather than being soldiered like on graphics cards.
  - bzzzt 3y ago
    Apple would like a word with you ;)
    - dathinab 3y ago
      apple is not selling to data centers given there use case of - premium consumer only products - which are mostly portable and have a low thermal budget - which are by most users hardly ever run on full throttle - a well documented tendency to proclaim any hardware failure as user fault makes this quite viable for them but if you run systems with 4TiB of RAM and 12k+$ CPU which is most time running full throttle potential for days at a time with a higher thermal budged and customers which might have docents or even thousands of such CPUs and will be very pushy about reclaiming any failure they think isn't their fault then having to replace over 20-50k$ just isn't viable (instead of ~1-3k...)
- CyberDildonics 3y ago
  It's very rare that software is able to use all the memory bandwidth on a CPU. Something well written with multiple cores is where you will see it happen and that's why higher core counts can often have more memory lanes.
  - deleted 3y ago
    [deleted]
  - rini17 3y ago
    Such as llama.cpp. It saturated memory bw while using only 6 of 8 cores on the xeon cpu i tried it on.
- MayeulC 3y ago
  I'd say that bandwidth matters less than latency for performance. Both are good, but this is part of the reason why we see ever-increasing cache sizes, and memory physically closer to the CPU. Edit: Of course, it depends on the benchmark. Latency is critical for programs that access many memory addresses randomly, not for data processing applications that scan memory linearly (and GPU use-cases often fall into that category).
- TerrifiedMouse 3y ago
  With CPUs the bottleneck isn't just memory bandwidth only but also memory latency. If prefetch fails to get the correct data loaded into the cache, the CPU just stalls for thousands of cycles waiting for RAM to deliver the data. If I'm correct, this is the difference between DDR RAM and GDDR RAM - the latter has higher bandwidth but also higher latency, which is great for graphics with its high bandwidth requirements and very predictable memory access patterns (except when you do ray tracing) which allows you to hide the latency, but you don't want to use GDDR for system RAM if you value performance.
  - Const-me 3y ago
    Yeah, the throughput/latency tradeoff is real. Here’s an unusual motherboard salvaged from Xbox Series X which combines AMD Zen2 CPU cores, GDDR6 memory, where the integrated GPU is disabled due to defects: https://www.tomshardware.com/news/4800s-xbox-chip-shows-us-why-cpus-do-not-come-with-gddr6 https://www.tomshardware.com/news/4800s-xbox-chip-shows-us-w... The performance is not great. GPUs are good at hiding the latency due to high degree of parallelism. GPU cores switch to another thread instead of waiting for the data. Each CPU core only runs 2 threads, it can’t quite do the same thing. GPU memory access patterns are only great for well written compute shaders (or CUDA kernels) when the developer who wrote that shader managed to do fully coalesced loads and stores. Memory access patterns for pixel shaders are not that great, especially when doing trilinear or anisotropic sampling in these shaders.
  - HappyDaoDude 3y ago
    Which is why I always found it funny to see consoles using unified GDDR for CPU and GPU. But you can optimize around the limitation (within reason) and don't have to content with a general non-specified OS. This is why I do wonder about the path not taken with the 1T-SRAM that powered the Gamecube/Wii. It was designed purely with low latency in mind but I don't think it scaled well.
- cogman10 3y ago
  > Is it that the CPU would be unable to effectively use all that bandwidth due to less parallelism compared with a GPU? Yes, but also it's due to simple physical limitations. If you look at how close CPUs are to RAM as well as the number of PCB lines coming out of CPUs, it becomes pretty readily apparent that there's not much room for additional lines. You have to recognize that each of those lines is creating a magnetic field, the closer you stack them, the more likely cross talk is. We could continue to lower the voltage on the memory lanes so you can pack them closer together, but then you have to start worrying about environmental interference on the lines. It likely won't be too long in the future (if it's not already here) that we'll start seeing shielding for the memory lanes so they can be packed closer together and more can be added. It wouldn't shock me if this became a part of DDR6. It also wouldn't shock me to start seeing more complex memory controllers doing things like lz4 compression to eck out even higher bandwidths (particularly when the memory is highly uniform). We are sort of seeing the precursor of this in DDR5 with commands to set whole blocks to a given value.
  - CoastalCoder 3y ago
    I'm curious if anytime soon we'll start seeing optical interconnections between CPUs and on-device / same-chasis memory. I'm not sure what that would add for latency and cost, but I'm guessing that with optical wave guides, crosstalk isn't a concern. At least for the optical part of the data path.
    - kimixa 3y ago
      The speed of light in an optical fiber is still only about 2*10^8 m/s - ddr5 6000's 3ghz clock means it still only travels 6.7cm/clock - so there'll still be difficulties around latency if it's not "close", and that's ignoring any issues of actually getting optical transceivers on the die itself.
      - billti 3y ago
        It blew my mind when I learned that even if you could hand wire conceptually perfect switches and wires, once your board was over a few inches, the laws of physics (e.g., the speed of light) would mean you still couldn’t beat a modern CPU for clock speed. It really is insane the level at which current hardware works.
- mnd999 3y ago
  Doesn’t EPYC already do this? You can put an EPYC in a desktop - I did.
- toast0 3y ago
  Server sockets have more memory channels and server motherboards usally have more memory slots. Layout limitations probably make ram on four sides not very feasible. But, more memory channels means more pins and larger sockets, and that adds cost and layout issues for motherboards. HBM ram that attaches on top of the cpu addresses the layout issue (and the bandwidth issue), but interferes with heat removal and removes expandability. Apple makes it work, but they limit clock speeds and were never big on expandability. Intel and AMD have a hard time in the market when their top consumer chips don't clock to the moon, even if the last 10% of clock speed takes 50% of the power and delivers 5% of performance (percentages made up). Server chips don't need to clock so high, so AMD can release clock limited compact cores that have about the same IPC in a bit more than half the area, and the difference in clocks for a max-cores regular zen4 and a max-cores zen4c isn't too much (L3/core is half for zen4c though, so may not be a benefit depending on your working set)
  - AnthonyMouse 3y ago
    > but interferes with heat removal and removes expandability Eh. RAM generally consumes a single digit number of watts and will do so regardless of where it's physically located in the system. If you put HBM on the CPU socket then you have to replace them together, but you can still replace them. It could also make sense to do both: If you have 16GB of HBM and 16GB of DDR5, you have 32GB of RAM and half of it is faster. Or 80GB of RAM and 16GB of it is faster. Cache hierarchies are a good way to strike a balance between performance and cost, and then the latter can still be expanded independent of the CPU.
    - ta988 3y ago
      Roughly 3W per 8GB in DDR4. DDR5 is said to use less.
- hedgehog 3y ago
  Two reasons: First, it's a lot of wiring which makes everything more expensive, complicated, power hungry, etc. Mobile SoCs (and Apple's chips) sidestep this issue by packaging the memory with the CPU so distances are short and the issues are avoided. Second, increasing memory bandwidth wouldn't be that much of a performance win because as you point out there isn't that much parallelism and normal software would benefit much more from lower latency than higher bandwidth. Together that means a many-slots design would cost a lot but not be proportionally faster. Tradeoffs are different for server boards (many cores in the CPU) and on platforms with integrated GPUs (becoming more common) so none of this is static. I suspect in-package RAM will become more common even for platforms that still have a couple channels of slotted memory, that would create a pool of faster memory available for graphics etc while still leaving room for more if you need to have a lot of Chrome tabs open or whatever.
- kllrnohj 3y ago
  Consider how GPUs & CPUs actually use memory: On a GPU, the memory access patterns are often very wide & easily controlled, such as fetching a block of texture memory. This is like streaming reads for a disk, where you can actually just throw essentially RAID 0 at the problem to scale up really easily. So scaling all the way up to ridiculously wide 2048-bit bus widths (ex. AMD Vega 64) is completely viable since you're working with relatively bulk array data almost constantly. Now most GPUs aren't going quite that wide to hit the speeds they want, but consider that 192-bit is still considered "low-end" here and that's already a lot wider than you'll find on most CPUs. On a CPU, especially in languages like Java, JavaScript, etc..., memory access patterns are often very random and very small. Consider just something like a simple boolean accessor call `if (foo->isEnabled()) doThing();` - you're reading let's say 8 bytes (foo pointer) to in turn access 1 byte (a boolean). Now the CPU is already reading in chunks of 64-bytes typically (cache line width), so you're already blowing 2 memory fetches to get 128-bytes of data just to read 9 bytes total. Going wider, which is how GPUs scale bandwidth, is utterly pointless here. You'd just be fetching even more data you didn't want in the first place just to occupy precious cache space on something you're probably not going to use anyway. In fact this is why DDR5 actually essentially shrinks the bus-width, from 64-bit in prior DDR generations to 2x32-bit in DDR5
  - YetAnotherNick 3y ago
    > especially in languages like Java, JavaScript, etc Not only there is sequential array in both the language, it is very often used. Things like iterating the array is one of the most common operation and is almost always memory bandwidth bound even for interpreted language.
    - logicchains 3y ago
      >Not only there is sequential array in both the language, it is very often used In Java only primitive arrays are memory-efficient for iteration; ArrayLists are not because every element is boxed, so even when iterating sequentially a lookup of some random memory address is needed for every element.
  - dathinab 3y ago
- fbdab103 3y ago
  Genuine question: how often am I memory bandwidth constrained? Or to put it another way, if I had infinite memory bandwidth, how much faster would browsing/gaming/compiling become?
  - deleted 3y ago
    [deleted]
  - champtar 3y ago
    I don't know how often it's a problem, but I work for a company doing software video encoding, and we always fill up all the dimm slots on servers to have as much bandwidth as possible, even if we have only really use maybe 1/4 of the RAM. I'm not sure any of the standard Linux tools can show you memory bandwidth usage easily (maybe perf), I know we use Intel PCM (https://github.com/intel/pcm https://github.com/intel/pcm) and AMDuProfPCM (https://www.amd.com/en/developer/uprof.html https://www.amd.com/en/developer/uprof.html)
  - semi-extrinsic 3y ago
    Depends on the task you are concerned with; see Roofline model [1]. It boils down to "how many ops will you do for each value you load from RAM to cache". I don't know about typical consumer workloads, but e.g. for physics simulations, any time you need to solve a system where the value in a point depends on a lot of other points you are bandwidth limited. Like solving the pressure equation in fluid simulations. Also I believe a lot of traditional database workloads fall into this category. You can measure memory bandwidth with the STREAMS benchmark. [1] https://en.m.wikipedia.org/wiki/Roofline_model https://en.m.wikipedia.org/wiki/Roofline_model
  - dathinab 3y ago
    lets say instead of infinite you get 16 instead of 2 channels then while probably all applications will profit for most consumer applications you might not notice much of a difference through you can measure some difference in benchmarks one exceptions would be various graphic editing software if you do stuff like editing 8k multi channel video content For compiling you might feel a noticeable difference. It won't be 8x but if you pair it with enough cores and things which have tons of parallel compilation parts it will really feel noticeable faster. But for most gaming you instead of getting 8x the bandwidth you probably benefit more from a speed improvement e.g. running 6000MT/s DDR5. Anyway you can get it today ;) The M2 Ultra has a pretty insane memory bandwidth (and so had the M1 Ultra).
- tedunangst 3y ago
  When was the last time you saw a GPU with socketed memory?
  - fodkodrasz 3y ago
    I had a Tseng ET4000 I think that had socketed memory, and has some Trident TVGA which I beefed up from 512k to 2MB to get 65k colours and 800x600 maybe... I had a GUS soundcard for a while and it also had an EDO slot I think.
- ptrwis 3y ago
  In DDR5 you have two memory channels per module.
  - dathinab 3y ago
    but depending on context you still refer to a pair of 32b DDR5 channels as "a single channel" e.g. if we speak about how many memory channels a x86 desktop, prosumer or server CPU have they count them as a single channel main reason for this is that DDR5 didn't split the channel instead of adding a new channel
- foota 3y ago
  Aside from what others have said about greater bit width only being useful for large fetches, there's also circuitry necessary to allow more parallel fetches from the L1 to other parts of the memory subsystem (for Intel, these are called line fill buffer) and they're the real limiting factor for single threaded memory bandwidth (at least on that platform). For whole chip bandwidth this isn't as much of an issue, but the things others have said are true there still.
- zamadatix 3y ago
  I'd be happy with just making each existing slot its own lane.
- dathinab 3y ago
  The number of slots doesn't matter but the number of channels, i.e. x86 board have 4 slots but only 2 channels (through nomenclature is a bit of a mess as as depending on context channel might mean something else). More memory bandwidth increases the price/size of the IO module on the CPU but I don't think that's a major reason. One of the major reason AFIK is to keep the number of channels low is that it directly maps to the number of pins a socket need to has and the number of high quality lanes which need to be routed independently equally long and chided from inference from e.g. power lines to the CPU. So especially e.g. for server design chip vendors balance the benefit of more memory channels with the cost of it increasing socket size/pins, complexity of mother boards etc. And while on the paper doubling the memory channels doubles the memory bandwidth it doesn't at all means you RAM is twice as fast. It's similar to the difference between duplicating the number of cores vs. duplication the per-core IPC. While all applications profit from more bus width the degree might differ hugely. Oversimplified super highly parallelizeable tasks (e.g. much of GPU tasks and many AI tasks) tend to also be better at taking advantage of more bandwidth through more channels but many other taks less so. This means that the degree to which more channels will yield benefits highly depend on the task the CPU vendor expect is majorly run on it. And the tasks which most benefit from it already tend to be run on GPUs anyway. Apples M1,M2 CPUs one the other hand have a much much smaller cost problem wrt. more memory channels due to how they stack ram on top of the CPU (I think) and they happen to also have much more memory bandwidth by having more bus width. But stacking RAM on top of CPUs has it's own issues. Like for example if the RAM dies the whole CPU is dead and you not being able to upgrade the RAM. Doesn't matter that much for apple, but matters a lot for AMD/Intel which core focus is the server center, not desktop CPUs. Having to replace a 10k CPU and multiple TiB of RAM because a single ram segment fails just isn't acceptable. Additionally for some use cases instead of stacking RAM on the CPU you get as much or more benefit by stacking tons of L3 cache on it (AMD 3D V-cache). There also seems to be in general be more issues with thermal budges, not an issue for apple of which most devices are mostly passive cooled. Issues with manufacturing cost making it harder to provide many CPU spec+RAM amount variants (again doesn't matter for apple but does for Intel/AMD). Potential cost issues making it only suitable for "premium" products (again not an issue for Apple but for Intel/AMD). And maybe more. So basically: - on server (or pro-sumer,i.e. Threadripper) more channels are already used, but it's always a cost benefit balance where there could be even more channels - on desktop, at least currently, it's not seen to be worth the price/complexity increase Through Apples is putting a bit of pressure on the market, but outside of the high end segment it still makes little sense and it's probably not worth it to have it just for the "high end but not pro-sumer" use-case, that market it too small. I would predict prosumer devices like Threadrippers to grow the number of memory channels with every now socket they have. And I do predict a major technology change for "normal desktop computers" 3-6 years down the line which probably will bring more channels to desktop computers. But only then. And it's very speculative.
- mastax 3y ago
  In addition to what everyone has already said, memory controllers are relatively big and expensive. System integrators often ship systems <~$1000 with only a single channel of memory. It’d probably be like pulling teeth to get them to actually ship a 4-channel configuration. Putting in unused memory controllers is a big waste. People who have many threaded workloads which can actually use all that memory bandwidth tend to get workstation or server platforms.
- fanf2 3y ago
  The Apple M1 and M2 use HBM to massively increase the memory bandwidth. The new Intel Xeon Max processors have 64GiBytes of HBM in the package, as well as offboard DDR5.
  - wtallis 3y ago
    Apple uses LPDDR, which is what's used in all smartphones and most really thin laptops. LPDDR has the exact opposite design goal from HBM: LPDDR is optimized for high bandwidth per pin to enable low pin counts, while HBM relies on advanced silicon interposer packaging to enable extremely high pin counts allowing for modest bandwidth per pin. The only similarity is that Apple puts the LPDDR in the package with the CPU, but it's an entirely different and more mundane packaging technology compared to what is used for HBM.
- touisteur 3y ago
  I guess we'll know pretty soon, with the availability of the Intel Max with HBM whether it's that simple?
ksec 3y ago
Zen 4, AMD Ryzen 5 7640U, 4.9Ghz, Single Core GB6 Score @ 2340. Apple A17 Pro, 3.8Ghz, Single Core GB6 Score @ 2914 I dont think asking for an 20% increase in IPC is too much at this point.
- KingOfCoders 3y ago
  AMD seems not interested. Intel will go the Apple way though, simpler cores and trading die area for ever larger caches, also on-die memory, downside you can't upgrade memory, upside it's faster and more power efficient. [Edit] Or someone could have said: I think you meant processor not die, and I would have said: Yes sure, not sure what was on my mind.
  - appplication 3y ago
    Upgrading memory is great in theory (and practice for those who do) but I think for 95% of users this is not a consideration.
    - kibwen 3y ago
      It becomes more of a consideration when accounting for the fact that you can buy a Macbook that ships with an un-upgradeable 8 GB of RAM, which, sadly, is deficient for even moderate tasks these days.
    - toast0 3y ago
      Upgradable memory also makes things more flexible for OEMs. Most OEMs continue to sell systems built around last years processor (or even a couple year old processors), and they can increase the memory in those systems as needed. Maybe 4GB was enough when they came out, but now you need 8GB or 16GB to sell a system with an older processor. Not a big deal with socketed RAM, not even a huge deal with soldered RAM, but the processors with 4GB on package become very hard to sell.
  - kllrnohj 3y ago
    > Intel will go the Apple way though Hmm, curious what the "Apple way" is here.. > simpler cores Apple's cores are very far from simple. They are much larger than the other ARM CPUs on the market, and this is also why they are faster than other ARM CPUs on the market. > trading die area for ever larger caches Apple has big L1 comparatively, but L2 & L3 are nothing special. Certainly nowhere close to the massive caches that AMD's X3D offers. > also on-die memory Apple doesn't have on-die memory. AMD is the only company to have shipped on-die memory to consumers (they were also the first to ship it), and they only did so on GPUs so far. Nvidia & Intel are playing with it for datacenter products, though, but that's about it (H100 & Xeon Max respectively)
pvg 3y ago
Discussion last week https://news.ycombinator.com/item?id=37715082 https://news.ycombinator.com/item?id=37715082
TheRealPomax 3y ago
It's not a leak unless the leaker proves its origin.
phkahler 3y ago
One thing I never see in pieces like this is comments on how a change might affect hyperthreading performance. It's not relevant for single threaded code, but then neither is doubling the core count from 4 to 8 or 16.
- ksec 3y ago
  I dont think they care as much as in the case of Multi Core era. It is easier just to sell more cores.
- nullc 3y ago
  Both intel and AMD's hyperthreading seems pretty lame compared to Power's for some reason. One cause may be that with just two threads the chance that both get blocked on a memory access is pretty good.
snvzz 3y ago
>(fusion) Conditional branching on x86 involves using an instruction that sets flags, and then a branch that jumps (or not) depending on flags. x86 shows its age and accumulated cruft here. RISC-V has test and jump in a single instruction, doing away with flags.
- stavros 3y ago
  Why can this not be added to x86? Is the ISA immutable now?
  - pritambaral 3y ago
    It could be added with an extension, but software would continue to be compiled without it. For many, many years.
    - stavros 3y ago
      Sure, but that's fine, no? Better late than never.
      - pritambaral 3y ago
        Now that op fusion has been implemented for this idiom, the cost of a new instruction does not seem worth the benefit. The benefit would be only a tiny improvement in code size, and that's restricted to when the compiler can be absolutely certain that the target CPU is going to be modern enough to support it. So, in practice, for many, many years, this benefit would be almost never reaped. This is not a new topic in the design of ISAs and their implementations in silicon. Idiom recognition and optimization has long existed in x86, ARM, and I'm sure other old ISA implementations. RISC-V can avoid it by adding dedicated instructions because it's just getting started (relatively speaking), but even so, I'm sure there will be opportunities for idiom recognition and optimization that RISC-V implementers will take.
        snvzz 3y ago
Moldoteck 3y ago
Can I ask why more manufacturers don't implement apple's strategy with soc with ram and all and offer option to add additional classic ram as a slower cache buffer between soc&ssd? This should both solve the lack of upgradability and unlock more performance per watt?
- segfaultex 3y ago
  Money. Sounds reductive, but I think oems don’t see much demand for an architecture that implements both. So they’ll go with one or the other.
- zamadatix 3y ago
  Intel did this is the Sapphire Rapids xeons via 64 GB HBM built in and DDR 5 slots. L4 cache is also not unheard of, having been tried a few times for well of a decade. The downsides are cost/complexity for relatively little gain.
- rajnathani 3y ago
  Your comment is ambiguous, but if you’re implying that Apple does this then it’s incorrect. They just have “unified memory” (i.e. the high bandwidth SoC substrate memory).
  - Moldoteck 3y ago
    Yes, and I'm asking why not allow to add extra ram through slots as a slower cache between unified mem and ssd? And why other manufacturers are not doing this to achieve apple lvl of performanca and also allowing upgradability
    - cma 3y ago
      I thought slots don't really slow things down much and aren't the primary source of latency, isn't it more about cost and power? Adding slots back in would remove the cost and power advantages wouldn't they? And now using more SRAM to track occupancy of an extra layer of cache, or if not SRAM for that something adding more latency.

One big difference between CPUs and GPUs is memory bandwidth. Dumb question, but why don't Intel/AMD massively increase memory bandwidth by adding more memory lanes? For example instead of the usual 4 slots on a desktop motherboard you would have 16 slots surrounding the CPU on all sides. Is it that the CPU would be unable to effectively use all that bandwidth due to less parallelism compared with a GPU?

ksec 3y ago
>Is it that the CPU would be unable to effectively use all that bandwidth due to less parallelism compared with a GPU? Yes. That is why you only see more memory channels for higher CPU count.
sapiogram 3y ago
The standard answer is cost, but I'm not sure how significant that actually is in this day and age. Maybe it's just artificial market segmentation at this point.
- wtallis 3y ago
  There's nothing artificial or imaginary about the extra costs of using a CPU socket with a 512-bit memory bus rather than a 128-bit bus. Bigger sockets and more memory slots make motherboards significantly more expensive, and the wider memory controller and faster interconnects makes a CPU more expensive and power-hungry. Those costs affect the whole platform, including entry-level systems that don't have enough memory to populate all those channels and ones that don't run workloads which would benefit from the extra bandwidth.
pitaj 3y ago
There's also the fact that CPUs and DRAM are socketed rather than being soldiered like on graphics cards.
- bzzzt 3y ago
  Apple would like a word with you ;)
  - dathinab 3y ago
    apple is not selling to data centers given there use case of - premium consumer only products - which are mostly portable and have a low thermal budget - which are by most users hardly ever run on full throttle - a well documented tendency to proclaim any hardware failure as user fault makes this quite viable for them but if you run systems with 4TiB of RAM and 12k+$ CPU which is most time running full throttle potential for days at a time with a higher thermal budged and customers which might have docents or even thousands of such CPUs and will be very pushy about reclaiming any failure they think isn't their fault then having to replace over 20-50k$ just isn't viable (instead of ~1-3k...)
CyberDildonics 3y ago
It's very rare that software is able to use all the memory bandwidth on a CPU. Something well written with multiple cores is where you will see it happen and that's why higher core counts can often have more memory lanes.
- deleted 3y ago
  [deleted]
- rini17 3y ago
  Such as llama.cpp. It saturated memory bw while using only 6 of 8 cores on the xeon cpu i tried it on.
MayeulC 3y ago
I'd say that bandwidth matters less than latency for performance. Both are good, but this is part of the reason why we see ever-increasing cache sizes, and memory physically closer to the CPU. Edit: Of course, it depends on the benchmark. Latency is critical for programs that access many memory addresses randomly, not for data processing applications that scan memory linearly (and GPU use-cases often fall into that category).
TerrifiedMouse 3y ago
With CPUs the bottleneck isn't just memory bandwidth only but also memory latency. If prefetch fails to get the correct data loaded into the cache, the CPU just stalls for thousands of cycles waiting for RAM to deliver the data. If I'm correct, this is the difference between DDR RAM and GDDR RAM - the latter has higher bandwidth but also higher latency, which is great for graphics with its high bandwidth requirements and very predictable memory access patterns (except when you do ray tracing) which allows you to hide the latency, but you don't want to use GDDR for system RAM if you value performance.
- Const-me 3y ago
  Yeah, the throughput/latency tradeoff is real. Here’s an unusual motherboard salvaged from Xbox Series X which combines AMD Zen2 CPU cores, GDDR6 memory, where the integrated GPU is disabled due to defects: https://www.tomshardware.com/news/4800s-xbox-chip-shows-us-why-cpus-do-not-come-with-gddr6 https://www.tomshardware.com/news/4800s-xbox-chip-shows-us-w... The performance is not great. GPUs are good at hiding the latency due to high degree of parallelism. GPU cores switch to another thread instead of waiting for the data. Each CPU core only runs 2 threads, it can’t quite do the same thing. GPU memory access patterns are only great for well written compute shaders (or CUDA kernels) when the developer who wrote that shader managed to do fully coalesced loads and stores. Memory access patterns for pixel shaders are not that great, especially when doing trilinear or anisotropic sampling in these shaders.
- HappyDaoDude 3y ago
  Which is why I always found it funny to see consoles using unified GDDR for CPU and GPU. But you can optimize around the limitation (within reason) and don't have to content with a general non-specified OS. This is why I do wonder about the path not taken with the 1T-SRAM that powered the Gamecube/Wii. It was designed purely with low latency in mind but I don't think it scaled well.
cogman10 3y ago
> Is it that the CPU would be unable to effectively use all that bandwidth due to less parallelism compared with a GPU? Yes, but also it's due to simple physical limitations. If you look at how close CPUs are to RAM as well as the number of PCB lines coming out of CPUs, it becomes pretty readily apparent that there's not much room for additional lines. You have to recognize that each of those lines is creating a magnetic field, the closer you stack them, the more likely cross talk is. We could continue to lower the voltage on the memory lanes so you can pack them closer together, but then you have to start worrying about environmental interference on the lines. It likely won't be too long in the future (if it's not already here) that we'll start seeing shielding for the memory lanes so they can be packed closer together and more can be added. It wouldn't shock me if this became a part of DDR6. It also wouldn't shock me to start seeing more complex memory controllers doing things like lz4 compression to eck out even higher bandwidths (particularly when the memory is highly uniform). We are sort of seeing the precursor of this in DDR5 with commands to set whole blocks to a given value.
- CoastalCoder 3y ago
  I'm curious if anytime soon we'll start seeing optical interconnections between CPUs and on-device / same-chasis memory. I'm not sure what that would add for latency and cost, but I'm guessing that with optical wave guides, crosstalk isn't a concern. At least for the optical part of the data path.
  - kimixa 3y ago
    The speed of light in an optical fiber is still only about 2*10^8 m/s - ddr5 6000's 3ghz clock means it still only travels 6.7cm/clock - so there'll still be difficulties around latency if it's not "close", and that's ignoring any issues of actually getting optical transceivers on the die itself.
    - billti 3y ago
      It blew my mind when I learned that even if you could hand wire conceptually perfect switches and wires, once your board was over a few inches, the laws of physics (e.g., the speed of light) would mean you still couldn’t beat a modern CPU for clock speed. It really is insane the level at which current hardware works.
mnd999 3y ago
Doesn’t EPYC already do this? You can put an EPYC in a desktop - I did.
toast0 3y ago
Server sockets have more memory channels and server motherboards usally have more memory slots. Layout limitations probably make ram on four sides not very feasible. But, more memory channels means more pins and larger sockets, and that adds cost and layout issues for motherboards. HBM ram that attaches on top of the cpu addresses the layout issue (and the bandwidth issue), but interferes with heat removal and removes expandability. Apple makes it work, but they limit clock speeds and were never big on expandability. Intel and AMD have a hard time in the market when their top consumer chips don't clock to the moon, even if the last 10% of clock speed takes 50% of the power and delivers 5% of performance (percentages made up). Server chips don't need to clock so high, so AMD can release clock limited compact cores that have about the same IPC in a bit more than half the area, and the difference in clocks for a max-cores regular zen4 and a max-cores zen4c isn't too much (L3/core is half for zen4c though, so may not be a benefit depending on your working set)
- AnthonyMouse 3y ago
  > but interferes with heat removal and removes expandability Eh. RAM generally consumes a single digit number of watts and will do so regardless of where it's physically located in the system. If you put HBM on the CPU socket then you have to replace them together, but you can still replace them. It could also make sense to do both: If you have 16GB of HBM and 16GB of DDR5, you have 32GB of RAM and half of it is faster. Or 80GB of RAM and 16GB of it is faster. Cache hierarchies are a good way to strike a balance between performance and cost, and then the latter can still be expanded independent of the CPU.
  - ta988 3y ago
    Roughly 3W per 8GB in DDR4. DDR5 is said to use less.
hedgehog 3y ago
Two reasons: First, it's a lot of wiring which makes everything more expensive, complicated, power hungry, etc. Mobile SoCs (and Apple's chips) sidestep this issue by packaging the memory with the CPU so distances are short and the issues are avoided. Second, increasing memory bandwidth wouldn't be that much of a performance win because as you point out there isn't that much parallelism and normal software would benefit much more from lower latency than higher bandwidth. Together that means a many-slots design would cost a lot but not be proportionally faster. Tradeoffs are different for server boards (many cores in the CPU) and on platforms with integrated GPUs (becoming more common) so none of this is static. I suspect in-package RAM will become more common even for platforms that still have a couple channels of slotted memory, that would create a pool of faster memory available for graphics etc while still leaving room for more if you need to have a lot of Chrome tabs open or whatever.
kllrnohj 3y ago
Consider how GPUs & CPUs actually use memory: On a GPU, the memory access patterns are often very wide & easily controlled, such as fetching a block of texture memory. This is like streaming reads for a disk, where you can actually just throw essentially RAID 0 at the problem to scale up really easily. So scaling all the way up to ridiculously wide 2048-bit bus widths (ex. AMD Vega 64) is completely viable since you're working with relatively bulk array data almost constantly. Now most GPUs aren't going quite that wide to hit the speeds they want, but consider that 192-bit is still considered "low-end" here and that's already a lot wider than you'll find on most CPUs. On a CPU, especially in languages like Java, JavaScript, etc..., memory access patterns are often very random and very small. Consider just something like a simple boolean accessor call `if (foo->isEnabled()) doThing();` - you're reading let's say 8 bytes (foo pointer) to in turn access 1 byte (a boolean). Now the CPU is already reading in chunks of 64-bytes typically (cache line width), so you're already blowing 2 memory fetches to get 128-bytes of data just to read 9 bytes total. Going wider, which is how GPUs scale bandwidth, is utterly pointless here. You'd just be fetching even more data you didn't want in the first place just to occupy precious cache space on something you're probably not going to use anyway. In fact this is why DDR5 actually essentially shrinks the bus-width, from 64-bit in prior DDR generations to 2x32-bit in DDR5
- YetAnotherNick 3y ago
  > especially in languages like Java, JavaScript, etc Not only there is sequential array in both the language, it is very often used. Things like iterating the array is one of the most common operation and is almost always memory bandwidth bound even for interpreted language.
  - logicchains 3y ago
    >Not only there is sequential array in both the language, it is very often used In Java only primitive arrays are memory-efficient for iteration; ArrayLists are not because every element is boxed, so even when iterating sequentially a lookup of some random memory address is needed for every element.
- dathinab 3y ago
fbdab103 3y ago
Genuine question: how often am I memory bandwidth constrained? Or to put it another way, if I had infinite memory bandwidth, how much faster would browsing/gaming/compiling become?
- deleted 3y ago
  [deleted]
- champtar 3y ago
  I don't know how often it's a problem, but I work for a company doing software video encoding, and we always fill up all the dimm slots on servers to have as much bandwidth as possible, even if we have only really use maybe 1/4 of the RAM. I'm not sure any of the standard Linux tools can show you memory bandwidth usage easily (maybe perf), I know we use Intel PCM (https://github.com/intel/pcm https://github.com/intel/pcm) and AMDuProfPCM (https://www.amd.com/en/developer/uprof.html https://www.amd.com/en/developer/uprof.html)
- semi-extrinsic 3y ago
  Depends on the task you are concerned with; see Roofline model [1]. It boils down to "how many ops will you do for each value you load from RAM to cache". I don't know about typical consumer workloads, but e.g. for physics simulations, any time you need to solve a system where the value in a point depends on a lot of other points you are bandwidth limited. Like solving the pressure equation in fluid simulations. Also I believe a lot of traditional database workloads fall into this category. You can measure memory bandwidth with the STREAMS benchmark. [1] https://en.m.wikipedia.org/wiki/Roofline_model https://en.m.wikipedia.org/wiki/Roofline_model
- dathinab 3y ago
  lets say instead of infinite you get 16 instead of 2 channels then while probably all applications will profit for most consumer applications you might not notice much of a difference through you can measure some difference in benchmarks one exceptions would be various graphic editing software if you do stuff like editing 8k multi channel video content For compiling you might feel a noticeable difference. It won't be 8x but if you pair it with enough cores and things which have tons of parallel compilation parts it will really feel noticeable faster. But for most gaming you instead of getting 8x the bandwidth you probably benefit more from a speed improvement e.g. running 6000MT/s DDR5. Anyway you can get it today ;) The M2 Ultra has a pretty insane memory bandwidth (and so had the M1 Ultra).
tedunangst 3y ago
When was the last time you saw a GPU with socketed memory?
- fodkodrasz 3y ago
  I had a Tseng ET4000 I think that had socketed memory, and has some Trident TVGA which I beefed up from 512k to 2MB to get 65k colours and 800x600 maybe... I had a GUS soundcard for a while and it also had an EDO slot I think.
ptrwis 3y ago
In DDR5 you have two memory channels per module.
- dathinab 3y ago
  but depending on context you still refer to a pair of 32b DDR5 channels as "a single channel" e.g. if we speak about how many memory channels a x86 desktop, prosumer or server CPU have they count them as a single channel main reason for this is that DDR5 didn't split the channel instead of adding a new channel
foota 3y ago
Aside from what others have said about greater bit width only being useful for large fetches, there's also circuitry necessary to allow more parallel fetches from the L1 to other parts of the memory subsystem (for Intel, these are called line fill buffer) and they're the real limiting factor for single threaded memory bandwidth (at least on that platform). For whole chip bandwidth this isn't as much of an issue, but the things others have said are true there still.
zamadatix 3y ago
I'd be happy with just making each existing slot its own lane.
dathinab 3y ago
The number of slots doesn't matter but the number of channels, i.e. x86 board have 4 slots but only 2 channels (through nomenclature is a bit of a mess as as depending on context channel might mean something else). More memory bandwidth increases the price/size of the IO module on the CPU but I don't think that's a major reason. One of the major reason AFIK is to keep the number of channels low is that it directly maps to the number of pins a socket need to has and the number of high quality lanes which need to be routed independently equally long and chided from inference from e.g. power lines to the CPU. So especially e.g. for server design chip vendors balance the benefit of more memory channels with the cost of it increasing socket size/pins, complexity of mother boards etc. And while on the paper doubling the memory channels doubles the memory bandwidth it doesn't at all means you RAM is twice as fast. It's similar to the difference between duplicating the number of cores vs. duplication the per-core IPC. While all applications profit from more bus width the degree might differ hugely. Oversimplified super highly parallelizeable tasks (e.g. much of GPU tasks and many AI tasks) tend to also be better at taking advantage of more bandwidth through more channels but many other taks less so. This means that the degree to which more channels will yield benefits highly depend on the task the CPU vendor expect is majorly run on it. And the tasks which most benefit from it already tend to be run on GPUs anyway. Apples M1,M2 CPUs one the other hand have a much much smaller cost problem wrt. more memory channels due to how they stack ram on top of the CPU (I think) and they happen to also have much more memory bandwidth by having more bus width. But stacking RAM on top of CPUs has it's own issues. Like for example if the RAM dies the whole CPU is dead and you not being able to upgrade the RAM. Doesn't matter that much for apple, but matters a lot for AMD/Intel which core focus is the server center, not desktop CPUs. Having to replace a 10k CPU and multiple TiB of RAM because a single ram segment fails just isn't acceptable. Additionally for some use cases instead of stacking RAM on the CPU you get as much or more benefit by stacking tons of L3 cache on it (AMD 3D V-cache). There also seems to be in general be more issues with thermal budges, not an issue for apple of which most devices are mostly passive cooled. Issues with manufacturing cost making it harder to provide many CPU spec+RAM amount variants (again doesn't matter for apple but does for Intel/AMD). Potential cost issues making it only suitable for "premium" products (again not an issue for Apple but for Intel/AMD). And maybe more. So basically: - on server (or pro-sumer,i.e. Threadripper) more channels are already used, but it's always a cost benefit balance where there could be even more channels - on desktop, at least currently, it's not seen to be worth the price/complexity increase Through Apples is putting a bit of pressure on the market, but outside of the high end segment it still makes little sense and it's probably not worth it to have it just for the "high end but not pro-sumer" use-case, that market it too small. I would predict prosumer devices like Threadrippers to grow the number of memory channels with every now socket they have. And I do predict a major technology change for "normal desktop computers" 3-6 years down the line which probably will bring more channels to desktop computers. But only then. And it's very speculative.
mastax 3y ago
In addition to what everyone has already said, memory controllers are relatively big and expensive. System integrators often ship systems <~$1000 with only a single channel of memory. It’d probably be like pulling teeth to get them to actually ship a 4-channel configuration. Putting in unused memory controllers is a big waste. People who have many threaded workloads which can actually use all that memory bandwidth tend to get workstation or server platforms.
fanf2 3y ago
The Apple M1 and M2 use HBM to massively increase the memory bandwidth. The new Intel Xeon Max processors have 64GiBytes of HBM in the package, as well as offboard DDR5.
- wtallis 3y ago
  Apple uses LPDDR, which is what's used in all smartphones and most really thin laptops. LPDDR has the exact opposite design goal from HBM: LPDDR is optimized for high bandwidth per pin to enable low pin counts, while HBM relies on advanced silicon interposer packaging to enable extremely high pin counts allowing for modest bandwidth per pin. The only similarity is that Apple puts the LPDDR in the package with the CPU, but it's an entirely different and more mundane packaging technology compared to what is used for HBM.
touisteur 3y ago
I guess we'll know pretty soon, with the availability of the Intel Max with HBM whether it's that simple?

ksec 3y ago

Zen 4, AMD Ryzen 5 7640U, 4.9Ghz, Single Core GB6 Score @ 2340. Apple A17 Pro, 3.8Ghz, Single Core GB6 Score @ 2914 I dont think asking for an 20% increase in IPC is too much at this point.

KingOfCoders 3y ago
AMD seems not interested. Intel will go the Apple way though, simpler cores and trading die area for ever larger caches, also on-die memory, downside you can't upgrade memory, upside it's faster and more power efficient. [Edit] Or someone could have said: I think you meant processor not die, and I would have said: Yes sure, not sure what was on my mind.
- appplication 3y ago
  Upgrading memory is great in theory (and practice for those who do) but I think for 95% of users this is not a consideration.
  - kibwen 3y ago
    It becomes more of a consideration when accounting for the fact that you can buy a Macbook that ships with an un-upgradeable 8 GB of RAM, which, sadly, is deficient for even moderate tasks these days.
  - toast0 3y ago
    Upgradable memory also makes things more flexible for OEMs. Most OEMs continue to sell systems built around last years processor (or even a couple year old processors), and they can increase the memory in those systems as needed. Maybe 4GB was enough when they came out, but now you need 8GB or 16GB to sell a system with an older processor. Not a big deal with socketed RAM, not even a huge deal with soldered RAM, but the processors with 4GB on package become very hard to sell.
- kllrnohj 3y ago
  > Intel will go the Apple way though Hmm, curious what the "Apple way" is here.. > simpler cores Apple's cores are very far from simple. They are much larger than the other ARM CPUs on the market, and this is also why they are faster than other ARM CPUs on the market. > trading die area for ever larger caches Apple has big L1 comparatively, but L2 & L3 are nothing special. Certainly nowhere close to the massive caches that AMD's X3D offers. > also on-die memory Apple doesn't have on-die memory. AMD is the only company to have shipped on-die memory to consumers (they were also the first to ship it), and they only did so on GPUs so far. Nvidia & Intel are playing with it for datacenter products, though, but that's about it (H100 & Xeon Max respectively)

pvg 3y ago

Discussion last week https://news.ycombinator.com/item?id=37715082 https://news.ycombinator.com/item?id=37715082

TheRealPomax 3y ago

It's not a leak unless the leaker proves its origin.

phkahler 3y ago

One thing I never see in pieces like this is comments on how a change might affect hyperthreading performance. It's not relevant for single threaded code, but then neither is doubling the core count from 4 to 8 or 16.

ksec 3y ago
I dont think they care as much as in the case of Multi Core era. It is easier just to sell more cores.
nullc 3y ago
Both intel and AMD's hyperthreading seems pretty lame compared to Power's for some reason. One cause may be that with just two threads the chance that both get blocked on a memory access is pretty good.

snvzz 3y ago

>(fusion) Conditional branching on x86 involves using an instruction that sets flags, and then a branch that jumps (or not) depending on flags. x86 shows its age and accumulated cruft here. RISC-V has test and jump in a single instruction, doing away with flags.

stavros 3y ago
Why can this not be added to x86? Is the ISA immutable now?
- pritambaral 3y ago
  It could be added with an extension, but software would continue to be compiled without it. For many, many years.
  - stavros 3y ago
    Sure, but that's fine, no? Better late than never.
    - pritambaral 3y ago
      Now that op fusion has been implemented for this idiom, the cost of a new instruction does not seem worth the benefit. The benefit would be only a tiny improvement in code size, and that's restricted to when the compiler can be absolutely certain that the target CPU is going to be modern enough to support it. So, in practice, for many, many years, this benefit would be almost never reaped. This is not a new topic in the design of ISAs and their implementations in silicon. Idiom recognition and optimization has long existed in x86, ARM, and I'm sure other old ISA implementations. RISC-V can avoid it by adding dedicated instructions because it's just getting started (relatively speaking), but even so, I'm sure there will be opportunities for idiom recognition and optimization that RISC-V implementers will take.
      - snvzz 3y ago

Moldoteck 3y ago

Can I ask why more manufacturers don't implement apple's strategy with soc with ram and all and offer option to add additional classic ram as a slower cache buffer between soc&ssd? This should both solve the lack of upgradability and unlock more performance per watt?

segfaultex 3y ago
Money. Sounds reductive, but I think oems don’t see much demand for an architecture that implements both. So they’ll go with one or the other.
zamadatix 3y ago
Intel did this is the Sapphire Rapids xeons via 64 GB HBM built in and DDR 5 slots. L4 cache is also not unheard of, having been tried a few times for well of a decade. The downsides are cost/complexity for relatively little gain.
rajnathani 3y ago
Your comment is ambiguous, but if you’re implying that Apple does this then it’s incorrect. They just have “unified memory” (i.e. the high bandwidth SoC substrate memory).
- Moldoteck 3y ago
  Yes, and I'm asking why not allow to add extra ram through slots as a slower cache between unified mem and ssd? And why other manufacturers are not doing this to achieve apple lvl of performanca and also allowing upgradability
  - cma 3y ago
    I thought slots don't really slow things down much and aren't the primary source of latency, isn't it more about cost and power? Adding slots back in would remove the cost and power advantages wouldn't they? And now using more SRAM to track occupancy of an extra layer of cache, or if not SRAM for that something adding more latency.