12 ms·

AMD may get across the CUDA moat

551 points by danzheng 3y ago

alecco 3y ago
Regurgitated months-old content. blogspam
IronWolve 3y ago
Yup, thank the hobbyists. Pytorch is allowing other hardware. Stable diffusion working on m chips, intel arc, and Amd. Now what I'd like to see is real benchmarks for compute power. Might even get a few startups to compete in this new area.
- mattnewton 3y ago
  Re: startups, Geohotz raised a few million for this already. https://tinygrad.org/ https://tinygrad.org/
  - IntelMiner 3y ago
    Didn't he do what he always does. Rake in a ton of money, fart around and then cash out exclaiming it's everyone else's fault? The way he stole Fail0verflow's work with the PS3 security leak after failing to find a hypervisor exploit for months absolutely soured any respect I had for him at the time
    - kinematikk 3y ago
      Do you have a source on the stealing part? A quick Google search didn't result in anything
      - IntelMiner 3y ago
        Marcan (of Asahi Linux fame) has talked about it many times before. But an abridged version Fail0verflow demoed how they were able to derive the private signing keys for the Sony Playstation 3 console at I believe CCC Geohot after watching the livestream raced into action to demo a "hello world!" jailbreak application and absolutely stole their thunder without giving any credit
        aftbit 3y ago
        This apparently worked pretty well for him, as I still remember him primarily as "that guy who hacked PS3". Some people let someone else do the hard technical core, then do all the other easy but boring stuff and claim 100% credit.
        22c 3y ago
        I remember geohot as being one of the people who developed a fairly successful jailbreak for iPhone. I understand that iPhone jailbreaking is often standing on the shoulders of predecessors, but I believe he does deserve significant credit for at least one popular iPhone jailbreak.
        ryanjshaw 3y ago
        If they demod something then they released it publically and it was fair game? In any case he absolutely did credit them, it's easily verifiable: https://web.archive.org/web/20110104040706/http://geohot.com/ https://web.archive.org/web/20110104040706/http://geohot.com... Sony sued them both, afterall!
    - throwitawayfam 3y ago
      Yep, did exactly that. IMO he threw a fit, even though AMD was working with him squashing bugs. https://github.com/RadeonOpenCompute/ROCm/issues/2198#issuecomment-1574383483 https://github.com/RadeonOpenCompute/ROCm/issues/2198#issuec...
      - aeyes 3y ago
        He's back on it after getting AMD's CEO to commit resources to this: https://twitter.com/realGeorgeHotz/status/1669803464082489347 https://twitter.com/realGeorgeHotz/status/166980346408248934... https://twitter.com/LisaSu/status/1669848494637735936 https://twitter.com/LisaSu/status/1669848494637735936
      - deleted 3y ago
        [deleted]
      - nomel 3y ago
        To be fair, kernel crashes from running an AMD provided demo loop isn’t something he should have to work with them on. That’s borderline incompetence. His perspective was around integration into his product, where every AMD bug is a bug in his product. They deserve criticism, and responded accordingly (actual resources to get their shit together). It’s not like GPU accelerated ML is some new thing.
      - JonChesterfield 3y ago
        That's a tough issue to read through, thanks for the link. 'Your demo code on a system setup exactly as you describe dereferences null in the kernel and falls over'. Fuzz testing + a vaguely reasonable kernel debugging workflow should make things like that much harder to find.
    - adastra22 3y ago
      Wow, TIL
    - ShamelessC 3y ago
      > The way he stole Fail0verflow's work with the PS3 security leak after failing to find a hypervisor exploit for months absolutely soured any respect I had for him at the time That sounds interesting. I tried googling about it but can't really find much other than that failoverflow found a key and didn't release it, and then geohot released his own subsequently. I'd love to hear more about how directly he "stole" the work from the Fail0verflow team. edit: Reading some sibling comments here, it seems you are either mistaken and/or were exaggerating your claim about the "theft" here. As far as I can tell, he simply took their findings and made his own version of an exploit that they had detailed publicly. That may be in poor taste in this particular community but it's certainly not theft. I do agree that his behavior there was lacking in decency, but not to the degree implied here where I was thinking he _literally_ stole their exploit by hacking them, or something similar to that.
      - cyrux004 3y ago
        People here generally try to bash people who are much smarter than them, throwing shade at their background. They will say that he abandoned his first company, gave up on tiny grad but both of them are very much alive projects
        deleted 3y ago
        [deleted]
    - deleted 3y ago
      [deleted]
    - georgehotz 3y ago
      Are you interested in being factually correct, or are you interested in hating? If it's the former, I think you should do some research. If it's the latter, :salute: I had GPT-4 do some research for you, hopefully you will incorporate it in future comments you make about me. https://chat.openai.com/share/d0fa24e9-3ed7-4b17-8497-24bfdd2efde0 https://chat.openai.com/share/d0fa24e9-3ed7-4b17-8497-24bfdd...
  - nomel 3y ago
    Obligatory Lex Fridman podcast, where he discusses it: https://youtu.be/dNrTrx42DGQ?t=2408 https://youtu.be/dNrTrx42DGQ?t=2408
- jauntywundrkind 3y ago
  Pytorch is just using Google's OpenXLA now, & OpenXLA is the actual cross platform thing, no? I'm not very well versed in this area, so pardon if mistaken. https://pytorch.org/blog/pytorch-2.0-xla-path-forward/ https://pytorch.org/blog/pytorch-2.0-xla-path-forward/
  - fotcorn 3y ago
    You can use OpenXLA, but it's not the default. The main use-case for OpenXLA is running PyTorch on Google TPUs. OpenXLA also supports GPUs, but I am not sure how many people use that. Afaik JAX uses OpenXLA as backend to run on GPUs. If you use model.compile() in PyTorch, you use TorchInductor and OpenAIs Triton by default.
    - jauntywundrkind 3y ago
      Thank you for saying something useful here. I was vaguely under the impression that pytorch 2.0 had fully flipped to defaulting to openxla. That seems to not be the case. Good to hear more than a cheap snub. OpenAI Triton as the reason other GPUs work is a real non-shit answer, it seems. And interesting to hear JAX too. Thank you for being robustly useful & informative.
  - mathisfun123 3y ago
    > Pytorch is just using Google's OpenXLA now this is so far from accurate it should be considered libelous; from the link > PyTorch/XLA is set to migrate to the open source OpenXLA so PyTorch on the XLA backend is set to migrate to use OpenXLA instead of XLA. but basically everyone moved from XLA to OpenXLA because there is no more OSS XLA. so that's it. in general, PyTorch has several backends, including plenty of homegrown CUDA and CPU kernels. in fact the majority of your PyTorch code runs through PyTorch's own kernels.
  - voz_ 3y ago
    Wrong.
- mandevil 3y ago
  It isn't the hobbyists who are making sure that PyTorch and other frameworks runs well on these chips, but teams of engineers who work for NVIDIA, AMD, Intel, etc. who are doing this as their primary assigned jobs, in exchange for money from their employer, who are paying those salaries because they want to sell chips into the enormous demand for running PyTorch faster. Hobbyist and open-source are definitely not synonyms.
  - Eisenstein 3y ago
    People don't usually get employed to make things with no demand, and people who work for companies with a budget line don't really care how much the nVidia tax is. You can thank hobbyists for creating a lot of demand for compatability with other cards.
    - kiratp 3y ago
      There are so many billions of dollar being spent on this hardware that everyone other than Nvidia is doing everything they can to make competition happen. Eg: https://www.intel.com/content/www/us/en/developer/videos/optimize-dl-workloads-intel-optimized-pytorch.html https://www.intel.com/content/www/us/en/developer/videos/opt... https://www.intel.com/content/www/us/en/developer/tools/oneapi/optimization-for-tensorflow.html https://www.intel.com/content/www/us/en/developer/tools/onea... https://developer.apple.com/metal/tensorflow-plugin/ https://developer.apple.com/metal/tensorflow-plugin/ Large scale opensource is, outside of a few exceptions, built by engineers paid to build it.
    - roenxi 3y ago
      There is huge demand for AMD cards that can efficiently multiply matrices together. The issue is that while there are currently isolated cases where people can make them do that, it doesn't seem to be possible at the scale that it needs to happen at. AMD are being dragged along by the market. Willingly, they aren't fighting it, but their focus has been on other areas.
      - viewtransform 3y ago
superkuh 3y ago
>There is also a version of PyTorch that uses AMD ROCm, an open-source software stack for AMD GPU programming. Crossing the CUDA moat for AMD GPUs may be as easy as using PyTorch. Unfortunately since the AMD firmware doesn't reliably do what it's supposed to those ROCm calls often don't either. That's if your AMD card is even still supported by ROCm: the AMD RX 580 I bought in 2021 (the great GPU shortage) had it's ROCm support dropped in 2022 (4 years support total). The only reliable interface in my experience has been via opencl.
- htrp 3y ago
  has opencl actually improved enough to be competitive?
  - orangepurple 3y ago
    I thought ONNX is supposed to be the ultimate common denominator for machine learning model cross platform compatibility
    - deleted 3y ago
      [deleted]
- zucker42 3y ago
  Do you mean OpenCL using Rusticl or something else? And what DL framework, if any?
  - superkuh 3y ago
    I should clarify that I mean for human person uses. Not commercial or institutional. But, clBLAST via llama.cpp for LLM currently. Or far in the past just pure opencl for things with AMD cards.
- 65a 3y ago
  ROCm works fine on my 2016 Vega Frontier edition, for what it's worth.
binarymax 3y ago
And the question for most that remains once AMD catches up: will the duopoly result in lower prices to a reasonable level for hobbyists or bootstrapped startups, or will AMD just gouge like NVidia?
- klysm 3y ago
  A simplistic economic take would suggest that the competition would result in lower prices, but given two players in the market who knows.
  - binarymax 3y ago
    My intuition is along the lines that if AMD had a competing product earlier, then it would have kept prices down. But since Nvidia has shown what the market will pay, AMD won't be able to resist overcharging. It will probably come down a little, but nowhere near to the point of affordability. I sure hope I'm wrong.
    - tyre 3y ago
      AMD might have to charge less to break into customers that are already bought into Nvidia. There has to be a discount to cover the switching costs + still provide savings (or access).
      - zirgs 3y ago
        AMD will have to provide a REALLY steep discount to convince me to come back.
  - sumtechguy 3y ago
    It is oligopoly pricing. https://www.investopedia.com/terms/o/oligopoly.asp https://www.investopedia.com/terms/o/oligopoly.asp With that few competitors pricing would not change much.
    - AnthonyMouse 3y ago
Zetobal 3y ago
They are just too late even if they catch up. Until they make a leap like they did with ryzen nothing will happen.
- Havoc 3y ago
  >They are just too late even if they catch up. Late certainly, too late I don't think so. If you can field a competitively priced consumer card that can run llama fast then you're already halfway there because then the ecosystem takes off. Especially since nvidia is being really stingy with their vram amounts. H100 & datacenter is a separate battle certainly, but on mindshare I think some deft moves from AMD will get them there quite fast once they pull their finger out their A and actually try sorting out the driver stack.
  - dylan604 3y ago
    >If you can field a competitively priced consumer card if this unicorn were to show up, what's to say that all the non-consumers won't just scarf up these equally performant yet lower priced cards causing the supply-demand situation we're in now? the only difference would be a sudden supply of the expensive Nvidia cards that nobody wants because of their price.
    - AnthonyMouse 3y ago
      The thing that causes it to be competitively priced is having enough production capacity to prevent that from happening. One way to do that may be to produce a card on an older process node (or the existing one when a new one comes out) that has a lot of VRAM. There is less demand for the older node so they can produce more of them and thereby sell them for a lower price without running out.
    - Havoc 3y ago
      >if this unicorn were to show up A unicorn like that showed up a couple hours ago. Someone posted a guide for getting llama to run on a 7900xtx https://old.reddit.com/r/LocalLLaMA/comments/170tghx/guide_installing_rocmhip_for_llamacpp_on_linux/ https://old.reddit.com/r/LocalLLaMA/comments/170tghx/guide_i... It's still slow and janky but this really isn't that far away. I don't buy that AMD can't make this happen if they actually tried. Go on fiverr, get them to compile a list of top 100 people in the DIY LLM space, send them all free 7900XTXs. Doesn't matter if half of it is wrong, just send it. Next take 1.2m USD, post a dozen 100k bounties against llama.cpp that are AMD specific - support & optimise the gear. Rinse and repeat with every other hobbyist LLM/stable diffusion project. A lot of these are zero profit open source / passion / hobby projects. If 6 figure bounties show up it'll absolute raise pulses. Next do all the big youtubers in the space - carefully on that one so that it doesn't come across as an attempted pay-off...but you want them to know that you want this space to grow and are willing to put your money where your mouth is. That'll cost AMD what 2m 3m? To move the needle on a multi billion market? That's the cheapest marketing you've ever seen. As I said the datacenter & enterprise market is another beast entirely full of moats and strategy, but I don't see why a suitably motivated senior AMD exec can't tackle the enthusiast market single handedly with a couple of emails, a cheque book and a tshirt that has the nike slogan on it. >what's to say that all the non-consumers won't just scarf up these equally performant yet lower priced cards It doesn't matter. They're in the business of selling cards. To consumers, to datacenters, to your grandmother. From a profit driven capitalist company the details don't matter as long as there is traction & volume. The above - opening up even the possibility of a new market - is gold in that perspective. And from a consumer perspective anything that breaks the nvidia cuda monopoly is a win.
ddtaylor 3y ago
It's worth noting that AMD also has a ROCm port of Tensorflow.
- ginko 3y ago
  When I try to install rocm-ml-sdk on Arch linux it'll tell me the total installed size would be about 18GB. What can possibly explain this much bloat for what should essentially be a library on top of a graphics driver as well as some tools (compiler, profiler etc.)? A couple hundred MB I could understand if they come with graphical apps and demos, but not this..
  - tomsmeding 3y ago
    A regular TensorFlow installation, just the Python library, is an 184 MB wheel that unpacks to about 1.2 GB of stuff. I have no clue what mess goes in there, but it's a lot. Still, if you're right that this package seems to take 18 GB disk size, something weird is going on.
    - slavik81 3y ago
      There's a lot of kernels that are specialized for particular sets of input parameters and tuned for improved performance on specific hardware, which makes the libraries a couple hundred megabytes per architecture. The ROCm libraries are huge because they are fat binaries containing native machine code for ~13 different GPU architectures.
    - Flameancer 3y ago
      He’s not wrong. I did a new arch install to try and get a 7800XT working with ROCm and PyTorch and was concussed on how I ran out of space but saw that ROCm was 18GB.
pixelesque 3y ago
Does AMD have a solution to forward device combatibility (like PTX for NVidia)? Last time I looked into ROCm (two years ago?), you seemed to have to compile stuff explicitly for the architecture you were using, so if a new card came out, you couldn't use it without a recompile.
- mnau 3y ago
  Not natively, but AdaptiveCpp (previously hiSycl, then OpenSycl) has a single source single compiler pass, where they basically store LLVM IR as an intermediate representation. https://github.com/AdaptiveCpp/AdaptiveCpp/blob/develop/doc/compilation.md https://github.com/AdaptiveCpp/AdaptiveCpp/blob/develop/doc/... Performance penalty was within ew precents, at least according to the paper (figure 9 and 10) https://cdrdv2-public.intel.com/786536/Heidelberg_IWOCL__SYCLCon_2023_paper_2566-1.pdf https://cdrdv2-public.intel.com/786536/Heidelberg_IWOCL__SYC...
- einpoklum 3y ago
  I don't know what they do with ROCm, but with OpenCL, the answer is: Certainly. It's called SPIR: https://www.khronos.org/spir/ https://www.khronos.org/spir/
ur-whale 3y ago
> AMD May Get Across the CUDA Moat I really wish they would, and properly, as in: fully open solution to match CUDA. CUDA is a cancer on the industry.
- mschuetz 3y ago
  What's wrong with CUDA? I avoided it for years because it's proprietory but about one year ago I started using it because all the alternatives (OpenGL/Vulkan compute, OpenCL, WebGPU, ...) couldn't quite do what I wanted, and it turned out to be a game changer. Nothing comes close to it. Now I'm hooked because there simply isn't an alternative that's as easy to use, yet powerfull and fast. I wish there was an open alternative, but NVIDIA did several things right that others, especially Khronos, do not: The UX is top-notch. It makes the common cases easy yet still fast, and from there you can optimize to your hearts content. Khronos, however, usually completely over-engineers things and makes the common case hard and cumbersome with massive entry barriers.
  - ur-whale 3y ago
    > What's wrong with CUDA? Read on > it's proprietory Yes indeed, proprietary > Now I'm hooked There you go. > I wish there was an open alternative So does the rest of the industry. Specifically, it forces you to run your stuff on NVidia hardware and gives you exactly zero guarantee of future support. Good luck trying to reproduce whatever research you are currently conducting in 10 years time. Vendor lock-in + no forward compatibility guarantee = surefire recipe for getting milked to the bone by NVidia.
    - mschuetz 3y ago
      Those are some poor arguments, imho, because there literally is no other option than CUDA. The alternatives are so bad, it's far better to be vendor-locked and being able to get stuff done, than not being able to get stuff done at all. As I said, I avoided it for years because of the reasons you mentioned. Turns out I could not avoid it any longer because it's the only (meaningful) option that could do what I needed, has serious support, and great UX. And NVIDIA is hardly to blame because they simply made sure to build a good product. It can't stop AMD, Intel or Khronos from creating a competitive alternative, but so far they haven't. And regarding support, so far NVIDIA has shown excellent continuous support for CUDA, whereas OpenCL and OpenGL are the ones that went down. And I've chosen CUDA over rocm precisely due to support reasons, because AMD has always treated it as some kind of side gig with uncertain future.
javchz 3y ago
CUDA is the only reason I have an Nvidia card, but if more projects start migrating to a more agnostic environment, I'll be really grateful. Running Nvidia in Linux isn't as much fun. Fedora and Debian can be incredibly reliable systems, but when you add an Nvidia card, I feel like I am back in Windows Vista with kernel crashes from time to time.
- smoldesu 3y ago
  Those problems might just be GNOME-related at this point. I've been daily-driving two different Nvidia cards for ~3 years now (1050 Ti then 3070 Ti) and Wayland has felt pretty stable for the past 12 months. The worst problem I had experienced in that time was Electron and Java apps drawing incorrectly in xWayland, but both of those are fixed upstream. I'm definitely not against better hardware support for AI, but I think your problems are more GNOME's fault than Nvidia's. KDE's Wayland session is almost flawless on Nvidia nowadays.
  - arsome 3y ago
    If GNOME can tank the kernel, it ain't GNOME's fault.
  - kombine 3y ago
    I really hope that with KDE 6 I can finally switch to Wayland!
    - Zardoz84 3y ago
      I'm using KDE on Debian 12 with AMD GPU with Wayland, and works. it keeps being a bit annoying compared with X11 with a few programs (Eclipse, Dbeaver... I need to launch both with flags to not use Wayland backend). But even I can play AAA games without problems
- wubrr 3y ago
  Yeah, nvidia linux support is meh, but still much better than amd.
nabla9 3y ago
> Crossing the CUDA moat for AMD GPUs may be as easy as using PyTorch. Nvidia has spent huge amount of work to make code run smoothly and fast. AMD has to work hard to catch up. ROCm code is slower , has more bugs, don't have enough features and they have compatibility issues between cards.
- latchkey 3y ago
  Lisa has said that they are committed to improving ROCm, especially for AI workloads. Recent releases (5.6/5.7) prove that.
- einpoklum 3y ago
  > Nvidia has spent huge amount of work to make code run smoothly and fast. Well, let's say "smoother" rather than "smoothly". > ROCm code is slower On physically-comparable hardware? Possible, but that's not an easy claim to make, certainly not as expansively as you have. References? > has more bugs Possible, but - NVIDIA keeps their bug database secret. I'm guessing you're concluding this from anecdotal experience? That's fair enough, but then - say so. > ROCm ... don't have enough features and Likely. while AMD has both spent less in that department (and had less to spend I guess); plus, and no less importantly - it tried to go along with the OpenCL initiative, as specified by the Khronos consortium, while NVIDIA has sort of "betrayed" the initiative by investing in it's vendor-locked, incompatible ecosystem and letting their OpenCL support decay in some respects. > they have compatibility issues between cards. such as?
  - kkielhofner 3y ago
    I wouldn’t say ROCm code is “slower”, per se, but in practice that’s how it presents. References: https://github.com/InternLM/lmdeploy https://github.com/InternLM/lmdeploy https://github.com/vllm-project/vllm https://github.com/vllm-project/vllm https://github.com/OpenNMT/CTranslate2 https://github.com/OpenNMT/CTranslate2 You know what’s missing from all of these and many more like them? Support for ROCm. This is all before you get to the really wildly performant stuff like Triton Inference Server, FasterTransformer, TensorRT-LLM, etc. ROCm is at the “get it to work stage” (see top comment, blog posts everywhere celebrating minor successes, etc). CUDA is at the “wring every last penny of performance out of this thing” stage. In terms of hardware support, I think that one is obvious. The U in CUDA originally stood for unified. Look at the list of chips supported by Nvidia drivers and CUDA releases. Literally anything from at least the past 10 years that has Nvidia printed on the box will just run CUDA code. One of my projects specifically targets Pascal up - when I thought even Pascal was a stretch. Cue my surprise when I got a report of someone casually firing it up on Maxwell when I was pretty certain there was no way it could work. A Maxwell laptop chip. It also runs just as well on an H100. THAT is hardware support.
RcouF1uZ4gsC 3y ago
I am not so sure. Everyone knows that CUDA is a core competency of Nvidia and they have stuck to it for years and years refining it, fixing bugs, and making the experience smoother on Nvidia hardware. On the other hand, AMD has not had the same level of commitment. They used to sing the praises of OpenCL. And then there is ROCm. Tomorrow, it might be something else. Thus, Nvidia CUDA will get a lot more attention and tuning from even the portability layers because they know that their investment in it will reap dividends even years from now, whereas their investment in AMD might be obsolete in a few years. In addition, even if there is theoretical support, getting specific driver support and working around driver bugs is likely to be more of a pain with AMD.
- AnthonyMouse 3y ago
  This is what people complain about, but at the same time there aren't enough cards, so the people with AMD cards want to use them. So they fix the bugs, or report them to AMD so they can fix them, and it gets better. Then more people use them and submit patches and bug reporters, and it gets better. At some point the old complaints are no longer valid.
pjmlp 3y ago
Unless they get their act together regarding CUDA polyglot tooling, I seriously doubt it.
bigcat12345678 3y ago
Cuda is the foundation NVIDIA moat is the years of work built by oss community, big corporations, research insistute They spend all time building for cuda, a lot of implicit designs are derived from cuda's characteristic That will be the main challenge
- mikepurvis 3y ago
  It depends on the domain. Increasingly people's interfaces to this stuff are the higher level libraries like tensorflow, pytorch, numpy/cupy, and to a lesser degree accelerated processing libraries such as opencv, PCL, suitesparse, ceres-solver, and friends. If you can add hardware support to a major library and improve on the packaging and deployment front while also undercutting on price, that's the moat gone overnight. CUDA itself only matters in terms of lock-in if you're calling CUDA's own functions.
  - bigcat12345678 3y ago
    what I meant is that all these stuff have 15 years of implicit accumulation of knowledge and tips and even hacks builtin in the software No matter what you depends on, you'll have a slew of larger or minor obstacles or annoyance That collectively is the most itself As you said, already it's clear that replacing cuda itself is not that daunting
omneity 3y ago
I was able to use ROCm recently with Pytorch and after pulling some hair it worked quite well. The Radeon GPU I had on hand was a bit old and underpowered (RDNA2) and it only supported matmul on fp64, but for the job I needed done I saw a 200x increase in it/s over CPU despite the need to cast everywhere, and that made me super happy. Best of all is that I simply set the device to `torch.device('cuda')` rather than openCL, which does wonders for compatibility and to keep code simple. Protip: Use the official ROCM Pytorch base docker image [0]. The AMD setup is so finicky and dependent on specific versions of sdk/drivers/libraries and it will be much harder to make work if you try to install them separately. [0]: https://rocm.docs.amd.com/en/latest/how_to/pytorch_install/pytorch_install.html https://rocm.docs.amd.com/en/latest/how_to/pytorch_install/p...
- mikepurvis 3y ago
  Sigh. It's great that these container images exist to give people an easy on-ramp, but they definitely don't work for every use case (especially once you're in embedded where space matters and you might not be online to pull multi-gb updates from some registry). So it's important that vendors don't feel let off the hook to provide sane packaging just because there's an option to use a kitchen-sink container image they rebuild every day from source.
  - fwsgonzo 3y ago
    I feel the same way, especially about build systems. OpenSSL and v8 are among a large list of things that have horrid build systems. Only way to build them sanely is to use some randos CMake fork, then it Just Works. Literally a two-liner in your build system to add them to your project with a sane CMake script.
    - mikepurvis 3y ago
      I was part of a Nix migration over the past two years, and literally one of the first things we checked is that there was already a community-maintained tensorflow+gpu package in nixpkgs because without that the whole thing would have been a complete non-starter, and we sure as heck didn't have the resources or know-how to figure it out for ourselves as a small DevOps team just trying to do basic packaging.
frnkng 3y ago
As a former ETH miner I learned the hard way that saving a few bucks on hardware may not be worth operational issues. I had a miner running with Nividia cards and a miner running with AMD cards. One of them had massive maintenance demand and the other did not. I will not state which brand was better imho. Currently I estimate that running miners and running gpu servers has similar operational requirements and finally at scale similar financial considerations. So, whatever is cheapest to operate in terms of time expenditure, hw cost, energy use,… will be used the most. P.s.: I ran the mining operation not to earn money but mainly out of curiosity. And it was a small scale business powered by a pv system and a attached heat pump.
- latchkey 3y ago
  I ran 150,000+ AMD cards for mining ETH. Once I fully automated all the vbios installs and individual card tuning, it ran beautifully. Took a lot of work to get there though! Fact is that every single GPU chip is a snowflake. No two operate the same.
  - rottencupcakes 3y ago
    Have you ever written about this enterprise? This sounds super unique and I would be very interested in hearing about how it was run and how it turned out.
    - latchkey 3y ago
      It was unique, not many people on the planet, that I know of, who've run as many GPUs as I have. Especially not working for a giant company with large teams of people. For the tech team, it was just me and one other guy. Everything had to be automated because there was no way we could survive otherwise. I've put a bunch of comments here on HN about the stuff I can talk about. It no longer exists after PoS.
      - freedomben 3y ago
        what type of cards did you have? what did you do with them after PoS? How did you even buy so many cards? Sorry, like the other commenter I'm extremely curious
fluxem 3y ago
I call it the 90% problem. If AMD works for 90% of my projects, I would still buy NVIDIA, which works for 100%, even though I’m paying a premium
- hot_gril 3y ago
  I'm lazy, so it's 99% for me. I don't even mess with AMD CPUs; I know they're not exactly the same instruction set as Intel, and more importantly they work with a different (and less mainstream) set of mobos, so I don't want em. If AMD manages to pull more customers their way, that's great, it just means lower Intel premium for me.
  - deleted 3y ago
    [deleted]
  - 65a 3y ago
    As an owner of some Sapphire Rapids parts, let me just direct you to: https://edc.intel.com/content/www/us/en/design/products-and-solutions/processors-and-chipsets/eagle-stream/sapphire-rapids-specification-update/001US/errata-summary-table/ https://edc.intel.com/content/www/us/en/design/products-and-...
    - hot_gril 3y ago
      To see errata tracked by Intel is a good sign.
  - bornfreddy 3y ago
    That's an interesting take. AMD mobos are no "less mainstream" than Intel ones are... When you choose a CPU you are also choosing a compatible mobo chipset. The companies that make motherboards are mostly the same, so there should be no big difference between those. Also, while the CPU instruction sets are not exactly equal, the same is true for Intel processors of different generations too. And it doesn't matter one bit... Unless there is a bug in CPU you will never notice the difference, because it is taken care of at the compiler / kernel level. Intel does have some advantages (and disadvantages too) over AMD, just not those.
hot_gril 3y ago
People complain about Nvidia being anticompetitive with CUDA, but I don't really see it. They saw a gap in the standards for on-GPU compute and put tons of effort into a proprietary alternative. They tied CUDA to their own hardware, which sorta makes technical sense given the optimizations involved, but it's their choice anyway. They still support the open standards, but many prefer CUDA and will pay the Nvidia premium for it because it's actually nicer. They also don't have CPU marketshare to tie things to. Good for them. We can hope the open side catches up either by improving their standards, or adding more layers like this article describes.
- zirgs 3y ago
  CUDA was released in 2007 and the development of it started even earlier - possibly even in the 90s. Back then nobody else cared about GPU compute. OpenCL came out 2 years after that.
  - killerstorm 3y ago
    Not true. People got interested in general-purpose GPU compute (GPGPU) in early 2000s when video cards with programmable shaders became available. https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units#History https://en.wikipedia.org/wiki/General-purpose_computing_on_g... People made a programming language & a compiler/runtime for GPGPU in 2004: https://en.wikipedia.org/wiki/BrookGPU https://en.wikipedia.org/wiki/BrookGPU
    - hot_gril 3y ago
      Everything has old beginnings that the specialists will remember, but GPU compute really reached mass popularity and became a large selling point for Nvidia in the 2010s.
pama 3y ago
There is only limited empirical evidence of AMD closing the gap that NVidia has created in the science or ML software. Even when considering pytorch only, the engineering effort to maintain specialized ROCm along with CUDA solutions is not trivial (think flashattention, or any customization that optimizes your own model). If your GPUs only need a simple ML workflow all times for a few years nonstop, maybe there exist corner cases where the finances make sense. It is hard for AMD now to close the gap across the scientific/industrial software base of CUDA. NVidia feels like a software company for the hardware they produce; luckily they make the money from hardware thus cannot lock the software libraries. (Edited “no” to limited empirical evidence after a fellow user mentioned El Capitan.)
- Certhas 3y ago
  The fact that El Capitan is AMD says that at least for Science/HPC there definitely is evidence of a closing gap.
  - pama 3y ago
    Thanks. You are actually right that this new supercomputer might move the needle once it is in production mode. I will wait and see how it goes.
- fotcorn 3y ago
  ROCm has HIP (1) which is a compatibility layer to run CUDA code on AMD GPUs. In theory, you only have to adjust #includes, and everything should just work, but as usual, reality is different. Newer backends for AI frameworks like OpenXLA and OpenAI Triton directly generate GPU native code using MLIR and LLVM, they do not use CUDA apart from some glue code to actually load the code onto the GPU and get the data there. Both already support ROCm, but from what I've read the support is not as mature yet compared to NVIDIA. 1: https://github.com/ROCm-Developer-Tools/HIP https://github.com/ROCm-Developer-Tools/HIP
- falconroar 3y ago
  I don't understand why developers of PyTorch and similar don't use OpenCL. Open standard, runs everywhere, similar performance - what's the downside??
whywhywhywhy 3y ago
Anyone who has to work in this ecosystem surely thinks this is a naive take
- freedomben 3y ago
  For someone who doesn't work in this ecosystem, can you elaborate? What's the real situation currently?
  - whywhywhywhy 3y ago
    Nvidia CUDA was first to market, easier to work with that OpenCL which was the only competition for the first decade then abandoned. Because of this then all the people serious about this are using Nvidia hardware therefore all the code is written for Nvidia hardware. Only way I could see AMD making inroads if they were willing to provide power of the level Nvidia puts in a data center at consumer prices and relaxed licensing to justify retooling the entire ML chain to work on a different architecture. Geohot has documented his troubles trying to go all in on AMD and he's back on Nvidia now I believe.
the__alchemist 3y ago
When coding using Vulkan, for graphics or compute (The latter is the relevant one here), you need to have CPU code (Written in C++, Rust etc), then serialize it as bytes, then have shaders which run on the graphics card. This 3-step process creates friction, much in the same way as backend/serialization/frontend does in web dev. Duplication of work, type checking not going across the bridge, the shader language being limited etc. My understanding is CUDA's main strength is avoiding this. Do you agree? Is that why it's such a big deal? Ie, why this article was written, since you could always do compute shaders on AMD etc using Vulkan.
atemerev 3y ago
Nope. PyTorch is not enough, you have to do come C++ occasionally (as the code there can be optimized radically, as we see in llama.cpp and the like). ROCm is unusable compared to CUDA (4x more code for the same problem). I don't understand why everyone neglects good, usable and performant lower-level APIs. ROCm is fast, low-level, but much much harder to use than CUDA, and the market seems to agree.
ris 3y ago
I don't understand the author's argument (if there is one) - pytorch has existed for ages. AMD's Instinct MI* range has existed for years now. If these are the key ingredients why has it not already happened?
jiggawatts 3y ago
Can I buy an MI300 or even rent one in a cloud?
- arcanus 3y ago
  Soon. The card is coming in Q4. The early shipments are likely all going to LLNL's El Capitan Exascale computer: https://www.tomshardware.com/news/amds-instinct-mi300-moves-into-el-capitan-installation https://www.tomshardware.com/news/amds-instinct-mi300-moves-...
einpoklum 3y ago
TL;DR: 1. Since PyTorch has grown very popular, and there's an AMD backend for that, one can switch GPU vendors when doing Generative AI work. 2. Like NVIDIA's Grace+Hopper CPU-GPU combo, AMD is/will be offering "Instinct MI300A", which improves performance over having the GPU across a PCIe bus from a regular CPU.
voz_ 3y ago
The amount of random wrong stuff about pytorch in this thread is pretty funny.
withwarmup 3y ago
CUDA is the result of years of NVIDIA supporting the ecosystem, some people likes to complain because they bought hardware that was cheaper but can't use it for what they want to use it, when you buy NVIDIA, you aren't buying only the hardware, but the insane amount of work they have put into the ecosystem, the same goes for Intel, mkl and scikit-learn intelex aren't free to develop. AMD has the hardware but the support for HPC is non-existent outside of the joke that is bliss and AOCL. I really wish for more competitors to enter the market in HPC, but AMD has a shitload of work to do.
- runiq 3y ago
  Yeah, that's a pretty shortsighted take of things. Do you really believe that Nvidia hasn't taken steps do make sure their moat is as wide as possible?
  - Blammar 3y ago
    The thing about owning the CUDA spec is that Nvidia can add new features quickly without having to argue with other hardware vendors. I find that a positive thing overall. Also, I choose to pay the ~$120 Windows tax once (per box), everything works very well, and I don't have the driver issues that some fraction of other users seem to have with Linux and Nvidia cards. Seems like a good use of my time.
    - anon291 3y ago
      Literally never had an issue with Nvidia and Linux in decades. Despite this, my windows installs have all sorts of issues.. as always
- arcanus 3y ago
  > AMD has the hardware but the support for HPC is non-existent outside of the joke that is bliss and AOCL. You are probably two years behind the state of the art. The world's largest supercomputer, OLCF's Frontier, runs AMD CPUs and GPUs. It's emphatically using ROCm, not just BLIS and AOCL. See for example: https://docs.olcf.ornl.gov/systems/frontier_user_guide.html https://docs.olcf.ornl.gov/systems/frontier_user_guide.html That's hardly non-existent support for HPC.
raggi 3y ago
Can we just get wgsl compute good enough and over the line instead, and do away with these moats?
- mschuetz 3y ago
  Not happening. WGSL wants to support the lowest common denominator, so it'll always mainly be a 5-year old mobile-phone API. Also if you want to beat CUDA, you'll need some functionality that's completely missing in compute shaders, especially WGSL. Like pointers and pointer casting (and that glsl buffer reference extension is the worst emulation of that feature I've every seen).
  - raggi 3y ago
    The language extensions feature is designed to provide these kinds of facilities is it not?
tails4e 3y ago
AMD playing catch up is a good thing, their SW solution is intended to run on any HW, and with hip being basically line for line compatible with cuda it makes porting very easy. They did it with FSR,and they are doing it with rocm. Hopefully it takes off as it's a more open ecosystem for the industry. Necessity is the mother of invention and all that.
spandextwins 3y ago
That’s like saying Ford is gonna catch Tesla.
- cantaloupe 3y ago
  Do you see that as an inevitability or an impossibility?
- tpmx 3y ago
  No, not really. They have similar enough silicon, they "just" need some software to make it work.
tormeh 3y ago
For LLM inference, a shoutout to MLC LLM, which runs LLM models on basically any API that's widely available: https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/mlc-llm
mark_l_watson 3y ago
NVidia hardware/CUDA stack is great, but I also love to see competition from AMD, George Hotz’s Tiny Corp, etc. Off topic, but I am also looking with great interest at Apple Silicon SOCs with large internal RAM. The internal bandwidth also keeps getting better which is important for running trained LLMs. Back on topic: I don’t own any current Intel computers but using Colab and services like Lambda Labs GPU VPSs is simple and flexible. A few people here mentioned if AMD can’t handle 100% of their workload they will stick with Intel and NVidia - understandable position, but there are workarounds.
KingLancelot 3y ago
[dead]
upbeat_general 3y ago
This article doesn’t address the real challenge [in my mind]. Framework support is one thing, but what about the million standalone CUDA kernels that have been written, especially common in research. Nobody wants to spend time re-writing/porting those, especially when they probably don’t understand the low-level details in the first place. Not to mention, what is the plan for comprehensive framework support? I’ve experienced the pain of porting models to different hardware architectures where various ops are unsupported. Is it realistic to get full coverage of e.g., PyTorch?
- blueboo 3y ago
  Research kernels mostly turn to ash upon publication anyway. The wheel turns and the next post-doc gives ROCm a try and we move on
- bdowling 3y ago
  Someone could reimplement CUDA for AMD hardware. That would be legal because copying APIs for compatibility purposes is not copyright infringement. (See Google LLC v. Oracle America, Inc., 593 U.S. ___ (2021)). AMD is unlikely to do this, however, because it would commodify their own products under their competitor’s API. A third party could do it though. It may make sense as an open source project.
Andrew018 3y ago
[dead]
Roark66 3y ago
I think the article claiming "PyTorch has dropped the drawbridge on the CUDA moat" is way over optimistic. Jest pytorch is widely used by researchers and by users to quickly iterate various over various ways to use the models, but when it comes to inference there are huge gains to be had by going a different route. Llama.cpp has showed 10x speedups on my hardware for example (32gb of gpu ram + 32gb of cpu ram)for models like falcon-40b-instruct, for much smaller models on the cpu (under 10b) I saw up to 3x speedup just by switching to onnc and openvino. Apple has showed us in practice the benefits of CPU/GPU memory sharing, will AMD be able to follow in their footsteps? The article claims AMD has a design with up to 192gb of shared ram. Apple is already shipping a design with the same amount of RAM(if you can afford it). I wish them-and) success, but I believe they need to aim higher than just matching apple in some unspecified future.
benreesman 3y ago
I know a lot of people don’t like George, I dislike plenty of people who are doing the right thing thing (including by some measures sama and siebel while they were pushing YC forward). But not admitting the tinygrad project is the best Rebel Alliance on this is just a matter of letting vibe overcome results.
physicsguy 3y ago
Don’t agree at all. PyTorch is one library - yes, it’s important that it supports AMD GPUs but it’s not enough. The ROCm libraries just aren’t good enough currently. The documentation is poor. AMD need to heavily invest in their software ecosystem around it, because library authors need decent support to adopt it. If you need to be a Facebook sized organisation to write an AMD and CUDA compatible library then the barrier to entry is too high.
- weebull 3y ago
  Disagree that the Rocm libraries are poor. Their integration with everything else is poor because everything else is so highly Nvidia centric, and AMD can't just write to the same API because it's copyright Nvidia (see Oracle's Java case). The adoption of CUDA has been such a coop for Nvidia, it's going to take some time to dismantle it.
  - physicsguy 3y ago
    I don’t use high level frameworks like PyTorch because my work is in computational physics so I do actually use the lower level libraries. The documentation doesn’t even come close although it has got better. But they’re just not at feature parity, and that’s not on anyone but AMD currently. They need to invest more in the core libraries. Just look at cuFFT vs rocFFT for e.g… they aren’t even close to being at feature parity - things like multi GPU is totally missing and callbacks are still “experimental”. These are pretty basic features - bear in mind that when people ported from CPU codes CUDA had to support these because they existed in FFTW (transforms over multiple CPUs rather than GPUs though via MPI).
nologic01 3y ago
If the AI hype persists the CUDA moat will be less relevant in ~2 yrs. Historically HPC was simply not sufficiently interesting (in commercial sense) for people to throw serious resources in the direction of making it a mass market capability. NVIDIA first capitalized on the niche crypto industry (which faded) and was then well positioned to jump into the AI hype. The question is how much of the hype will become real business. The critical factor for the post-CUDA world is not any circumstantial moat but who will be making money servicing stable, long term computing needs. I.e., who will be buying this hardware not with speculative hot money but with cashflow from clients that regularly use and pay for a HPC-type application. These actors will be the long term buyers of commercially relevant HPC and they will have quite a bit of influence on this market.
hankman86 3y ago
I suspect that AMD will use their improved compatibility with the leading ML stack for data center deals. Presumably by offering steep discounts over NVIDIA’s GPUs. This might help them to break into the market. Individual ML practitioners will probably not be tempted to switch to AMD cards anytime soon. Whatever the price difference is: it will hardly offset the time that is subsequently sunk into working around remaining issues resulting from a non-CUDA (and less mature) stack underneath PyTorch.
jeffreygoesto 3y ago
I am hoping for SYCL and SPIR-V to gain traction...
sharonzhou 3y ago
ROCm is great. We were able to get run and finetune LLMs on AMD Instincts with parity to NVIDIA A100s - and built an SDK that’s as easy to use as HuggingFace or easier (Lamini). Or at the very least, our designer is able to finetune/train the latest LLMs on them like Llama 2 - 70B and Mistral 7B with ease. The ROCm library isn’t as easy to use as CUDA because as another poster said, the ecosystem was built around CUDA. For example, it’s even called “.cuda()” in PyTorch to put a model on a GPU, when in reality you’d use it for an AMD GPU too.
falconroar 3y ago
Is there any reason OpenCL is not the standard in implementations like PyTorch? Similar performance, open standard, runs everywhere - what's the downside?
- LoganDark 3y ago
  IIRC, ease of implementation (for the GPU kernels), and cross-compatibility (the same bytecode can be loaded by multiple models of GPU).
  - ealloc 3y ago
    How is CUDA-C that much easier than OpenCL? Having ported back and forth myself, the base C-like languages are virtually identical. Just sub "__syncthreads();" for "barrier(CL_MEM_FENCE)" and so on. To me the main problem is that Nvidia hobbles OpenCL on their GPUs by not updating their CL compiler to OpenCL 2.0, so some special features are missing, such as many atomics.
    - LoganDark 3y ago
      Never used it myself, these are just the main reasons I've heard from friends.
  - jacobgorm 3y ago
    The ease of implementation using CUDA means that your code because effed for life, because it is no longer valid C/C++, unless you totally litter it with #ifdefs to special case for CUDA. In my own proprietary AI inference pipeline I've ended up code-generating to a bunch of different backends (OpenCL SpirV, Metal, CUDA, HLSL, CPU w. OpenMP), giving no special treatment to CUDA, and the resulting code is much cleaner and builds with standard open source toolchains.
    - LoganDark 3y ago
      > The ease of implementation using CUDA means that your code because effed for life yes, yes it absolutely does. establishing market dominance as everyone wants to use CUDA but almost nobody wants to write their kernel twice.
ElectronBadger 3y ago
On my PC workstation (Debian Testing) I have absolutely no problems running NVIDIA PNY Quadro P2200, which I'm going to upgrade with PNY Quadro RTX 4000 soon. I'd love to make a switch for AMD Radeon, but the very short (and shrinking) list of ROCm supported cards makes this move highly improbable for the not-so-nearest future.

551 points by danzheng 3y ago

Regurgitated months-old content. blogspam

Yup, thank the hobbyists. Pytorch is allowing other hardware. Stable diffusion working on m chips, intel arc, and Amd. Now what I'd like to see is real benchmarks for compute power. Might even get a few startups to compete in this new area.

mattnewton 3y ago
Re: startups, Geohotz raised a few million for this already. https://tinygrad.org/ https://tinygrad.org/
- IntelMiner 3y ago
  Didn't he do what he always does. Rake in a ton of money, fart around and then cash out exclaiming it's everyone else's fault? The way he stole Fail0verflow's work with the PS3 security leak after failing to find a hypervisor exploit for months absolutely soured any respect I had for him at the time
  - kinematikk 3y ago
    Do you have a source on the stealing part? A quick Google search didn't result in anything
    - IntelMiner 3y ago
      Marcan (of Asahi Linux fame) has talked about it many times before. But an abridged version Fail0verflow demoed how they were able to derive the private signing keys for the Sony Playstation 3 console at I believe CCC Geohot after watching the livestream raced into action to demo a "hello world!" jailbreak application and absolutely stole their thunder without giving any credit
      - aftbit 3y ago
        This apparently worked pretty well for him, as I still remember him primarily as "that guy who hacked PS3". Some people let someone else do the hard technical core, then do all the other easy but boring stuff and claim 100% credit.
        22c 3y ago
        I remember geohot as being one of the people who developed a fairly successful jailbreak for iPhone. I understand that iPhone jailbreaking is often standing on the shoulders of predecessors, but I believe he does deserve significant credit for at least one popular iPhone jailbreak.
      - ryanjshaw 3y ago
        If they demod something then they released it publically and it was fair game? In any case he absolutely did credit them, it's easily verifiable: https://web.archive.org/web/20110104040706/http://geohot.com/ https://web.archive.org/web/20110104040706/http://geohot.com... Sony sued them both, afterall!
  - throwitawayfam 3y ago
    Yep, did exactly that. IMO he threw a fit, even though AMD was working with him squashing bugs. https://github.com/RadeonOpenCompute/ROCm/issues/2198#issuecomment-1574383483 https://github.com/RadeonOpenCompute/ROCm/issues/2198#issuec...
    - aeyes 3y ago
      He's back on it after getting AMD's CEO to commit resources to this: https://twitter.com/realGeorgeHotz/status/1669803464082489347 https://twitter.com/realGeorgeHotz/status/166980346408248934... https://twitter.com/LisaSu/status/1669848494637735936 https://twitter.com/LisaSu/status/1669848494637735936
    - deleted 3y ago
      [deleted]
    - nomel 3y ago
      To be fair, kernel crashes from running an AMD provided demo loop isn’t something he should have to work with them on. That’s borderline incompetence. His perspective was around integration into his product, where every AMD bug is a bug in his product. They deserve criticism, and responded accordingly (actual resources to get their shit together). It’s not like GPU accelerated ML is some new thing.
    - JonChesterfield 3y ago
      That's a tough issue to read through, thanks for the link. 'Your demo code on a system setup exactly as you describe dereferences null in the kernel and falls over'. Fuzz testing + a vaguely reasonable kernel debugging workflow should make things like that much harder to find.
  - adastra22 3y ago
    Wow, TIL
  - ShamelessC 3y ago
    > The way he stole Fail0verflow's work with the PS3 security leak after failing to find a hypervisor exploit for months absolutely soured any respect I had for him at the time That sounds interesting. I tried googling about it but can't really find much other than that failoverflow found a key and didn't release it, and then geohot released his own subsequently. I'd love to hear more about how directly he "stole" the work from the Fail0verflow team. edit: Reading some sibling comments here, it seems you are either mistaken and/or were exaggerating your claim about the "theft" here. As far as I can tell, he simply took their findings and made his own version of an exploit that they had detailed publicly. That may be in poor taste in this particular community but it's certainly not theft. I do agree that his behavior there was lacking in decency, but not to the degree implied here where I was thinking he _literally_ stole their exploit by hacking them, or something similar to that.
    - cyrux004 3y ago
      People here generally try to bash people who are much smarter than them, throwing shade at their background. They will say that he abandoned his first company, gave up on tiny grad but both of them are very much alive projects
      - deleted 3y ago
        [deleted]
  - deleted 3y ago
    [deleted]
  - georgehotz 3y ago
    Are you interested in being factually correct, or are you interested in hating? If it's the former, I think you should do some research. If it's the latter, :salute: I had GPT-4 do some research for you, hopefully you will incorporate it in future comments you make about me. https://chat.openai.com/share/d0fa24e9-3ed7-4b17-8497-24bfdd2efde0 https://chat.openai.com/share/d0fa24e9-3ed7-4b17-8497-24bfdd...
- nomel 3y ago
  Obligatory Lex Fridman podcast, where he discusses it: https://youtu.be/dNrTrx42DGQ?t=2408 https://youtu.be/dNrTrx42DGQ?t=2408
jauntywundrkind 3y ago
Pytorch is just using Google's OpenXLA now, & OpenXLA is the actual cross platform thing, no? I'm not very well versed in this area, so pardon if mistaken. https://pytorch.org/blog/pytorch-2.0-xla-path-forward/ https://pytorch.org/blog/pytorch-2.0-xla-path-forward/
- fotcorn 3y ago
  You can use OpenXLA, but it's not the default. The main use-case for OpenXLA is running PyTorch on Google TPUs. OpenXLA also supports GPUs, but I am not sure how many people use that. Afaik JAX uses OpenXLA as backend to run on GPUs. If you use model.compile() in PyTorch, you use TorchInductor and OpenAIs Triton by default.
  - jauntywundrkind 3y ago
    Thank you for saying something useful here. I was vaguely under the impression that pytorch 2.0 had fully flipped to defaulting to openxla. That seems to not be the case. Good to hear more than a cheap snub. OpenAI Triton as the reason other GPUs work is a real non-shit answer, it seems. And interesting to hear JAX too. Thank you for being robustly useful & informative.
- mathisfun123 3y ago
  > Pytorch is just using Google's OpenXLA now this is so far from accurate it should be considered libelous; from the link > PyTorch/XLA is set to migrate to the open source OpenXLA so PyTorch on the XLA backend is set to migrate to use OpenXLA instead of XLA. but basically everyone moved from XLA to OpenXLA because there is no more OSS XLA. so that's it. in general, PyTorch has several backends, including plenty of homegrown CUDA and CPU kernels. in fact the majority of your PyTorch code runs through PyTorch's own kernels.
- voz_ 3y ago
  Wrong.
mandevil 3y ago
It isn't the hobbyists who are making sure that PyTorch and other frameworks runs well on these chips, but teams of engineers who work for NVIDIA, AMD, Intel, etc. who are doing this as their primary assigned jobs, in exchange for money from their employer, who are paying those salaries because they want to sell chips into the enormous demand for running PyTorch faster. Hobbyist and open-source are definitely not synonyms.
- Eisenstein 3y ago
  People don't usually get employed to make things with no demand, and people who work for companies with a budget line don't really care how much the nVidia tax is. You can thank hobbyists for creating a lot of demand for compatability with other cards.
  - kiratp 3y ago
    There are so many billions of dollar being spent on this hardware that everyone other than Nvidia is doing everything they can to make competition happen. Eg: https://www.intel.com/content/www/us/en/developer/videos/optimize-dl-workloads-intel-optimized-pytorch.html https://www.intel.com/content/www/us/en/developer/videos/opt... https://www.intel.com/content/www/us/en/developer/tools/oneapi/optimization-for-tensorflow.html https://www.intel.com/content/www/us/en/developer/tools/onea... https://developer.apple.com/metal/tensorflow-plugin/ https://developer.apple.com/metal/tensorflow-plugin/ Large scale opensource is, outside of a few exceptions, built by engineers paid to build it.
  - roenxi 3y ago
    There is huge demand for AMD cards that can efficiently multiply matrices together. The issue is that while there are currently isolated cases where people can make them do that, it doesn't seem to be possible at the scale that it needs to happen at. AMD are being dragged along by the market. Willingly, they aren't fighting it, but their focus has been on other areas.
    - viewtransform 3y ago

superkuh 3y ago

>There is also a version of PyTorch that uses AMD ROCm, an open-source software stack for AMD GPU programming. Crossing the CUDA moat for AMD GPUs may be as easy as using PyTorch. Unfortunately since the AMD firmware doesn't reliably do what it's supposed to those ROCm calls often don't either. That's if your AMD card is even still supported by ROCm: the AMD RX 580 I bought in 2021 (the great GPU shortage) had it's ROCm support dropped in 2022 (4 years support total). The only reliable interface in my experience has been via opencl.

htrp 3y ago
has opencl actually improved enough to be competitive?
- orangepurple 3y ago
  I thought ONNX is supposed to be the ultimate common denominator for machine learning model cross platform compatibility
  - deleted 3y ago
    [deleted]
zucker42 3y ago
Do you mean OpenCL using Rusticl or something else? And what DL framework, if any?
- superkuh 3y ago
  I should clarify that I mean for human person uses. Not commercial or institutional. But, clBLAST via llama.cpp for LLM currently. Or far in the past just pure opencl for things with AMD cards.
65a 3y ago
ROCm works fine on my 2016 Vega Frontier edition, for what it's worth.

binarymax 3y ago

And the question for most that remains once AMD catches up: will the duopoly result in lower prices to a reasonable level for hobbyists or bootstrapped startups, or will AMD just gouge like NVidia?

klysm 3y ago
A simplistic economic take would suggest that the competition would result in lower prices, but given two players in the market who knows.
- binarymax 3y ago
  My intuition is along the lines that if AMD had a competing product earlier, then it would have kept prices down. But since Nvidia has shown what the market will pay, AMD won't be able to resist overcharging. It will probably come down a little, but nowhere near to the point of affordability. I sure hope I'm wrong.
  - tyre 3y ago
    AMD might have to charge less to break into customers that are already bought into Nvidia. There has to be a discount to cover the switching costs + still provide savings (or access).
    - zirgs 3y ago
      AMD will have to provide a REALLY steep discount to convince me to come back.
- sumtechguy 3y ago
  It is oligopoly pricing. https://www.investopedia.com/terms/o/oligopoly.asp https://www.investopedia.com/terms/o/oligopoly.asp With that few competitors pricing would not change much.
  - AnthonyMouse 3y ago

Zetobal 3y ago

They are just too late even if they catch up. Until they make a leap like they did with ryzen nothing will happen.

Havoc 3y ago
>They are just too late even if they catch up. Late certainly, too late I don't think so. If you can field a competitively priced consumer card that can run llama fast then you're already halfway there because then the ecosystem takes off. Especially since nvidia is being really stingy with their vram amounts. H100 & datacenter is a separate battle certainly, but on mindshare I think some deft moves from AMD will get them there quite fast once they pull their finger out their A and actually try sorting out the driver stack.
- dylan604 3y ago
  >If you can field a competitively priced consumer card if this unicorn were to show up, what's to say that all the non-consumers won't just scarf up these equally performant yet lower priced cards causing the supply-demand situation we're in now? the only difference would be a sudden supply of the expensive Nvidia cards that nobody wants because of their price.
  - AnthonyMouse 3y ago
    The thing that causes it to be competitively priced is having enough production capacity to prevent that from happening. One way to do that may be to produce a card on an older process node (or the existing one when a new one comes out) that has a lot of VRAM. There is less demand for the older node so they can produce more of them and thereby sell them for a lower price without running out.
  - Havoc 3y ago
    >if this unicorn were to show up A unicorn like that showed up a couple hours ago. Someone posted a guide for getting llama to run on a 7900xtx https://old.reddit.com/r/LocalLLaMA/comments/170tghx/guide_installing_rocmhip_for_llamacpp_on_linux/ https://old.reddit.com/r/LocalLLaMA/comments/170tghx/guide_i... It's still slow and janky but this really isn't that far away. I don't buy that AMD can't make this happen if they actually tried. Go on fiverr, get them to compile a list of top 100 people in the DIY LLM space, send them all free 7900XTXs. Doesn't matter if half of it is wrong, just send it. Next take 1.2m USD, post a dozen 100k bounties against llama.cpp that are AMD specific - support & optimise the gear. Rinse and repeat with every other hobbyist LLM/stable diffusion project. A lot of these are zero profit open source / passion / hobby projects. If 6 figure bounties show up it'll absolute raise pulses. Next do all the big youtubers in the space - carefully on that one so that it doesn't come across as an attempted pay-off...but you want them to know that you want this space to grow and are willing to put your money where your mouth is. That'll cost AMD what 2m 3m? To move the needle on a multi billion market? That's the cheapest marketing you've ever seen. As I said the datacenter & enterprise market is another beast entirely full of moats and strategy, but I don't see why a suitably motivated senior AMD exec can't tackle the enthusiast market single handedly with a couple of emails, a cheque book and a tshirt that has the nike slogan on it. >what's to say that all the non-consumers won't just scarf up these equally performant yet lower priced cards It doesn't matter. They're in the business of selling cards. To consumers, to datacenters, to your grandmother. From a profit driven capitalist company the details don't matter as long as there is traction & volume. The above - opening up even the possibility of a new market - is gold in that perspective. And from a consumer perspective anything that breaks the nvidia cuda monopoly is a win.

ddtaylor 3y ago

It's worth noting that AMD also has a ROCm port of Tensorflow.

ginko 3y ago
When I try to install rocm-ml-sdk on Arch linux it'll tell me the total installed size would be about 18GB. What can possibly explain this much bloat for what should essentially be a library on top of a graphics driver as well as some tools (compiler, profiler etc.)? A couple hundred MB I could understand if they come with graphical apps and demos, but not this..
- tomsmeding 3y ago
  A regular TensorFlow installation, just the Python library, is an 184 MB wheel that unpacks to about 1.2 GB of stuff. I have no clue what mess goes in there, but it's a lot. Still, if you're right that this package seems to take 18 GB disk size, something weird is going on.
  - slavik81 3y ago
    There's a lot of kernels that are specialized for particular sets of input parameters and tuned for improved performance on specific hardware, which makes the libraries a couple hundred megabytes per architecture. The ROCm libraries are huge because they are fat binaries containing native machine code for ~13 different GPU architectures.
  - Flameancer 3y ago
    He’s not wrong. I did a new arch install to try and get a 7800XT working with ROCm and PyTorch and was concussed on how I ran out of space but saw that ROCm was 18GB.

pixelesque 3y ago

Does AMD have a solution to forward device combatibility (like PTX for NVidia)? Last time I looked into ROCm (two years ago?), you seemed to have to compile stuff explicitly for the architecture you were using, so if a new card came out, you couldn't use it without a recompile.

mnau 3y ago
Not natively, but AdaptiveCpp (previously hiSycl, then OpenSycl) has a single source single compiler pass, where they basically store LLVM IR as an intermediate representation. https://github.com/AdaptiveCpp/AdaptiveCpp/blob/develop/doc/compilation.md https://github.com/AdaptiveCpp/AdaptiveCpp/blob/develop/doc/... Performance penalty was within ew precents, at least according to the paper (figure 9 and 10) https://cdrdv2-public.intel.com/786536/Heidelberg_IWOCL__SYCLCon_2023_paper_2566-1.pdf https://cdrdv2-public.intel.com/786536/Heidelberg_IWOCL__SYC...
einpoklum 3y ago
I don't know what they do with ROCm, but with OpenCL, the answer is: Certainly. It's called SPIR: https://www.khronos.org/spir/ https://www.khronos.org/spir/

ur-whale 3y ago

> AMD May Get Across the CUDA Moat I really wish they would, and properly, as in: fully open solution to match CUDA. CUDA is a cancer on the industry.

mschuetz 3y ago
What's wrong with CUDA? I avoided it for years because it's proprietory but about one year ago I started using it because all the alternatives (OpenGL/Vulkan compute, OpenCL, WebGPU, ...) couldn't quite do what I wanted, and it turned out to be a game changer. Nothing comes close to it. Now I'm hooked because there simply isn't an alternative that's as easy to use, yet powerfull and fast. I wish there was an open alternative, but NVIDIA did several things right that others, especially Khronos, do not: The UX is top-notch. It makes the common cases easy yet still fast, and from there you can optimize to your hearts content. Khronos, however, usually completely over-engineers things and makes the common case hard and cumbersome with massive entry barriers.
- ur-whale 3y ago
  > What's wrong with CUDA? Read on > it's proprietory Yes indeed, proprietary > Now I'm hooked There you go. > I wish there was an open alternative So does the rest of the industry. Specifically, it forces you to run your stuff on NVidia hardware and gives you exactly zero guarantee of future support. Good luck trying to reproduce whatever research you are currently conducting in 10 years time. Vendor lock-in + no forward compatibility guarantee = surefire recipe for getting milked to the bone by NVidia.
  - mschuetz 3y ago
    Those are some poor arguments, imho, because there literally is no other option than CUDA. The alternatives are so bad, it's far better to be vendor-locked and being able to get stuff done, than not being able to get stuff done at all. As I said, I avoided it for years because of the reasons you mentioned. Turns out I could not avoid it any longer because it's the only (meaningful) option that could do what I needed, has serious support, and great UX. And NVIDIA is hardly to blame because they simply made sure to build a good product. It can't stop AMD, Intel or Khronos from creating a competitive alternative, but so far they haven't. And regarding support, so far NVIDIA has shown excellent continuous support for CUDA, whereas OpenCL and OpenGL are the ones that went down. And I've chosen CUDA over rocm precisely due to support reasons, because AMD has always treated it as some kind of side gig with uncertain future.

javchz 3y ago

CUDA is the only reason I have an Nvidia card, but if more projects start migrating to a more agnostic environment, I'll be really grateful. Running Nvidia in Linux isn't as much fun. Fedora and Debian can be incredibly reliable systems, but when you add an Nvidia card, I feel like I am back in Windows Vista with kernel crashes from time to time.

smoldesu 3y ago
Those problems might just be GNOME-related at this point. I've been daily-driving two different Nvidia cards for ~3 years now (1050 Ti then 3070 Ti) and Wayland has felt pretty stable for the past 12 months. The worst problem I had experienced in that time was Electron and Java apps drawing incorrectly in xWayland, but both of those are fixed upstream. I'm definitely not against better hardware support for AI, but I think your problems are more GNOME's fault than Nvidia's. KDE's Wayland session is almost flawless on Nvidia nowadays.
- arsome 3y ago
  If GNOME can tank the kernel, it ain't GNOME's fault.
- kombine 3y ago
  I really hope that with KDE 6 I can finally switch to Wayland!
  - Zardoz84 3y ago
    I'm using KDE on Debian 12 with AMD GPU with Wayland, and works. it keeps being a bit annoying compared with X11 with a few programs (Eclipse, Dbeaver... I need to launch both with flags to not use Wayland backend). But even I can play AAA games without problems
wubrr 3y ago
Yeah, nvidia linux support is meh, but still much better than amd.

nabla9 3y ago

> Crossing the CUDA moat for AMD GPUs may be as easy as using PyTorch. Nvidia has spent huge amount of work to make code run smoothly and fast. AMD has to work hard to catch up. ROCm code is slower , has more bugs, don't have enough features and they have compatibility issues between cards.

latchkey 3y ago
Lisa has said that they are committed to improving ROCm, especially for AI workloads. Recent releases (5.6/5.7) prove that.
einpoklum 3y ago
> Nvidia has spent huge amount of work to make code run smoothly and fast. Well, let's say "smoother" rather than "smoothly". > ROCm code is slower On physically-comparable hardware? Possible, but that's not an easy claim to make, certainly not as expansively as you have. References? > has more bugs Possible, but - NVIDIA keeps their bug database secret. I'm guessing you're concluding this from anecdotal experience? That's fair enough, but then - say so. > ROCm ... don't have enough features and Likely. while AMD has both spent less in that department (and had less to spend I guess); plus, and no less importantly - it tried to go along with the OpenCL initiative, as specified by the Khronos consortium, while NVIDIA has sort of "betrayed" the initiative by investing in it's vendor-locked, incompatible ecosystem and letting their OpenCL support decay in some respects. > they have compatibility issues between cards. such as?
- kkielhofner 3y ago
  I wouldn’t say ROCm code is “slower”, per se, but in practice that’s how it presents. References: https://github.com/InternLM/lmdeploy https://github.com/InternLM/lmdeploy https://github.com/vllm-project/vllm https://github.com/vllm-project/vllm https://github.com/OpenNMT/CTranslate2 https://github.com/OpenNMT/CTranslate2 You know what’s missing from all of these and many more like them? Support for ROCm. This is all before you get to the really wildly performant stuff like Triton Inference Server, FasterTransformer, TensorRT-LLM, etc. ROCm is at the “get it to work stage” (see top comment, blog posts everywhere celebrating minor successes, etc). CUDA is at the “wring every last penny of performance out of this thing” stage. In terms of hardware support, I think that one is obvious. The U in CUDA originally stood for unified. Look at the list of chips supported by Nvidia drivers and CUDA releases. Literally anything from at least the past 10 years that has Nvidia printed on the box will just run CUDA code. One of my projects specifically targets Pascal up - when I thought even Pascal was a stretch. Cue my surprise when I got a report of someone casually firing it up on Maxwell when I was pretty certain there was no way it could work. A Maxwell laptop chip. It also runs just as well on an H100. THAT is hardware support.

RcouF1uZ4gsC 3y ago

I am not so sure. Everyone knows that CUDA is a core competency of Nvidia and they have stuck to it for years and years refining it, fixing bugs, and making the experience smoother on Nvidia hardware. On the other hand, AMD has not had the same level of commitment. They used to sing the praises of OpenCL. And then there is ROCm. Tomorrow, it might be something else. Thus, Nvidia CUDA will get a lot more attention and tuning from even the portability layers because they know that their investment in it will reap dividends even years from now, whereas their investment in AMD might be obsolete in a few years. In addition, even if there is theoretical support, getting specific driver support and working around driver bugs is likely to be more of a pain with AMD.

AnthonyMouse 3y ago
This is what people complain about, but at the same time there aren't enough cards, so the people with AMD cards want to use them. So they fix the bugs, or report them to AMD so they can fix them, and it gets better. Then more people use them and submit patches and bug reporters, and it gets better. At some point the old complaints are no longer valid.

pjmlp 3y ago

Unless they get their act together regarding CUDA polyglot tooling, I seriously doubt it.

bigcat12345678 3y ago

Cuda is the foundation NVIDIA moat is the years of work built by oss community, big corporations, research insistute They spend all time building for cuda, a lot of implicit designs are derived from cuda's characteristic That will be the main challenge

mikepurvis 3y ago
It depends on the domain. Increasingly people's interfaces to this stuff are the higher level libraries like tensorflow, pytorch, numpy/cupy, and to a lesser degree accelerated processing libraries such as opencv, PCL, suitesparse, ceres-solver, and friends. If you can add hardware support to a major library and improve on the packaging and deployment front while also undercutting on price, that's the moat gone overnight. CUDA itself only matters in terms of lock-in if you're calling CUDA's own functions.
- bigcat12345678 3y ago
  what I meant is that all these stuff have 15 years of implicit accumulation of knowledge and tips and even hacks builtin in the software No matter what you depends on, you'll have a slew of larger or minor obstacles or annoyance That collectively is the most itself As you said, already it's clear that replacing cuda itself is not that daunting

omneity 3y ago

I was able to use ROCm recently with Pytorch and after pulling some hair it worked quite well. The Radeon GPU I had on hand was a bit old and underpowered (RDNA2) and it only supported matmul on fp64, but for the job I needed done I saw a 200x increase in it/s over CPU despite the need to cast everywhere, and that made me super happy. Best of all is that I simply set the device to `torch.device('cuda')` rather than openCL, which does wonders for compatibility and to keep code simple. Protip: Use the official ROCM Pytorch base docker image [0]. The AMD setup is so finicky and dependent on specific versions of sdk/drivers/libraries and it will be much harder to make work if you try to install them separately. [0]: https://rocm.docs.amd.com/en/latest/how_to/pytorch_install/pytorch_install.html https://rocm.docs.amd.com/en/latest/how_to/pytorch_install/p...

mikepurvis 3y ago
Sigh. It's great that these container images exist to give people an easy on-ramp, but they definitely don't work for every use case (especially once you're in embedded where space matters and you might not be online to pull multi-gb updates from some registry). So it's important that vendors don't feel let off the hook to provide sane packaging just because there's an option to use a kitchen-sink container image they rebuild every day from source.
- fwsgonzo 3y ago
  I feel the same way, especially about build systems. OpenSSL and v8 are among a large list of things that have horrid build systems. Only way to build them sanely is to use some randos CMake fork, then it Just Works. Literally a two-liner in your build system to add them to your project with a sane CMake script.
  - mikepurvis 3y ago
    I was part of a Nix migration over the past two years, and literally one of the first things we checked is that there was already a community-maintained tensorflow+gpu package in nixpkgs because without that the whole thing would have been a complete non-starter, and we sure as heck didn't have the resources or know-how to figure it out for ourselves as a small DevOps team just trying to do basic packaging.

frnkng 3y ago

As a former ETH miner I learned the hard way that saving a few bucks on hardware may not be worth operational issues. I had a miner running with Nividia cards and a miner running with AMD cards. One of them had massive maintenance demand and the other did not. I will not state which brand was better imho. Currently I estimate that running miners and running gpu servers has similar operational requirements and finally at scale similar financial considerations. So, whatever is cheapest to operate in terms of time expenditure, hw cost, energy use,… will be used the most. P.s.: I ran the mining operation not to earn money but mainly out of curiosity. And it was a small scale business powered by a pv system and a attached heat pump.

latchkey 3y ago
I ran 150,000+ AMD cards for mining ETH. Once I fully automated all the vbios installs and individual card tuning, it ran beautifully. Took a lot of work to get there though! Fact is that every single GPU chip is a snowflake. No two operate the same.
- rottencupcakes 3y ago
  Have you ever written about this enterprise? This sounds super unique and I would be very interested in hearing about how it was run and how it turned out.
  - latchkey 3y ago
    It was unique, not many people on the planet, that I know of, who've run as many GPUs as I have. Especially not working for a giant company with large teams of people. For the tech team, it was just me and one other guy. Everything had to be automated because there was no way we could survive otherwise. I've put a bunch of comments here on HN about the stuff I can talk about. It no longer exists after PoS.
    - freedomben 3y ago
      what type of cards did you have? what did you do with them after PoS? How did you even buy so many cards? Sorry, like the other commenter I'm extremely curious

fluxem 3y ago

I call it the 90% problem. If AMD works for 90% of my projects, I would still buy NVIDIA, which works for 100%, even though I’m paying a premium

hot_gril 3y ago
I'm lazy, so it's 99% for me. I don't even mess with AMD CPUs; I know they're not exactly the same instruction set as Intel, and more importantly they work with a different (and less mainstream) set of mobos, so I don't want em. If AMD manages to pull more customers their way, that's great, it just means lower Intel premium for me.
- deleted 3y ago
  [deleted]
- 65a 3y ago
  As an owner of some Sapphire Rapids parts, let me just direct you to: https://edc.intel.com/content/www/us/en/design/products-and-solutions/processors-and-chipsets/eagle-stream/sapphire-rapids-specification-update/001US/errata-summary-table/ https://edc.intel.com/content/www/us/en/design/products-and-...
  - hot_gril 3y ago
    To see errata tracked by Intel is a good sign.
- bornfreddy 3y ago
  That's an interesting take. AMD mobos are no "less mainstream" than Intel ones are... When you choose a CPU you are also choosing a compatible mobo chipset. The companies that make motherboards are mostly the same, so there should be no big difference between those. Also, while the CPU instruction sets are not exactly equal, the same is true for Intel processors of different generations too. And it doesn't matter one bit... Unless there is a bug in CPU you will never notice the difference, because it is taken care of at the compiler / kernel level. Intel does have some advantages (and disadvantages too) over AMD, just not those.

hot_gril 3y ago

People complain about Nvidia being anticompetitive with CUDA, but I don't really see it. They saw a gap in the standards for on-GPU compute and put tons of effort into a proprietary alternative. They tied CUDA to their own hardware, which sorta makes technical sense given the optimizations involved, but it's their choice anyway. They still support the open standards, but many prefer CUDA and will pay the Nvidia premium for it because it's actually nicer. They also don't have CPU marketshare to tie things to. Good for them. We can hope the open side catches up either by improving their standards, or adding more layers like this article describes.

zirgs 3y ago
CUDA was released in 2007 and the development of it started even earlier - possibly even in the 90s. Back then nobody else cared about GPU compute. OpenCL came out 2 years after that.
- killerstorm 3y ago
  Not true. People got interested in general-purpose GPU compute (GPGPU) in early 2000s when video cards with programmable shaders became available. https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units#History https://en.wikipedia.org/wiki/General-purpose_computing_on_g... People made a programming language & a compiler/runtime for GPGPU in 2004: https://en.wikipedia.org/wiki/BrookGPU https://en.wikipedia.org/wiki/BrookGPU
  - hot_gril 3y ago
    Everything has old beginnings that the specialists will remember, but GPU compute really reached mass popularity and became a large selling point for Nvidia in the 2010s.

pama 3y ago

There is only limited empirical evidence of AMD closing the gap that NVidia has created in the science or ML software. Even when considering pytorch only, the engineering effort to maintain specialized ROCm along with CUDA solutions is not trivial (think flashattention, or any customization that optimizes your own model). If your GPUs only need a simple ML workflow all times for a few years nonstop, maybe there exist corner cases where the finances make sense. It is hard for AMD now to close the gap across the scientific/industrial software base of CUDA. NVidia feels like a software company for the hardware they produce; luckily they make the money from hardware thus cannot lock the software libraries. (Edited “no” to limited empirical evidence after a fellow user mentioned El Capitan.)

Certhas 3y ago
The fact that El Capitan is AMD says that at least for Science/HPC there definitely is evidence of a closing gap.
- pama 3y ago
  Thanks. You are actually right that this new supercomputer might move the needle once it is in production mode. I will wait and see how it goes.
fotcorn 3y ago
ROCm has HIP (1) which is a compatibility layer to run CUDA code on AMD GPUs. In theory, you only have to adjust #includes, and everything should just work, but as usual, reality is different. Newer backends for AI frameworks like OpenXLA and OpenAI Triton directly generate GPU native code using MLIR and LLVM, they do not use CUDA apart from some glue code to actually load the code onto the GPU and get the data there. Both already support ROCm, but from what I've read the support is not as mature yet compared to NVIDIA. 1: https://github.com/ROCm-Developer-Tools/HIP https://github.com/ROCm-Developer-Tools/HIP
falconroar 3y ago
I don't understand why developers of PyTorch and similar don't use OpenCL. Open standard, runs everywhere, similar performance - what's the downside??

whywhywhywhy 3y ago

Anyone who has to work in this ecosystem surely thinks this is a naive take

freedomben 3y ago
For someone who doesn't work in this ecosystem, can you elaborate? What's the real situation currently?
- whywhywhywhy 3y ago
  Nvidia CUDA was first to market, easier to work with that OpenCL which was the only competition for the first decade then abandoned. Because of this then all the people serious about this are using Nvidia hardware therefore all the code is written for Nvidia hardware. Only way I could see AMD making inroads if they were willing to provide power of the level Nvidia puts in a data center at consumer prices and relaxed licensing to justify retooling the entire ML chain to work on a different architecture. Geohot has documented his troubles trying to go all in on AMD and he's back on Nvidia now I believe.

the__alchemist 3y ago

When coding using Vulkan, for graphics or compute (The latter is the relevant one here), you need to have CPU code (Written in C++, Rust etc), then serialize it as bytes, then have shaders which run on the graphics card. This 3-step process creates friction, much in the same way as backend/serialization/frontend does in web dev. Duplication of work, type checking not going across the bridge, the shader language being limited etc. My understanding is CUDA's main strength is avoiding this. Do you agree? Is that why it's such a big deal? Ie, why this article was written, since you could always do compute shaders on AMD etc using Vulkan.

atemerev 3y ago

Nope. PyTorch is not enough, you have to do come C++ occasionally (as the code there can be optimized radically, as we see in llama.cpp and the like). ROCm is unusable compared to CUDA (4x more code for the same problem). I don't understand why everyone neglects good, usable and performant lower-level APIs. ROCm is fast, low-level, but much much harder to use than CUDA, and the market seems to agree.

ris 3y ago

I don't understand the author's argument (if there is one) - pytorch has existed for ages. AMD's Instinct MI* range has existed for years now. If these are the key ingredients why has it not already happened?

jiggawatts 3y ago

Can I buy an MI300 or even rent one in a cloud?

arcanus 3y ago
Soon. The card is coming in Q4. The early shipments are likely all going to LLNL's El Capitan Exascale computer: https://www.tomshardware.com/news/amds-instinct-mi300-moves-into-el-capitan-installation https://www.tomshardware.com/news/amds-instinct-mi300-moves-...

einpoklum 3y ago

TL;DR: 1. Since PyTorch has grown very popular, and there's an AMD backend for that, one can switch GPU vendors when doing Generative AI work. 2. Like NVIDIA's Grace+Hopper CPU-GPU combo, AMD is/will be offering "Instinct MI300A", which improves performance over having the GPU across a PCIe bus from a regular CPU.

voz_ 3y ago

The amount of random wrong stuff about pytorch in this thread is pretty funny.

withwarmup 3y ago

CUDA is the result of years of NVIDIA supporting the ecosystem, some people likes to complain because they bought hardware that was cheaper but can't use it for what they want to use it, when you buy NVIDIA, you aren't buying only the hardware, but the insane amount of work they have put into the ecosystem, the same goes for Intel, mkl and scikit-learn intelex aren't free to develop. AMD has the hardware but the support for HPC is non-existent outside of the joke that is bliss and AOCL. I really wish for more competitors to enter the market in HPC, but AMD has a shitload of work to do.

runiq 3y ago
Yeah, that's a pretty shortsighted take of things. Do you really believe that Nvidia hasn't taken steps do make sure their moat is as wide as possible?
- Blammar 3y ago
  The thing about owning the CUDA spec is that Nvidia can add new features quickly without having to argue with other hardware vendors. I find that a positive thing overall. Also, I choose to pay the ~$120 Windows tax once (per box), everything works very well, and I don't have the driver issues that some fraction of other users seem to have with Linux and Nvidia cards. Seems like a good use of my time.
  - anon291 3y ago
    Literally never had an issue with Nvidia and Linux in decades. Despite this, my windows installs have all sorts of issues.. as always
arcanus 3y ago
> AMD has the hardware but the support for HPC is non-existent outside of the joke that is bliss and AOCL. You are probably two years behind the state of the art. The world's largest supercomputer, OLCF's Frontier, runs AMD CPUs and GPUs. It's emphatically using ROCm, not just BLIS and AOCL. See for example: https://docs.olcf.ornl.gov/systems/frontier_user_guide.html https://docs.olcf.ornl.gov/systems/frontier_user_guide.html That's hardly non-existent support for HPC.

raggi 3y ago

Can we just get wgsl compute good enough and over the line instead, and do away with these moats?

mschuetz 3y ago
Not happening. WGSL wants to support the lowest common denominator, so it'll always mainly be a 5-year old mobile-phone API. Also if you want to beat CUDA, you'll need some functionality that's completely missing in compute shaders, especially WGSL. Like pointers and pointer casting (and that glsl buffer reference extension is the worst emulation of that feature I've every seen).
- raggi 3y ago
  The language extensions feature is designed to provide these kinds of facilities is it not?

tails4e 3y ago

AMD playing catch up is a good thing, their SW solution is intended to run on any HW, and with hip being basically line for line compatible with cuda it makes porting very easy. They did it with FSR,and they are doing it with rocm. Hopefully it takes off as it's a more open ecosystem for the industry. Necessity is the mother of invention and all that.

spandextwins 3y ago

That’s like saying Ford is gonna catch Tesla.

cantaloupe 3y ago
Do you see that as an inevitability or an impossibility?
tpmx 3y ago
No, not really. They have similar enough silicon, they "just" need some software to make it work.

tormeh 3y ago

For LLM inference, a shoutout to MLC LLM, which runs LLM models on basically any API that's widely available: https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/mlc-llm

mark_l_watson 3y ago

NVidia hardware/CUDA stack is great, but I also love to see competition from AMD, George Hotz’s Tiny Corp, etc. Off topic, but I am also looking with great interest at Apple Silicon SOCs with large internal RAM. The internal bandwidth also keeps getting better which is important for running trained LLMs. Back on topic: I don’t own any current Intel computers but using Colab and services like Lambda Labs GPU VPSs is simple and flexible. A few people here mentioned if AMD can’t handle 100% of their workload they will stick with Intel and NVidia - understandable position, but there are workarounds.

KingLancelot 3y ago

[dead]

upbeat_general 3y ago

This article doesn’t address the real challenge [in my mind]. Framework support is one thing, but what about the million standalone CUDA kernels that have been written, especially common in research. Nobody wants to spend time re-writing/porting those, especially when they probably don’t understand the low-level details in the first place. Not to mention, what is the plan for comprehensive framework support? I’ve experienced the pain of porting models to different hardware architectures where various ops are unsupported. Is it realistic to get full coverage of e.g., PyTorch?

blueboo 3y ago
Research kernels mostly turn to ash upon publication anyway. The wheel turns and the next post-doc gives ROCm a try and we move on
bdowling 3y ago
Someone could reimplement CUDA for AMD hardware. That would be legal because copying APIs for compatibility purposes is not copyright infringement. (See Google LLC v. Oracle America, Inc., 593 U.S. ___ (2021)). AMD is unlikely to do this, however, because it would commodify their own products under their competitor’s API. A third party could do it though. It may make sense as an open source project.

Andrew018 3y ago

[dead]

Roark66 3y ago

I think the article claiming "PyTorch has dropped the drawbridge on the CUDA moat" is way over optimistic. Jest pytorch is widely used by researchers and by users to quickly iterate various over various ways to use the models, but when it comes to inference there are huge gains to be had by going a different route. Llama.cpp has showed 10x speedups on my hardware for example (32gb of gpu ram + 32gb of cpu ram)for models like falcon-40b-instruct, for much smaller models on the cpu (under 10b) I saw up to 3x speedup just by switching to onnc and openvino. Apple has showed us in practice the benefits of CPU/GPU memory sharing, will AMD be able to follow in their footsteps? The article claims AMD has a design with up to 192gb of shared ram. Apple is already shipping a design with the same amount of RAM(if you can afford it). I wish them-and) success, but I believe they need to aim higher than just matching apple in some unspecified future.

benreesman 3y ago

I know a lot of people don’t like George, I dislike plenty of people who are doing the right thing thing (including by some measures sama and siebel while they were pushing YC forward). But not admitting the tinygrad project is the best Rebel Alliance on this is just a matter of letting vibe overcome results.

physicsguy 3y ago

Don’t agree at all. PyTorch is one library - yes, it’s important that it supports AMD GPUs but it’s not enough. The ROCm libraries just aren’t good enough currently. The documentation is poor. AMD need to heavily invest in their software ecosystem around it, because library authors need decent support to adopt it. If you need to be a Facebook sized organisation to write an AMD and CUDA compatible library then the barrier to entry is too high.

weebull 3y ago
Disagree that the Rocm libraries are poor. Their integration with everything else is poor because everything else is so highly Nvidia centric, and AMD can't just write to the same API because it's copyright Nvidia (see Oracle's Java case). The adoption of CUDA has been such a coop for Nvidia, it's going to take some time to dismantle it.
- physicsguy 3y ago
  I don’t use high level frameworks like PyTorch because my work is in computational physics so I do actually use the lower level libraries. The documentation doesn’t even come close although it has got better. But they’re just not at feature parity, and that’s not on anyone but AMD currently. They need to invest more in the core libraries. Just look at cuFFT vs rocFFT for e.g… they aren’t even close to being at feature parity - things like multi GPU is totally missing and callbacks are still “experimental”. These are pretty basic features - bear in mind that when people ported from CPU codes CUDA had to support these because they existed in FFTW (transforms over multiple CPUs rather than GPUs though via MPI).

nologic01 3y ago

If the AI hype persists the CUDA moat will be less relevant in ~2 yrs. Historically HPC was simply not sufficiently interesting (in commercial sense) for people to throw serious resources in the direction of making it a mass market capability. NVIDIA first capitalized on the niche crypto industry (which faded) and was then well positioned to jump into the AI hype. The question is how much of the hype will become real business. The critical factor for the post-CUDA world is not any circumstantial moat but who will be making money servicing stable, long term computing needs. I.e., who will be buying this hardware not with speculative hot money but with cashflow from clients that regularly use and pay for a HPC-type application. These actors will be the long term buyers of commercially relevant HPC and they will have quite a bit of influence on this market.

hankman86 3y ago

I suspect that AMD will use their improved compatibility with the leading ML stack for data center deals. Presumably by offering steep discounts over NVIDIA’s GPUs. This might help them to break into the market. Individual ML practitioners will probably not be tempted to switch to AMD cards anytime soon. Whatever the price difference is: it will hardly offset the time that is subsequently sunk into working around remaining issues resulting from a non-CUDA (and less mature) stack underneath PyTorch.

jeffreygoesto 3y ago

I am hoping for SYCL and SPIR-V to gain traction...

sharonzhou 3y ago

ROCm is great. We were able to get run and finetune LLMs on AMD Instincts with parity to NVIDIA A100s - and built an SDK that’s as easy to use as HuggingFace or easier (Lamini). Or at the very least, our designer is able to finetune/train the latest LLMs on them like Llama 2 - 70B and Mistral 7B with ease. The ROCm library isn’t as easy to use as CUDA because as another poster said, the ecosystem was built around CUDA. For example, it’s even called “.cuda()” in PyTorch to put a model on a GPU, when in reality you’d use it for an AMD GPU too.

falconroar 3y ago

Is there any reason OpenCL is not the standard in implementations like PyTorch? Similar performance, open standard, runs everywhere - what's the downside?

LoganDark 3y ago
IIRC, ease of implementation (for the GPU kernels), and cross-compatibility (the same bytecode can be loaded by multiple models of GPU).
- ealloc 3y ago
  How is CUDA-C that much easier than OpenCL? Having ported back and forth myself, the base C-like languages are virtually identical. Just sub "__syncthreads();" for "barrier(CL_MEM_FENCE)" and so on. To me the main problem is that Nvidia hobbles OpenCL on their GPUs by not updating their CL compiler to OpenCL 2.0, so some special features are missing, such as many atomics.
  - LoganDark 3y ago
    Never used it myself, these are just the main reasons I've heard from friends.
- jacobgorm 3y ago
  The ease of implementation using CUDA means that your code because effed for life, because it is no longer valid C/C++, unless you totally litter it with #ifdefs to special case for CUDA. In my own proprietary AI inference pipeline I've ended up code-generating to a bunch of different backends (OpenCL SpirV, Metal, CUDA, HLSL, CPU w. OpenMP), giving no special treatment to CUDA, and the resulting code is much cleaner and builds with standard open source toolchains.
  - LoganDark 3y ago
    > The ease of implementation using CUDA means that your code because effed for life yes, yes it absolutely does. establishing market dominance as everyone wants to use CUDA but almost nobody wants to write their kernel twice.

ElectronBadger 3y ago

On my PC workstation (Debian Testing) I have absolutely no problems running NVIDIA PNY Quadro P2200, which I'm going to upgrade with PNY Quadro RTX 4000 soon. I'd love to make a switch for AMD Radeon, but the very short (and shrinking) list of ROCm supported cards makes this move highly improbable for the not-so-nearest future.