6 ms·

A bug that taught me more about PyTorch than years of using it

465 points by bblcla 11mo ago

modeless 11mo ago
Another reason people use Nvidia. You know that Nvidia is the most used backend and the most likely to have this kind of bug found and fixed before you encounter it.
brilee 11mo ago
Great write-up, but I admit that I found the interweaving of human and AI-written content/headlines/summaries pretty distracting. I kept on wanting to scroll past, but had to keep on backtracking to find the human thread again. I think if you want to give your reader a quick intro to, e.g., what is the Adam optimizer, a simple link to Wikipedia is fine. No need to copy-paste an AI tutorial on Adam into the blog post.
- CaptainOfCoit 11mo ago
  To be fair, you can easily click to hide those expanded sections. I found it a neat compromise between "Link to (usually) obtuse Wikipedia article" which aren't usually written for laypersons, and forcing me to read through stuff I already know about, I just hid the sections I already understood but found value in the others.
- reilly3000 11mo ago
  I came here to say the same thing. Claude’s voice was pretty evident, but became actually grating when the header was “The Fix”.
cadamsdotcom 11mo ago
Sounds like Placeholder should somehow be split into InputPlaceholder and OutputPlaceholder, based on the usage. Even identical classes could help future folks know copying back is platform specific: “hm, we wrote to an OutputPlaceholder but didn’t read back from it, that seems wrong”.
- ramses0 11mo ago
  Apps Hungarian v. System Hungarian: https://herbsutter.com/2008/07/15/hungarian-notation-is-clearly-goodbad/ https://herbsutter.com/2008/07/15/hungarian-notation-is-clea...
kccqzy 11mo ago
This is a minor quibble but I don't really like the author calling Placeholder a leaky abstraction. It's just straight up an incomplete abstraction that only handles inputs but not outputs. As the author says, Placeholder should know about the difference and do the copy-back itself.
airza 11mo ago
I too have been insanely burned by an MPS bug. I wish Apple would throw an engineer or two at making sure their hardware works with PyTorch.
montebicyclelo 11mo ago
Incorrect Pytorch gradients with Apple MPS backend... Yep this kind of thing can happen. I found and reported incorrect gradients for Apple's Metal-backed tensorflow conv2d in 2021 [1]. (Pretty sure I've seen incorrect gradients with another Pytorch backend, but that was a few years ago and I don't seem to have raised an issue to refer to... ) One might think this class of errors would be caught by a test suite. Autodiff can be tested quite comprehensively against numerical differentiation [2]. (Although this example is from a much simpler lib than Pytorch, so I could be missing something.) [1] https://github.com/apple/tensorflow_macos/issues/230 https://github.com/apple/tensorflow_macos/issues/230 [2] https://github.com/sradc/SmallPebble/blob/2cd915c4ba72bf2d92350401da2ab891c9098727/smallpebble/tests/test_smallpebble.py https://github.com/sradc/SmallPebble/blob/2cd915c4ba72bf2d92...
- liuliu 11mo ago
  Yeah, luckily, you can unit tests these and fix them. They are not concurrency bugs (again, luckily). BTW, numeric differentiation can only be tested very limitedly (due to algorithmic complexity when you doing big matrix). It is much easier / effective to test against multiple implementations.
  - deleted 11mo ago
    [deleted]
  - antoine-levitt 11mo ago
    You can easily test a gradient using only the forward pass by doing f(x+h) ~ f(x) + dot(g, h) for a random h
- gcr 11mo ago
  I’ve also found that some versions of torch get quite different inference results on MPS, ignoring gradient. See https://gist.github.com/gcr/4d8833bb63a85fc8ef1fd77de6622770 https://gist.github.com/gcr/4d8833bb63a85fc8ef1fd77de6622770
CaptainOfCoit 11mo ago
Only slightly related, but how common are bugs in GPUs and/or CUDA? I'm currently on Day 5 of trying to debug why my GPT-OSS implementation (not using PyTorch) I've made from scratch isn't working correctly, and while I have it somewhat working with some naive and slow methods, I'm now doing an implementation of the tensor cores and have been just stuck for 2-3 days because of some small numerical difference I can't understand why it's happening. Every day I'm getting closer to believing this is some sort of hardware bug in Blackwell or in CUDA itself, but as we know, the bug is (almost) never in the compiler or in the hardware. Until it is...
- QuadmasterXLII 11mo ago
  You may be running into jensen (huang)’s inequality, E(loss).cuda() <= E(loss.cuda())
  - CaptainOfCoit 11mo ago
    Would make sense I suppose if I was using two different GPUs for the same thing and get two different outcomes. But instead I have two implementations (one naive, one tensor cores) running on the same GPU, but getting different outcomes, where they should be the same. But then this joke might be flying above my head as well.
    - p1esk 11mo ago
      Tensor cores use lower precision, so small numerical differences should be expected.
    - deleted 11mo ago
      [deleted]
- saagarjha 11mo ago
  How big is the numerical difference? If it's small it might be within the precision of the operation itself.
  - 11mo ago
gugagore 11mo ago
This is the first time I see "SGD" to mean "standard gradient descent" and not "stochastic gradient descent".
- tavianator 11mo ago
  Presumably that's just a mistake. The author calls it "stochastic gradient descent" correctly elsewhere in the article
- elanapearl 11mo ago
  haha oops yeah the other comment is correct- that was just a mistake I originally wrote "vanilla" there but didn't want to repeat that word twice in a row so swapped it for "standard" without realizing it now looked like the SGD acronym just fixed that to avoid confusion- thanks for pointing it out!
saagarjha 11mo ago
Non-contiguous tensors have to be the #1 source of bugs in PyTorch lol
jebarker 11mo ago
This is a great write up and I’d love to see more like it. Debugging this sort of thing in the megatron->pytorch->CUDA stack is what my team spends more than half of their time on as an ML research team.
- ddelnano 11mo ago
  Wouldn't the Nsight Systems suite provide coverage here? Are the tricky cases difficult to debug with the standard CUDA tooling stack?
  - jebarker 11mo ago
    Yes, nsys is very helpful, especially when looking at perf issues. It’s often the case that bugs present like in this blog though - you just notice that training curves have regressed somehow - so even with good tooling it can be hard to figure out where to start looking in these very complex systems. Only gets worse if the symptoms only show up when running for a long time and at scale in a cluster.
hobom 11mo ago
What a fantastic way to write a post mortem, pedagogically very useful.
dangoodmanUT 11mo ago
The tinygrad folks talk about this a lot. Not that I understand much of what they say, but it appears there are a lot of correctness bugs in pytorch that are flying under the radar, probably having a measurable impact on the results of model quality. It would be interesting to see model weights comparison of the same model trained with the two to see if they exhibit meaningfully different behavior.
- CaptainOfCoit 11mo ago
  > Not that I understand much of what they say, but it appears there are a lot of correctness bugs in pytorch that are flying under the radar, probably having a measurable impact on the results of model quality. Do you have any links to public thoughts about this? As if it was true, could mean a lot of research could be invalidated, so obviously would make huge news. Also feels like something that would be relatively easy to make reproducible test cases from, so easy to prove if that's true or not. And finally if something is easy to validate, and would make huge news, I feel like someone would already have attempted to prove this, and if it was true, would have published something a long time ago.
  - dangoodmanUT 11mo ago
    Check their Twitter, I saw something either yesterday or earlier today iirc
  - Calavar 11mo ago
    There are many more ways to degrade model performance than to enhance it, so I would expect the vast majority of bugs to lead to artificially reduced accuracy, not artificially increased accuracy. So if PyTorch is full of numerical flaws, that would likely mean many models with mediocre/borderline performance were discarded (never published) because they just failed to meet the threshold where the authors felt it was worth their time to package it up for a mid-tier conference. A finding that many would-be mediocre papers are actually slightly less mediocre than believed would be an utterly unremarkable conclusion and I believe that's why we haven't seen a bombshell analysis of PyTorch flaws and reproducibility at NeurIPS. A software error in, say, a stats routine or a data preprocessing routine would be a different story because the degrees of freedom are fewer, leaving a greater probability of an error hitting a path that pushes a result to look artificially better as opposed to artificially worse
dataflow 11mo ago
Dumb question: why isn't there some kind of assertion to sanity-check some bits of the GPU results against CPU's?
nraynaud 11mo ago
Naive question: ML tensor libraries don’t use a Z-order memory layout like textures do? It’s not beneficial like it is for textures?
- matusp 11mo ago
  I think that z-order is used to increase speed of loading texture from RAM. But this is not an issue in ML. You usually have all your model weights directly loaded into your GPU memory and you do not need caching for your inputs. At the same time, the entire stack for ML is heavily optimized for other memory layouts already.
hinkley 11mo ago
Reminds me of the largest AJAX app I worked on, back when jquery was still hot and IE6 still existed as a problem. The landing page in our app used jqueryUI’s drag and drop support, back around the time they declared bankruptcy on the confusing buggy code and wouldn’t even accept bug fixes because they were replacing it component by component (which was taking almost 3x as long as predicted). We had columns you could drag items between but they had a max height and scroll bars and it turned out jqueryUI would let you drag items into different rows if the overflow area for adjacent drag targets overlapped your row. The person who found it couldn’t fix it. The other fixer couldn’t fix it. I diagnosed it but the spaghetti code was a recursive mess and I could not find a spot where I could fix it. Especially given I couldn’t send in a patch to them. So I spent half of my free time the last day of every (2 week) sprint for almost six months before I finally found a small function I could monkey patch to wrap it in a short circuit check for clipping region. I spent maybe 20,30 hours on this, a lot of it just getting back to the same situation to debug. But it felt like it took forever to fix it. The short circuit also made drag and drop faster, which was just getting in the edge of distracting. Particularly on a crowded page.
- CaptainOfCoit 11mo ago
  I remember many similar cycles of having different browsers open side-by-side, and trying to pinpoint (without the developer tools we know and love today) the exact reason why one border was one pixel in one browser, and two pixels in the other, throwing the whole layout off. Also remembering when Firebug for Firefox appeared, and made so many things so much easier. Suddenly things that took hours took days, and it was so much easier when you had some introspection tools.
  - yard2010 11mo ago
    * { border: red 1px solid } Remember when IE6 was a thing? The kids today are angry at chrome for good reasons and yet, there was a time in which the most popular browser didn't implement jack shit from the specs. And it was the kind of browser that ships with the OS. God the bad karma for working with this crap. I'm glad it's over.
ipsum2 11mo ago
Apple used to contribute to the PyTorch MPS backend, but decided to create their own framework (MLX) instead, fragmenting the ecosystem for very little gain. (MLX is basically PyTorch, but invented-at-apple) Meta, the creator and main contributor to PyTorch, does not use Macs for their day-to-day ML work (they focus on GPUs and CPUs), so the MPS backend is sadly incomplete and has errors like the one you see here.
- almostgotcaught 11mo ago
  none of this is correct (except the part where FB doesn't use apple in prod). EDIT: for the downvoters - i'll repeat, this is not a correct assessment of the relationship between Apple and PyTorch. but you can keep downvoting if you want <shrug>
  - ipsum2 11mo ago
    Please be specific if you have anything to say. By the way, the co-creator and core maintainer of PyTorch has the same opinion as me. https://x.com/soumithchintala/status/1978848796953161754 https://x.com/soumithchintala/status/1978848796953161754 "MacStudio you ask? Apple Engineering's *actual* time spent on PyTorch support has't given me confidence that PyTorch Mac experience would get anywhere close to NVIDIA's any time soon, if ever. The Meta engineers continue to do a huge amount of heavy-lifting for improving the MPS backend, including feeling the responsibility for the Mac experience. Apple's priorities keep changing, the number of engineering hours they contribute keeps changing and their interest in actually and wholly owning the PyTorch MPS backend keeps varying. If Apple wants MacStudio to become an actual AI devbox, and not just an AI inference machine, then prioritizing software support for PyTorch (>90% marketshare in AI) would probably be a good idea."
    - hedgehog 11mo ago
      Apple has never cared about ML research on their hardware. I've never been able to pin down a specific reason why, best I can figure out is they don't see it bringing enough additional hardware sales to be a focus.
mirekrusin 11mo ago
Nice work, surprising, I'd imagine implementations are cross tested all the time and this kind of bugs have no way of appearing?
cryber 11mo ago
this is a great writeup! methodical without being pedantic.
hershyb_ 11mo ago
awesome read!
anal_reactor 11mo ago
If I understand correctly, the root cause of the bug was improper use of object-oriented programming. A `Placeholder` object behaves differently depending on how it was created, and requires the user to have this awareness. The check `if is_continuous` should only ever exist inside the code of the `Placeholder` class.
albertzeyer 11mo ago
The bug was with non-contiguous data in tensors. I also had a very similar bug a while ago, broken gradients due to non-contiguous data for masked_select: https://github.com/pytorch/pytorch/issues/99638 https://github.com/pytorch/pytorch/issues/99638 In my case, it was easier to identify: I had another implementation of my loss function before that did not use masked_select. But then I thought I can be clever and use masked_select to take out the non-masked frames and calculate the loss only on those. But it wasn't working. Also, it only happened for some models, not for all. It turns out, it was always happening when the data coming out of the model was non-contiguous. I think the bugs with non-contiguous data are not so uncommon. I wonder how much of that we still have.
dcl 11mo ago
Is this why I cannot seem to fine tune YOLO models on a Apple M4? The loss hits nan after a few batches. Same code using Windows PC and Google Colab CPU and GPU is fine...
EdwardDiego 11mo ago
Kudos to Elana for a) such a thorough deep dive and b) a great write-up of it. I understand very little about ML libraries, but was able to follow this easily :)
farhanhubble 11mo ago
Great work hunting the bug down the stack. The writeup is top notch. I wish I documented some of the nastiest bugs I found in such detail. Funnily, only a few days ago I was thinking about just how far the field has come since 2014 or so when you'd build a computational graph, initialize weights manually and so on, versus now, where you just have to use a library like Ultralytics or HuggingFace most of the time. Then I thought about just how many deep, undetected bugs there would be in this mountain of abstraction. Bugs that make the computation invalid.
Rileyen 11mo ago
Just read the article and it instantly brought back memories of when I spent days trying to fix a broken loss in a PyTorch model. Turned out I had passed the wrong optimizer parameters. I ended up digging all the way from the model to the CUDA kernel. Debugging took longer than training. What’s the trickiest bug you’ve ever run into?

465 points by bblcla 11mo ago

Another reason people use Nvidia. You know that Nvidia is the most used backend and the most likely to have this kind of bug found and fixed before you encounter it.

brilee 11mo ago

Great write-up, but I admit that I found the interweaving of human and AI-written content/headlines/summaries pretty distracting. I kept on wanting to scroll past, but had to keep on backtracking to find the human thread again. I think if you want to give your reader a quick intro to, e.g., what is the Adam optimizer, a simple link to Wikipedia is fine. No need to copy-paste an AI tutorial on Adam into the blog post.

CaptainOfCoit 11mo ago
To be fair, you can easily click to hide those expanded sections. I found it a neat compromise between "Link to (usually) obtuse Wikipedia article" which aren't usually written for laypersons, and forcing me to read through stuff I already know about, I just hid the sections I already understood but found value in the others.
reilly3000 11mo ago
I came here to say the same thing. Claude’s voice was pretty evident, but became actually grating when the header was “The Fix”.

cadamsdotcom 11mo ago

Sounds like Placeholder should somehow be split into InputPlaceholder and OutputPlaceholder, based on the usage. Even identical classes could help future folks know copying back is platform specific: “hm, we wrote to an OutputPlaceholder but didn’t read back from it, that seems wrong”.

ramses0 11mo ago
Apps Hungarian v. System Hungarian: https://herbsutter.com/2008/07/15/hungarian-notation-is-clearly-goodbad/ https://herbsutter.com/2008/07/15/hungarian-notation-is-clea...

kccqzy 11mo ago

This is a minor quibble but I don't really like the author calling Placeholder a leaky abstraction. It's just straight up an incomplete abstraction that only handles inputs but not outputs. As the author says, Placeholder should know about the difference and do the copy-back itself.

airza 11mo ago

I too have been insanely burned by an MPS bug. I wish Apple would throw an engineer or two at making sure their hardware works with PyTorch.

montebicyclelo 11mo ago

Incorrect Pytorch gradients with Apple MPS backend... Yep this kind of thing can happen. I found and reported incorrect gradients for Apple's Metal-backed tensorflow conv2d in 2021 [1]. (Pretty sure I've seen incorrect gradients with another Pytorch backend, but that was a few years ago and I don't seem to have raised an issue to refer to... ) One might think this class of errors would be caught by a test suite. Autodiff can be tested quite comprehensively against numerical differentiation [2]. (Although this example is from a much simpler lib than Pytorch, so I could be missing something.) [1] https://github.com/apple/tensorflow_macos/issues/230 https://github.com/apple/tensorflow_macos/issues/230 [2] https://github.com/sradc/SmallPebble/blob/2cd915c4ba72bf2d92350401da2ab891c9098727/smallpebble/tests/test_smallpebble.py https://github.com/sradc/SmallPebble/blob/2cd915c4ba72bf2d92...

liuliu 11mo ago
Yeah, luckily, you can unit tests these and fix them. They are not concurrency bugs (again, luckily). BTW, numeric differentiation can only be tested very limitedly (due to algorithmic complexity when you doing big matrix). It is much easier / effective to test against multiple implementations.
- deleted 11mo ago
  [deleted]
- antoine-levitt 11mo ago
  You can easily test a gradient using only the forward pass by doing f(x+h) ~ f(x) + dot(g, h) for a random h
gcr 11mo ago
I’ve also found that some versions of torch get quite different inference results on MPS, ignoring gradient. See https://gist.github.com/gcr/4d8833bb63a85fc8ef1fd77de6622770 https://gist.github.com/gcr/4d8833bb63a85fc8ef1fd77de6622770

CaptainOfCoit 11mo ago

Only slightly related, but how common are bugs in GPUs and/or CUDA? I'm currently on Day 5 of trying to debug why my GPT-OSS implementation (not using PyTorch) I've made from scratch isn't working correctly, and while I have it somewhat working with some naive and slow methods, I'm now doing an implementation of the tensor cores and have been just stuck for 2-3 days because of some small numerical difference I can't understand why it's happening. Every day I'm getting closer to believing this is some sort of hardware bug in Blackwell or in CUDA itself, but as we know, the bug is (almost) never in the compiler or in the hardware. Until it is...

QuadmasterXLII 11mo ago
You may be running into jensen (huang)’s inequality, E(loss).cuda() <= E(loss.cuda())
- CaptainOfCoit 11mo ago
  Would make sense I suppose if I was using two different GPUs for the same thing and get two different outcomes. But instead I have two implementations (one naive, one tensor cores) running on the same GPU, but getting different outcomes, where they should be the same. But then this joke might be flying above my head as well.
  - p1esk 11mo ago
    Tensor cores use lower precision, so small numerical differences should be expected.
  - deleted 11mo ago
    [deleted]
saagarjha 11mo ago
How big is the numerical difference? If it's small it might be within the precision of the operation itself.
- 11mo ago

gugagore 11mo ago

This is the first time I see "SGD" to mean "standard gradient descent" and not "stochastic gradient descent".

tavianator 11mo ago
Presumably that's just a mistake. The author calls it "stochastic gradient descent" correctly elsewhere in the article
elanapearl 11mo ago
haha oops yeah the other comment is correct- that was just a mistake I originally wrote "vanilla" there but didn't want to repeat that word twice in a row so swapped it for "standard" without realizing it now looked like the SGD acronym just fixed that to avoid confusion- thanks for pointing it out!

saagarjha 11mo ago

Non-contiguous tensors have to be the #1 source of bugs in PyTorch lol

jebarker 11mo ago

This is a great write up and I’d love to see more like it. Debugging this sort of thing in the megatron->pytorch->CUDA stack is what my team spends more than half of their time on as an ML research team.

ddelnano 11mo ago
Wouldn't the Nsight Systems suite provide coverage here? Are the tricky cases difficult to debug with the standard CUDA tooling stack?
- jebarker 11mo ago
  Yes, nsys is very helpful, especially when looking at perf issues. It’s often the case that bugs present like in this blog though - you just notice that training curves have regressed somehow - so even with good tooling it can be hard to figure out where to start looking in these very complex systems. Only gets worse if the symptoms only show up when running for a long time and at scale in a cluster.

hobom 11mo ago

What a fantastic way to write a post mortem, pedagogically very useful.

dangoodmanUT 11mo ago

The tinygrad folks talk about this a lot. Not that I understand much of what they say, but it appears there are a lot of correctness bugs in pytorch that are flying under the radar, probably having a measurable impact on the results of model quality. It would be interesting to see model weights comparison of the same model trained with the two to see if they exhibit meaningfully different behavior.

CaptainOfCoit 11mo ago
> Not that I understand much of what they say, but it appears there are a lot of correctness bugs in pytorch that are flying under the radar, probably having a measurable impact on the results of model quality. Do you have any links to public thoughts about this? As if it was true, could mean a lot of research could be invalidated, so obviously would make huge news. Also feels like something that would be relatively easy to make reproducible test cases from, so easy to prove if that's true or not. And finally if something is easy to validate, and would make huge news, I feel like someone would already have attempted to prove this, and if it was true, would have published something a long time ago.
- dangoodmanUT 11mo ago
  Check their Twitter, I saw something either yesterday or earlier today iirc
- Calavar 11mo ago
  There are many more ways to degrade model performance than to enhance it, so I would expect the vast majority of bugs to lead to artificially reduced accuracy, not artificially increased accuracy. So if PyTorch is full of numerical flaws, that would likely mean many models with mediocre/borderline performance were discarded (never published) because they just failed to meet the threshold where the authors felt it was worth their time to package it up for a mid-tier conference. A finding that many would-be mediocre papers are actually slightly less mediocre than believed would be an utterly unremarkable conclusion and I believe that's why we haven't seen a bombshell analysis of PyTorch flaws and reproducibility at NeurIPS. A software error in, say, a stats routine or a data preprocessing routine would be a different story because the degrees of freedom are fewer, leaving a greater probability of an error hitting a path that pushes a result to look artificially better as opposed to artificially worse

dataflow 11mo ago

Dumb question: why isn't there some kind of assertion to sanity-check some bits of the GPU results against CPU's?

nraynaud 11mo ago

Naive question: ML tensor libraries don’t use a Z-order memory layout like textures do? It’s not beneficial like it is for textures?

matusp 11mo ago
I think that z-order is used to increase speed of loading texture from RAM. But this is not an issue in ML. You usually have all your model weights directly loaded into your GPU memory and you do not need caching for your inputs. At the same time, the entire stack for ML is heavily optimized for other memory layouts already.

hinkley 11mo ago

Reminds me of the largest AJAX app I worked on, back when jquery was still hot and IE6 still existed as a problem. The landing page in our app used jqueryUI’s drag and drop support, back around the time they declared bankruptcy on the confusing buggy code and wouldn’t even accept bug fixes because they were replacing it component by component (which was taking almost 3x as long as predicted). We had columns you could drag items between but they had a max height and scroll bars and it turned out jqueryUI would let you drag items into different rows if the overflow area for adjacent drag targets overlapped your row. The person who found it couldn’t fix it. The other fixer couldn’t fix it. I diagnosed it but the spaghetti code was a recursive mess and I could not find a spot where I could fix it. Especially given I couldn’t send in a patch to them. So I spent half of my free time the last day of every (2 week) sprint for almost six months before I finally found a small function I could monkey patch to wrap it in a short circuit check for clipping region. I spent maybe 20,30 hours on this, a lot of it just getting back to the same situation to debug. But it felt like it took forever to fix it. The short circuit also made drag and drop faster, which was just getting in the edge of distracting. Particularly on a crowded page.

CaptainOfCoit 11mo ago
I remember many similar cycles of having different browsers open side-by-side, and trying to pinpoint (without the developer tools we know and love today) the exact reason why one border was one pixel in one browser, and two pixels in the other, throwing the whole layout off. Also remembering when Firebug for Firefox appeared, and made so many things so much easier. Suddenly things that took hours took days, and it was so much easier when you had some introspection tools.
- yard2010 11mo ago
  * { border: red 1px solid } Remember when IE6 was a thing? The kids today are angry at chrome for good reasons and yet, there was a time in which the most popular browser didn't implement jack shit from the specs. And it was the kind of browser that ships with the OS. God the bad karma for working with this crap. I'm glad it's over.

ipsum2 11mo ago

Apple used to contribute to the PyTorch MPS backend, but decided to create their own framework (MLX) instead, fragmenting the ecosystem for very little gain. (MLX is basically PyTorch, but invented-at-apple) Meta, the creator and main contributor to PyTorch, does not use Macs for their day-to-day ML work (they focus on GPUs and CPUs), so the MPS backend is sadly incomplete and has errors like the one you see here.

almostgotcaught 11mo ago
none of this is correct (except the part where FB doesn't use apple in prod). EDIT: for the downvoters - i'll repeat, this is not a correct assessment of the relationship between Apple and PyTorch. but you can keep downvoting if you want <shrug>
- ipsum2 11mo ago
  Please be specific if you have anything to say. By the way, the co-creator and core maintainer of PyTorch has the same opinion as me. https://x.com/soumithchintala/status/1978848796953161754 https://x.com/soumithchintala/status/1978848796953161754 "MacStudio you ask? Apple Engineering's *actual* time spent on PyTorch support has't given me confidence that PyTorch Mac experience would get anywhere close to NVIDIA's any time soon, if ever. The Meta engineers continue to do a huge amount of heavy-lifting for improving the MPS backend, including feeling the responsibility for the Mac experience. Apple's priorities keep changing, the number of engineering hours they contribute keeps changing and their interest in actually and wholly owning the PyTorch MPS backend keeps varying. If Apple wants MacStudio to become an actual AI devbox, and not just an AI inference machine, then prioritizing software support for PyTorch (>90% marketshare in AI) would probably be a good idea."
  - hedgehog 11mo ago
    Apple has never cared about ML research on their hardware. I've never been able to pin down a specific reason why, best I can figure out is they don't see it bringing enough additional hardware sales to be a focus.

mirekrusin 11mo ago

Nice work, surprising, I'd imagine implementations are cross tested all the time and this kind of bugs have no way of appearing?

cryber 11mo ago

this is a great writeup! methodical without being pedantic.

hershyb_ 11mo ago

awesome read!

anal_reactor 11mo ago

If I understand correctly, the root cause of the bug was improper use of object-oriented programming. A `Placeholder` object behaves differently depending on how it was created, and requires the user to have this awareness. The check `if is_continuous` should only ever exist inside the code of the `Placeholder` class.

albertzeyer 11mo ago

The bug was with non-contiguous data in tensors. I also had a very similar bug a while ago, broken gradients due to non-contiguous data for masked_select: https://github.com/pytorch/pytorch/issues/99638 https://github.com/pytorch/pytorch/issues/99638 In my case, it was easier to identify: I had another implementation of my loss function before that did not use masked_select. But then I thought I can be clever and use masked_select to take out the non-masked frames and calculate the loss only on those. But it wasn't working. Also, it only happened for some models, not for all. It turns out, it was always happening when the data coming out of the model was non-contiguous. I think the bugs with non-contiguous data are not so uncommon. I wonder how much of that we still have.

dcl 11mo ago

Is this why I cannot seem to fine tune YOLO models on a Apple M4? The loss hits nan after a few batches. Same code using Windows PC and Google Colab CPU and GPU is fine...

EdwardDiego 11mo ago

Kudos to Elana for a) such a thorough deep dive and b) a great write-up of it. I understand very little about ML libraries, but was able to follow this easily :)

farhanhubble 11mo ago

Great work hunting the bug down the stack. The writeup is top notch. I wish I documented some of the nastiest bugs I found in such detail. Funnily, only a few days ago I was thinking about just how far the field has come since 2014 or so when you'd build a computational graph, initialize weights manually and so on, versus now, where you just have to use a library like Ultralytics or HuggingFace most of the time. Then I thought about just how many deep, undetected bugs there would be in this mountain of abstraction. Bugs that make the computation invalid.

Rileyen 11mo ago

Just read the article and it instantly brought back memories of when I spent days trying to fix a broken loss in a PyTorch model. Turned out I had passed the wrong optimizer parameters. I ended up digging all the way from the model to the CUDA kernel. Debugging took longer than training. What’s the trickiest bug you’ve ever run into?