7 ms·

When your hash becomes a string: Hunting Ruby's million-to-one memory bug

137 points by phmx 11mo ago

alexnewman 10mo ago
I don’t get it. Also it reads llmish
mwkaufma 10mo ago
A little strange to write up a bug hunt that was resolved by the ffi upstream already, and not by the hunt itself. OP didn't fix the bug, though identifying that the upgrade was relevant is of some interest. Writing could have been clearer.
- mbac32768 10mo ago
  The bug that was fixed in upstream manifested differently than what he was experiencing so the journey was to validate it for his case. OTOH I'm a bit surprised he didn't pull back earlier and suggest to his user to update to the latest version though and let him know.
  - eichin 10mo ago
    15 or so years ago I had a similar journey - a single python interpreter "impossible" segfault in production that turned out to be a bug in glibc realloc, that had already been fixed in an update, we just didn't figure out to even look for one until we'd narrowed it down that far. (We were shipping custom Debian installs on DVD, a fair number of our customer installs weren't internet accessible so casual upgrades were both impossible and unwanted, but it was also a process mistake on my part to not notice the existence of the upgrade sooner.) Never wrote it up externally because it was already solved and "Debian updates to existing releases are so rare that you really want to pay attention to all of them" (1) was already obvious (2) was only relevant to a really small set of people (3) this somewhat tortured example wasn't going to reach that small set anyway. (Made a reasonable interview story, though.)
ryandv 10mo ago
[flagged]
fleshmonad 10mo ago
LLM slop. Why do people (presumably) take the time to debug something like this, do tests and go to great lengths, but are too lazy to do a little manual writeup? Maybe the hour saved makes up for being associated with publishing AI slop under your own name? Like there is no way the author would have written a text that reads more convoluted than what we have here.
- ryandv 10mo ago
  [flagged]
  - skrebbel 10mo ago
    > which is par for Rubyists Pro-tip: re-read your comment before you submit and take out the bits that make you sound like an asshole.
    - ryandv 10mo ago
      [flagged]
    - dpark 10mo ago
      It’s truly weird how some people are just full of hate like this and either cannot see or do not care that they are unpleasant assholes.
      - deleted 10mo ago
        [deleted]
- sb8244 10mo ago
  I read it just fine and everything made sense in it. I would spend similar time debugging this if I were the author. It's a pretty serious bug, a non obvious issue, and would be impossible to connect to the ffi fix unless you already knew the problem.
- dpark 10mo ago
  Sorry, why is this LLM slop? I only got about halfway through because I don’t care about this enough to finish the read, but I don’t see the “obvious LLM” signal you do.
philipp-gayret 10mo ago
Had me in the first half. But from the "The Microsecond Window" chapter and on...; > No warning. No error. Just different methods that make no sense. > This is why write barriers exist. They're not optional extras for C extension authors. They're how you tell the garbage collector: "I'm holding a reference. Don't free this It's all ChatGPT LinkedIn and Instagram spam type slop. An unfortunate end to an otherwise interesting writeup.
khazhoux 10mo ago
I don’t understand why people are saying this article was AI generated. Do you think the author told chatgpt “Write me an article (with diagrams) about a Ruby hash race condition” and pasted that to their blog?
- Jweb_Guru 10mo ago
  Parts of it being generated by Claude or ChatGPT (which they very clearly were) does not necessarily mean that the whole article was fabricated.
YouAreWRONGtoo 10mo ago
[dead]
YouAreWRONGtoo 10mo ago
[dead]
Animats 10mo ago
So they turned on GC after every allocate ("GC stress"), and "With GC.stress = true, the GC runs after every possible allocation. That causes immediate segfaults because objects get freed before Ruby can even allocate new objects in their memory slots." That would seem to indicate a situation so broken that you can't expect anything to work reliably. The wrong-value situation would seem to be a subset of a bigger problem. It's like finding C code that depends on use-after-free working and which fails when you turn on buffer scrubbing at free.
- deleted 10mo ago
  [deleted]
- xerxes901 10mo ago
  That’s exactly what it was. He discovered the customer was using a version of ffi that had this “use-after-free” (ish) bug, but the question “is this actually what my customer was seeing or is there _another_ bug lurking” still needed to be answered.
  - seeknotfind 10mo ago
    It's nice that there is only a few weird behaviors produced. Often use-after-free leads to so many different random bugs, you might gorble a hubalu.
davebranton 10mo ago
If I see another AI-written trash article I am going to scream. Overlong, overwritten garbage. People used to write, and there was personality in that writing. Now people believe it's acceptable to generate reams of utter formless shite and post it on the internet. If you cannot be bothered to write something, why on God's good earth would you expect anyone to be bothered to read it?
- hansvm 10mo ago
  I'd normally agree, but this is a case I don't see often -- despite the form being terrible the content is good. I certainly would strongly prefer the same post with better writing, but if the entire 2019 internet were replaced with articles like this (on orthogonal topics/micro-topics) I think it'd be a better place.
dmix 10mo ago
A good example of why everyone should learn a bit of C and low level memory management
lifthrasiir 10mo ago
> Million-to-one bugs are real, not theoretical. They happen during initialization and restart, not runtime. When they trigger, they cascade - 2,500 errors from one root cause. In high-restart environments, rare becomes routine. Million-to-one bugs are not only real but high enough to matter, depending on which million. Many years ago I had a rare bug that corrupted timestamps in the logs, with an emperical probability of about one to 3--5 million (IIRC). Turned out that that seemingly benign bug was connected to a critical data corruption issue with real consumer complaints. (I have described this bug in detail in the past, see my past comment for details.)

137 points by phmx 11mo ago

I don’t get it. Also it reads llmish

A little strange to write up a bug hunt that was resolved by the ffi upstream already, and not by the hunt itself. OP didn't fix the bug, though identifying that the upgrade was relevant is of some interest. Writing could have been clearer.

mbac32768 10mo ago
The bug that was fixed in upstream manifested differently than what he was experiencing so the journey was to validate it for his case. OTOH I'm a bit surprised he didn't pull back earlier and suggest to his user to update to the latest version though and let him know.
- eichin 10mo ago
  15 or so years ago I had a similar journey - a single python interpreter "impossible" segfault in production that turned out to be a bug in glibc realloc, that had already been fixed in an update, we just didn't figure out to even look for one until we'd narrowed it down that far. (We were shipping custom Debian installs on DVD, a fair number of our customer installs weren't internet accessible so casual upgrades were both impossible and unwanted, but it was also a process mistake on my part to not notice the existence of the upgrade sooner.) Never wrote it up externally because it was already solved and "Debian updates to existing releases are so rare that you really want to pay attention to all of them" (1) was already obvious (2) was only relevant to a really small set of people (3) this somewhat tortured example wasn't going to reach that small set anyway. (Made a reasonable interview story, though.)

ryandv 10mo ago

[flagged]

fleshmonad 10mo ago

LLM slop. Why do people (presumably) take the time to debug something like this, do tests and go to great lengths, but are too lazy to do a little manual writeup? Maybe the hour saved makes up for being associated with publishing AI slop under your own name? Like there is no way the author would have written a text that reads more convoluted than what we have here.

ryandv 10mo ago
[flagged]
- skrebbel 10mo ago
  > which is par for Rubyists Pro-tip: re-read your comment before you submit and take out the bits that make you sound like an asshole.
  - ryandv 10mo ago
    [flagged]
  - dpark 10mo ago
    It’s truly weird how some people are just full of hate like this and either cannot see or do not care that they are unpleasant assholes.
    - deleted 10mo ago
      [deleted]
sb8244 10mo ago
I read it just fine and everything made sense in it. I would spend similar time debugging this if I were the author. It's a pretty serious bug, a non obvious issue, and would be impossible to connect to the ffi fix unless you already knew the problem.
dpark 10mo ago
Sorry, why is this LLM slop? I only got about halfway through because I don’t care about this enough to finish the read, but I don’t see the “obvious LLM” signal you do.

philipp-gayret 10mo ago

Had me in the first half. But from the "The Microsecond Window" chapter and on...; > No warning. No error. Just different methods that make no sense. > This is why write barriers exist. They're not optional extras for C extension authors. They're how you tell the garbage collector: "I'm holding a reference. Don't free this It's all ChatGPT LinkedIn and Instagram spam type slop. An unfortunate end to an otherwise interesting writeup.

khazhoux 10mo ago

I don’t understand why people are saying this article was AI generated. Do you think the author told chatgpt “Write me an article (with diagrams) about a Ruby hash race condition” and pasted that to their blog?

Jweb_Guru 10mo ago
Parts of it being generated by Claude or ChatGPT (which they very clearly were) does not necessarily mean that the whole article was fabricated.

YouAreWRONGtoo 10mo ago

[dead]

YouAreWRONGtoo 10mo ago

[dead]

Animats 10mo ago

So they turned on GC after every allocate ("GC stress"), and "With GC.stress = true, the GC runs after every possible allocation. That causes immediate segfaults because objects get freed before Ruby can even allocate new objects in their memory slots." That would seem to indicate a situation so broken that you can't expect anything to work reliably. The wrong-value situation would seem to be a subset of a bigger problem. It's like finding C code that depends on use-after-free working and which fails when you turn on buffer scrubbing at free.

deleted 10mo ago
[deleted]
xerxes901 10mo ago
That’s exactly what it was. He discovered the customer was using a version of ffi that had this “use-after-free” (ish) bug, but the question “is this actually what my customer was seeing or is there _another_ bug lurking” still needed to be answered.
- seeknotfind 10mo ago
  It's nice that there is only a few weird behaviors produced. Often use-after-free leads to so many different random bugs, you might gorble a hubalu.

davebranton 10mo ago

If I see another AI-written trash article I am going to scream. Overlong, overwritten garbage. People used to write, and there was personality in that writing. Now people believe it's acceptable to generate reams of utter formless shite and post it on the internet. If you cannot be bothered to write something, why on God's good earth would you expect anyone to be bothered to read it?

hansvm 10mo ago
I'd normally agree, but this is a case I don't see often -- despite the form being terrible the content is good. I certainly would strongly prefer the same post with better writing, but if the entire 2019 internet were replaced with articles like this (on orthogonal topics/micro-topics) I think it'd be a better place.

dmix 10mo ago

A good example of why everyone should learn a bit of C and low level memory management

lifthrasiir 10mo ago

> Million-to-one bugs are real, not theoretical. They happen during initialization and restart, not runtime. When they trigger, they cascade - 2,500 errors from one root cause. In high-restart environments, rare becomes routine. Million-to-one bugs are not only real but high enough to matter, depending on which million. Many years ago I had a rare bug that corrupted timestamps in the logs, with an emperical probability of about one to 3--5 million (IIRC). Turned out that that seemingly benign bug was connected to a critical data corruption issue with real consumer complaints. (I have described this bug in detail in the past, see my past comment for details.)