8 ms·

Reddit's photo albums broke due to Integer overflow of Signed Int32

328 points by Rebles 4y ago

Rebles 4y ago
Two days ago, Reddit ids have finally incremented passed the 2,147,483,647, or the maximum range of a signed int32. It seems one of Reddit's subsystems, the one that serves its photo albums broke due to the integer overflow.
- fdgsdfogijq 4y ago
  Someone probably joked they would never reach that scale when they wrote that code
  - kristopolous 4y ago
    Thinking "if that ever gets anywhere close to a problem we'll have vast resources and plenty of time to fix it" and then, I'm guessing, that person left a few months later and nobody owned that part of the code because it worked. Then 10 or so years went by... Whenever I write code like that which may break in say, 5 years, I'll sign it in the comments and put my personal email and phone number inviting future people to call me and I'll fix it for free (cause I take responsibilities for my code pretty seriously). Nobody has ever taken me up on it though...
    - silisili 4y ago
      Oh god, bad memories. You reminded me of a guy I worked with who wrote absolutely terrible, unsafe code and when called out said they are 'caviar problems.' When I asked what in the hell a caviar problem was, he said it meant by the time we had to worry about those things, we'd all be so rich we'd be eating caviar.
      - bagels 4y ago
        The reality is that some other person brought in later that is not rich will have to fix it. What a jerk.
    - jacobr1 4y ago
      And they weren't even wrong! The challenge is addressing the tech-debt, not the choice to incur it.
    - monsieurbanana 4y ago
      To people reading this: please never ever do that, unless it was agreed upon with your client/employer beforehand (and compensated accordingly). If you know the code may break in 5 years, why don't you fix it now? Because your employer doesn't want to pay you to do that (probably for good reason, surely there's higher priorities). Then why would you do it for free years later?
      - kristopolous 4y ago
        No. You can say, anticipate breaking changes that is calendared to happen as an eventually rollover or deprecation Essentially things have to work one way now and there's solid reason to think it will no longer work that way later and there's no valid path to fix because exogenous conditions change. This alternatively, can be done in bad faith to guarantee a future paycheck by intentionally placing timebombs in and then charging high rates to come back. There's incentives for abuse. I'm making it clear and acting in good faith. I can give them 2 hours for free in 5 years, whatever. I'm a competent and responsible person and I get paid well by not being so cheap.
        monsieurbanana 4y ago
        We're not talking about working in bad faith, that's bad obviously. But there's no reason for most people to give away even 2 hours of their time for free. If you're working as a freelancer and it's your way of building a portfolio of clients, that's a good reason to give away those two hours. But I'd rather not have someone with little work experience read that, think it's reasonable, and apply it themselves. In my first year of university I was working for a very small web company. I got paid per website an amount that was enough for my student needs, nothing much. I wasn't a very good developer in my 1st year, so I often received emails about bugs in past projects. The bugs were my fault, so naturally I fixed them in my own free time. It took me 2 years to see how badly I was getting owned by the company. If they wanted less bugs, they should have asked someone with experience and a much higher salary.
        josho 4y ago
        The employer also shouldn't take you up on the offer because now you are engaging in work without a contract, which opens up contract issues, probably breaks things like SOC compliance, etc. I like your accountability, but it's sufficient to document the issue in code, open a jira (or equivalent) and move on.
    - shock-value 4y ago
      Good practice would be leaving a comment with a TODO, an explanation of the problem, and some basic instructions as to a possible fix. Leaving your email and phone number is absurd and should not be promoted as good practice. No sane company would take you up on the offer anyway.
    - lmm 4y ago
      That's maybe a reasonable thing to do if you're an independent business selling the code, but it's a scab move if you're doing that as an employee. Never work for free.
    - nonethewiser 4y ago
      Hm. I wonder why no one ever called you.
      - bagels 4y ago
        It's bold. I'm sure if one of those time bombs went off, the call might not be to ask for help, but to impolitely complain.
      - kristopolous 4y ago
        Because there's a dumb culture of treating people who left like they're from beyond the grave where you have to divine their intentions like some programming ouija board. I broke that unspoken rule and got a really intractable problem that held us up for days finally fixed by making a short phone call early in my career and told myself if I was ever causing what we were going through I'd want to make it clear that I'm open to that rule being broken as well. Apparently even suggesting such a rule should be questioned and to try human solutions to human problems is met with mockery and disdain. Alright, cool.
    - CyberDildonics 4y ago
      Why would you write something that might break in 5 years? Why not just use a 64 bit integer in any situation like this an be done with it?
    - geobmx540 4y ago
      Probably sufficient to write a test that will break in 5 years - X [days|months] with notes in the test about the problem and how to fix
  - sph 4y ago
    So you migrate to int64 and one day someone will wonder why the hell did we ever think no one would reach 2^64 rows in a database table. Or that 2^128 IP addresses would be enough for everyone.
    - sokoloff 4y ago
      In my experience, you first do an emergency re-seed to use the negative half of the int32 space. Been through it twice in my career…
    - marcosdumay 4y ago
      Those numbers are much, much larger than 2^32. No process that touches the real world gets into 2^64. No process at all gets into 2^128, at lwast not while we live around a single star.
    - WhitneyLand 4y ago
      “32-bit IPv4 addresses to 128-bit IPv6 addresses means going from 4.3 billion to 340,282,366,920,938,463,463,374,607,431,768,211,456” Yes 128 bits is only 4x times the bit length, but the address space is exponentially bigger. Some predict if we do run out, it might take ~100 years.
      - josephg 4y ago
        > Some predict if we do run out, it might take ~100 years. You think we can use 340,282,366,920,938,463,463,374,607,431,768,211,456 addresses in 100 years? Thats ... ahem, ambitious!
        Dylan16807 4y ago
        In practical terms, IPv6 is arranged as a 64 bit subnet ID and 64 bit device ID. And the subnets are in a hierarchy that helps with routing but limits the packing efficiency. We could feasibly "run out" to the point that we can't keep doing things this way.
        ipython 4y ago
- cowsup 4y ago
  Strange thing is that photo albums re relatively new. Imgur was the go-to host for Reddit, and then they made their own uploader a looong time later. The "albums" functionality only came out in July of 2020, according to a Google search. Seems this was less likely a "someone else will deal with it" problem, and more of a development / QA testing problem.
  - Gigachad 4y ago
    For some reason most stuff still defaults to i32 and a lot of people use them for new code. At this point I'd not be against linters warning against using 32 bit ints unless you have a good reason.
    - kuroguro 4y ago
      What I don't get is why you'd want a signed int for an id.
    - bushbaba 4y ago
      My alma maters vending machines used an unsigned integer for understanding school debt card balance. However the school debt card used a signed integer to allow for small negative balances. Well students uncovered the flaw and had their cards be at negative 1 dollar, which the vending machine read as a very large balance.
    - thegeomaster 4y ago
      They're still a good choice for most perf critical code due to the reduced memory use.
  - jimmytucson 4y ago
    A new feature but it wasn’t built on a new codebase. Reddit is a monolith and a lot of things users think of as different “entities” live in the same set of tables.
  - curioussavage 4y ago
    I’m pretty sure the table in question stores image metadata for all user uploaded images. As well as images scraped from posted links which goes back way before images in posts
madrox 4y ago
A long time ago we discovered Twitter used a round robin of three servers for assigning IDs to tweets. We inferred the round robin was done by doing mod 3 of a signed int32, and because that space doesn't divide neatly by two it meant one of the three servers saw less load than the others and we could map ID assignment volume according to how often it overflowed and hence estimate total tweet volume for a given period. Some of the details escape me (this was a decade ago) but it was a fun combination of statistical inference and CS knowledge that I don't get to use often. Whenever integer overflow comes up in a systems engineering context I get a little tickled.
- ipqk 4y ago
  Reminds me of the self-quoting tweet: https://news.ycombinator.com/item?id=25244872 https://news.ycombinator.com/item?id=25244872
- _3u10 4y ago
  If they were incrementing and modding wouldn’t that server see an extra 1/2 billionth more traffic? I don’t get how mod 3 affects anything if you’re just incrementing…
- vecter 4y ago
  Wouldn't that unevenness only affect 2^31 - 2 and 2^31 - 1, so a negligible fraction of the integers? Was that tiny discrepancy enough to make your calculations? In other words, what do you mean that it was done by doing mod 3 of a signed int32? If it was a monotonically increasing or random int32, I don't see how that unevenness would manifest in a meaningful way.
  - madrox 4y ago
    In another subthread, we realized my memory was wrong and we were measuring millisecond collisions. The serving ID imbalance was a side-effect. Also, it might've been an int16 I was thinking of but turns out the whole thing was shadows on cave walls.
- pedrovhb 4y ago
Thaxll 4y ago
The famous AUTO_INCREMENT that you though you would never reach...
- btown 4y ago
  Fun fact: if you do a lot of INSERT... ON CONFLICT calls in Postgres from automated systems that are updating much more often than you insert, your autoincrement primary key can increment far far faster than your data volume (since it doesn't de-increment on a conflict) and overflow an int, grinding things to a halt. One of the more maddening outages I've had to deal with!
  - hu3 4y ago
    Similar for MySQL. If you open a transaction, INSERT with AUTO_INCREMENT, then rollback the transaction, no data is saved, except the auto generated id is used and the next INSERT uses id+1.
    - bcrosby95 4y ago
      But MySQL's equivalent, insert...on duplicate, does not cause extra auto increments. IMHO postgres' is a much larger problem. That said, I say that in my experience based upon locking selects in MySQL - when the row doesn't exist is a big problem, oftentimes leading to deadlocks. So insert... on duplicate key is a life saver. I don't have as much experience using postgres, so I'm not sure how much of a problem that is there, so it's possible the "on conflict" isn't as useful.
    - ryanianian 4y ago
      Same with sequences in Oracle. They don't participate in transactions. Looking at the gaps between "consecutive" IDs shows surprisingly interesting behavior.
davidjfelix 4y ago
A classic case of "ids aren't numbers even if you choose to make them numeric"
- knodi123 4y ago
  ids being sortable has a lot of advantages over random guids.
  - iamdual 4y ago
    There are timestamps for sorting.
  - davidjfelix 4y ago
    This is really a false dichotomy you don't have to use guid/uuid. I'm saying even if you use sortable auto increment numbers, stop storing them like numbers.
    - function_seven 4y ago
      Doesn't numeric storage save space and make indexing faster? (This is a naive question, I'm not asserting it.) Yeah, numbers get you fun stuff like overflows and wrap-arounds, etc. But sorting is faster (sorting "99" before "100" is more complex than 99 before 100) and space requirement is lower (6 bytes can store a unique ID for 281 trillion objects, but 6 characters only permits 1 million if you're storing them as strings). Or is there some datatype that combines the space- and sort-efficiency of a numeric type, but doesn't bring the baggage of assuming they represent quantities?
      - skunkworker 4y ago
        XID wraparound is one of the things that makes me wish Postgres would go 64bit on some of their internal data structures.
    - Dylan16807 4y ago
      Note that reddit is currently generating base36 ids starting with z. You want everything to treat that as text? Sure. That text has been 6 characters long for ages, and it's about to hit 7. I personally expect to see more things break when that happens. The problem of overflow isn't special to numbers.
scrame 4y ago
Ha! Slashdot had a similar problem in the early 2000s because they did a difficult migration for user/post ids, but left the indexes at 32(?). So, everything worked great until it didn't, and they segment a lot of time future proofing it.
boosteri 4y ago
Nostalgic flashback to Premier Manager games, where players stats decreased as they aged. When they went /below 0/ they flipped around to 127. So a good strategy was to scout out really bad players from lower leagues about to hit age 30+. And offer them very long contracts to prevent them retiring .. and give time for most of their stats to flip around, turning them into superstars.
- johnfarrelldev 4y ago
  I always found the funniest occurrence of this was in the Civ game though it seems it originally being a bug is disputed. https://en.wikipedia.org/wiki/Nuclear_Gandhi https://en.wikipedia.org/wiki/Nuclear_Gandhi
  - vikingerik 4y ago
    The bug never existed at all in Civ 1. It was an urban legend all along. Similar behavior was intentional in Civ 5 as a joke, which convinced everyone that it really did happen in Civ 1 when it never did.
    - RedShift1 4y ago
      Mandela effect
      - yen223 4y ago
        It's the result of people repeating what they read on the Internet, not what they personally experienced. This happens way too often
        fknorangesite 4y ago
        > This happens way too often You're not wrong; but fyi it only happens in your originating universe. Over here we all experienced it.
      - 4y ago
_gabe_ 4y ago
I'm just curious, I know it's a long running joke about how we're so stupid to think that we would never run out of unique digits with 2^32 possible values, but is this also the case with 64 bit values? Every new bit doubles the amount of information, so if 32 bits lasted reddit 10 years, presumably 33 bits would last them 20 years, 34 would last 40 and so on. Eventually, 64 bits would last them 10×2^32 years, which seems like a safe bet. So am I being naive when I use 64 bit values for unique IDs? Or is it actually plausible that 64 bits is plenty of information for centuries to come? Edit: Also, technically reddit was using signed int32s. So they really only had 2^16 unique digits. If they used unsigned int32s, then that would have bought them a lot of time.
- loteck 4y ago
  presumably Isn't the answer to your question buried in your assumptions? For example, you seem to assume the rate of comsumption of unique IDs is static.
  - _gabe_ 4y ago
    It would have to increase A LOT to significantly change the outcome. For example, according to this website[0] (no idea how accurate it is) Facebook users upload more than 300 million photos a day. At that rate, 2^32/300,000,000 = 14.3. So 32 bits would give Facebook 14 days worth of unique ids for their photos. Whereas 2^64/300,000,000 = 61,489,147,000 days, which is around 168 million years worth of ids. All I'm saying is the jump from 2^32 to 2^64 is astronomical. I don't see using 64 bit integers for uids in my hobby code as something to be concerned about. In production code for a company I would use something more significant, but even then I feel like 64 bits will last a very long time. [0]: https://bernardmarr.com/how-much-data-do-we-create-every-day-the-mind-blowing-stats-everyone-should-read/ https://bernardmarr.com/how-much-data-do-we-create-every-day...
- lilyball 4y ago
  If 32 bits lasted 10 years, 33 bits would not last 20, as engagement is not a constant. If we look at the last 10 years, presumably there were a lot more posts in the second 5 years than in the first 5 years. I don't know what Reddit's actual user/engagement metrics show though. Certainly if they got to a stable point where they had approximately the same activity going forward as they have going back then it would, but that's not usually how social sites work. That said, it would still take quite a while to get to 64 bits.
steve76 4y ago
mhh__ 4y ago
For a supposed tech company reddit really are bad.
_iyhy 4y ago
It's 2022 we still are using int32 for anything.
TEP_Kim_Il_Sung 4y ago
BooneJS 4y ago
Happened a few years ago to YouTube[0]. I don’t know why counters that start at zero and only increment are stored as signed integers. 0: https://arstechnica.com/information-technology/2014/12/gangnam-style-overflows-int_max-forces-youtube-to-go-64-bit/amp/ https://arstechnica.com/information-technology/2014/12/gangn...
dmtroyer 4y ago
unique identifiers are so passé.
akoster 4y ago
Reminds me of a similar Chess.com iPad app issue from a few years back https://news.ycombinator.com/item?id=14539770 https://news.ycombinator.com/item?id=14539770
darylteo 4y ago
If I'm understanding the shitty change log exactly, was the solution to add an extra bit?
- maverwa 4y ago
  I guess thats a joke. Adding a single extra bit would usually be more complex then going to 64bit or going to 32bit unsigned. Well, I say that, but actually, "adding an extra bit" is basically what going from signed to unsigned would do. So maybe they just added an extra (32nd) bit?
dvh 4y ago
-2,147,483,648 photos should be enough for anybody

328 points by Rebles 4y ago

Two days ago, Reddit ids have finally incremented passed the 2,147,483,647, or the maximum range of a signed int32. It seems one of Reddit's subsystems, the one that serves its photo albums broke due to the integer overflow.

fdgsdfogijq 4y ago
Someone probably joked they would never reach that scale when they wrote that code
- kristopolous 4y ago
  Thinking "if that ever gets anywhere close to a problem we'll have vast resources and plenty of time to fix it" and then, I'm guessing, that person left a few months later and nobody owned that part of the code because it worked. Then 10 or so years went by... Whenever I write code like that which may break in say, 5 years, I'll sign it in the comments and put my personal email and phone number inviting future people to call me and I'll fix it for free (cause I take responsibilities for my code pretty seriously). Nobody has ever taken me up on it though...
  - silisili 4y ago
    Oh god, bad memories. You reminded me of a guy I worked with who wrote absolutely terrible, unsafe code and when called out said they are 'caviar problems.' When I asked what in the hell a caviar problem was, he said it meant by the time we had to worry about those things, we'd all be so rich we'd be eating caviar.
    - bagels 4y ago
      The reality is that some other person brought in later that is not rich will have to fix it. What a jerk.
  - jacobr1 4y ago
    And they weren't even wrong! The challenge is addressing the tech-debt, not the choice to incur it.
  - monsieurbanana 4y ago
    To people reading this: please never ever do that, unless it was agreed upon with your client/employer beforehand (and compensated accordingly). If you know the code may break in 5 years, why don't you fix it now? Because your employer doesn't want to pay you to do that (probably for good reason, surely there's higher priorities). Then why would you do it for free years later?
    - kristopolous 4y ago
      No. You can say, anticipate breaking changes that is calendared to happen as an eventually rollover or deprecation Essentially things have to work one way now and there's solid reason to think it will no longer work that way later and there's no valid path to fix because exogenous conditions change. This alternatively, can be done in bad faith to guarantee a future paycheck by intentionally placing timebombs in and then charging high rates to come back. There's incentives for abuse. I'm making it clear and acting in good faith. I can give them 2 hours for free in 5 years, whatever. I'm a competent and responsible person and I get paid well by not being so cheap.
      - monsieurbanana 4y ago
        We're not talking about working in bad faith, that's bad obviously. But there's no reason for most people to give away even 2 hours of their time for free. If you're working as a freelancer and it's your way of building a portfolio of clients, that's a good reason to give away those two hours. But I'd rather not have someone with little work experience read that, think it's reasonable, and apply it themselves. In my first year of university I was working for a very small web company. I got paid per website an amount that was enough for my student needs, nothing much. I wasn't a very good developer in my 1st year, so I often received emails about bugs in past projects. The bugs were my fault, so naturally I fixed them in my own free time. It took me 2 years to see how badly I was getting owned by the company. If they wanted less bugs, they should have asked someone with experience and a much higher salary.
      - josho 4y ago
        The employer also shouldn't take you up on the offer because now you are engaging in work without a contract, which opens up contract issues, probably breaks things like SOC compliance, etc. I like your accountability, but it's sufficient to document the issue in code, open a jira (or equivalent) and move on.
  - shock-value 4y ago
    Good practice would be leaving a comment with a TODO, an explanation of the problem, and some basic instructions as to a possible fix. Leaving your email and phone number is absurd and should not be promoted as good practice. No sane company would take you up on the offer anyway.
  - lmm 4y ago
    That's maybe a reasonable thing to do if you're an independent business selling the code, but it's a scab move if you're doing that as an employee. Never work for free.
  - nonethewiser 4y ago
    Hm. I wonder why no one ever called you.
    - bagels 4y ago
      It's bold. I'm sure if one of those time bombs went off, the call might not be to ask for help, but to impolitely complain.
    - kristopolous 4y ago
      Because there's a dumb culture of treating people who left like they're from beyond the grave where you have to divine their intentions like some programming ouija board. I broke that unspoken rule and got a really intractable problem that held us up for days finally fixed by making a short phone call early in my career and told myself if I was ever causing what we were going through I'd want to make it clear that I'm open to that rule being broken as well. Apparently even suggesting such a rule should be questioned and to try human solutions to human problems is met with mockery and disdain. Alright, cool.
  - CyberDildonics 4y ago
    Why would you write something that might break in 5 years? Why not just use a 64 bit integer in any situation like this an be done with it?
  - geobmx540 4y ago
    Probably sufficient to write a test that will break in 5 years - X [days|months] with notes in the test about the problem and how to fix
- sph 4y ago
  So you migrate to int64 and one day someone will wonder why the hell did we ever think no one would reach 2^64 rows in a database table. Or that 2^128 IP addresses would be enough for everyone.
  - sokoloff 4y ago
    In my experience, you first do an emergency re-seed to use the negative half of the int32 space. Been through it twice in my career…
  - marcosdumay 4y ago
    Those numbers are much, much larger than 2^32. No process that touches the real world gets into 2^64. No process at all gets into 2^128, at lwast not while we live around a single star.
  - WhitneyLand 4y ago
    “32-bit IPv4 addresses to 128-bit IPv6 addresses means going from 4.3 billion to 340,282,366,920,938,463,463,374,607,431,768,211,456” Yes 128 bits is only 4x times the bit length, but the address space is exponentially bigger. Some predict if we do run out, it might take ~100 years.
    - josephg 4y ago
      > Some predict if we do run out, it might take ~100 years. You think we can use 340,282,366,920,938,463,463,374,607,431,768,211,456 addresses in 100 years? Thats ... ahem, ambitious!
      - Dylan16807 4y ago
        In practical terms, IPv6 is arranged as a 64 bit subnet ID and 64 bit device ID. And the subnets are in a hierarchy that helps with routing but limits the packing efficiency. We could feasibly "run out" to the point that we can't keep doing things this way.
      - ipython 4y ago
cowsup 4y ago
Strange thing is that photo albums re relatively new. Imgur was the go-to host for Reddit, and then they made their own uploader a looong time later. The "albums" functionality only came out in July of 2020, according to a Google search. Seems this was less likely a "someone else will deal with it" problem, and more of a development / QA testing problem.
- Gigachad 4y ago
  For some reason most stuff still defaults to i32 and a lot of people use them for new code. At this point I'd not be against linters warning against using 32 bit ints unless you have a good reason.
  - kuroguro 4y ago
    What I don't get is why you'd want a signed int for an id.
  - bushbaba 4y ago
    My alma maters vending machines used an unsigned integer for understanding school debt card balance. However the school debt card used a signed integer to allow for small negative balances. Well students uncovered the flaw and had their cards be at negative 1 dollar, which the vending machine read as a very large balance.
  - thegeomaster 4y ago
    They're still a good choice for most perf critical code due to the reduced memory use.
- jimmytucson 4y ago
  A new feature but it wasn’t built on a new codebase. Reddit is a monolith and a lot of things users think of as different “entities” live in the same set of tables.
- curioussavage 4y ago
  I’m pretty sure the table in question stores image metadata for all user uploaded images. As well as images scraped from posted links which goes back way before images in posts

madrox 4y ago

A long time ago we discovered Twitter used a round robin of three servers for assigning IDs to tweets. We inferred the round robin was done by doing mod 3 of a signed int32, and because that space doesn't divide neatly by two it meant one of the three servers saw less load than the others and we could map ID assignment volume according to how often it overflowed and hence estimate total tweet volume for a given period. Some of the details escape me (this was a decade ago) but it was a fun combination of statistical inference and CS knowledge that I don't get to use often. Whenever integer overflow comes up in a systems engineering context I get a little tickled.

ipqk 4y ago
Reminds me of the self-quoting tweet: https://news.ycombinator.com/item?id=25244872 https://news.ycombinator.com/item?id=25244872
_3u10 4y ago
If they were incrementing and modding wouldn’t that server see an extra 1/2 billionth more traffic? I don’t get how mod 3 affects anything if you’re just incrementing…
vecter 4y ago
Wouldn't that unevenness only affect 2^31 - 2 and 2^31 - 1, so a negligible fraction of the integers? Was that tiny discrepancy enough to make your calculations? In other words, what do you mean that it was done by doing mod 3 of a signed int32? If it was a monotonically increasing or random int32, I don't see how that unevenness would manifest in a meaningful way.
- madrox 4y ago
  In another subthread, we realized my memory was wrong and we were measuring millisecond collisions. The serving ID imbalance was a side-effect. Also, it might've been an int16 I was thinking of but turns out the whole thing was shadows on cave walls.
pedrovhb 4y ago

Thaxll 4y ago

The famous AUTO_INCREMENT that you though you would never reach...

btown 4y ago
Fun fact: if you do a lot of INSERT... ON CONFLICT calls in Postgres from automated systems that are updating much more often than you insert, your autoincrement primary key can increment far far faster than your data volume (since it doesn't de-increment on a conflict) and overflow an int, grinding things to a halt. One of the more maddening outages I've had to deal with!
- hu3 4y ago
  Similar for MySQL. If you open a transaction, INSERT with AUTO_INCREMENT, then rollback the transaction, no data is saved, except the auto generated id is used and the next INSERT uses id+1.
  - bcrosby95 4y ago
    But MySQL's equivalent, insert...on duplicate, does not cause extra auto increments. IMHO postgres' is a much larger problem. That said, I say that in my experience based upon locking selects in MySQL - when the row doesn't exist is a big problem, oftentimes leading to deadlocks. So insert... on duplicate key is a life saver. I don't have as much experience using postgres, so I'm not sure how much of a problem that is there, so it's possible the "on conflict" isn't as useful.
  - ryanianian 4y ago
    Same with sequences in Oracle. They don't participate in transactions. Looking at the gaps between "consecutive" IDs shows surprisingly interesting behavior.

davidjfelix 4y ago

A classic case of "ids aren't numbers even if you choose to make them numeric"

knodi123 4y ago
ids being sortable has a lot of advantages over random guids.
- iamdual 4y ago
  There are timestamps for sorting.
- davidjfelix 4y ago
  This is really a false dichotomy you don't have to use guid/uuid. I'm saying even if you use sortable auto increment numbers, stop storing them like numbers.
  - function_seven 4y ago
    Doesn't numeric storage save space and make indexing faster? (This is a naive question, I'm not asserting it.) Yeah, numbers get you fun stuff like overflows and wrap-arounds, etc. But sorting is faster (sorting "99" before "100" is more complex than 99 before 100) and space requirement is lower (6 bytes can store a unique ID for 281 trillion objects, but 6 characters only permits 1 million if you're storing them as strings). Or is there some datatype that combines the space- and sort-efficiency of a numeric type, but doesn't bring the baggage of assuming they represent quantities?
    - skunkworker 4y ago
      XID wraparound is one of the things that makes me wish Postgres would go 64bit on some of their internal data structures.
  - Dylan16807 4y ago
    Note that reddit is currently generating base36 ids starting with z. You want everything to treat that as text? Sure. That text has been 6 characters long for ages, and it's about to hit 7. I personally expect to see more things break when that happens. The problem of overflow isn't special to numbers.

scrame 4y ago

Ha! Slashdot had a similar problem in the early 2000s because they did a difficult migration for user/post ids, but left the indexes at 32(?). So, everything worked great until it didn't, and they segment a lot of time future proofing it.

boosteri 4y ago

Nostalgic flashback to Premier Manager games, where players stats decreased as they aged. When they went /below 0/ they flipped around to 127. So a good strategy was to scout out really bad players from lower leagues about to hit age 30+. And offer them very long contracts to prevent them retiring .. and give time for most of their stats to flip around, turning them into superstars.

johnfarrelldev 4y ago
I always found the funniest occurrence of this was in the Civ game though it seems it originally being a bug is disputed. https://en.wikipedia.org/wiki/Nuclear_Gandhi https://en.wikipedia.org/wiki/Nuclear_Gandhi
- vikingerik 4y ago
  The bug never existed at all in Civ 1. It was an urban legend all along. Similar behavior was intentional in Civ 5 as a joke, which convinced everyone that it really did happen in Civ 1 when it never did.
  - RedShift1 4y ago
    Mandela effect
    - yen223 4y ago
      It's the result of people repeating what they read on the Internet, not what they personally experienced. This happens way too often
      - fknorangesite 4y ago
        > This happens way too often You're not wrong; but fyi it only happens in your originating universe. Over here we all experienced it.
    - 4y ago

_gabe_ 4y ago

I'm just curious, I know it's a long running joke about how we're so stupid to think that we would never run out of unique digits with 2^32 possible values, but is this also the case with 64 bit values? Every new bit doubles the amount of information, so if 32 bits lasted reddit 10 years, presumably 33 bits would last them 20 years, 34 would last 40 and so on. Eventually, 64 bits would last them 10×2^32 years, which seems like a safe bet. So am I being naive when I use 64 bit values for unique IDs? Or is it actually plausible that 64 bits is plenty of information for centuries to come? Edit: Also, technically reddit was using signed int32s. So they really only had 2^16 unique digits. If they used unsigned int32s, then that would have bought them a lot of time.

loteck 4y ago
presumably Isn't the answer to your question buried in your assumptions? For example, you seem to assume the rate of comsumption of unique IDs is static.
- _gabe_ 4y ago
  It would have to increase A LOT to significantly change the outcome. For example, according to this website[0] (no idea how accurate it is) Facebook users upload more than 300 million photos a day. At that rate, 2^32/300,000,000 = 14.3. So 32 bits would give Facebook 14 days worth of unique ids for their photos. Whereas 2^64/300,000,000 = 61,489,147,000 days, which is around 168 million years worth of ids. All I'm saying is the jump from 2^32 to 2^64 is astronomical. I don't see using 64 bit integers for uids in my hobby code as something to be concerned about. In production code for a company I would use something more significant, but even then I feel like 64 bits will last a very long time. [0]: https://bernardmarr.com/how-much-data-do-we-create-every-day-the-mind-blowing-stats-everyone-should-read/ https://bernardmarr.com/how-much-data-do-we-create-every-day...
lilyball 4y ago
If 32 bits lasted 10 years, 33 bits would not last 20, as engagement is not a constant. If we look at the last 10 years, presumably there were a lot more posts in the second 5 years than in the first 5 years. I don't know what Reddit's actual user/engagement metrics show though. Certainly if they got to a stable point where they had approximately the same activity going forward as they have going back then it would, but that's not usually how social sites work. That said, it would still take quite a while to get to 64 bits.

steve76 4y ago

mhh__ 4y ago

For a supposed tech company reddit really are bad.

_iyhy 4y ago

It's 2022 we still are using int32 for anything.

TEP_Kim_Il_Sung 4y ago

BooneJS 4y ago

Happened a few years ago to YouTube[0]. I don’t know why counters that start at zero and only increment are stored as signed integers. 0: https://arstechnica.com/information-technology/2014/12/gangnam-style-overflows-int_max-forces-youtube-to-go-64-bit/amp/ https://arstechnica.com/information-technology/2014/12/gangn...

dmtroyer 4y ago

unique identifiers are so passé.

akoster 4y ago

Reminds me of a similar Chess.com iPad app issue from a few years back https://news.ycombinator.com/item?id=14539770 https://news.ycombinator.com/item?id=14539770

darylteo 4y ago

If I'm understanding the shitty change log exactly, was the solution to add an extra bit?

maverwa 4y ago
I guess thats a joke. Adding a single extra bit would usually be more complex then going to 64bit or going to 32bit unsigned. Well, I say that, but actually, "adding an extra bit" is basically what going from signed to unsigned would do. So maybe they just added an extra (32nd) bit?

dvh 4y ago

-2,147,483,648 photos should be enough for anybody