6 ms·

Hey since this is blown up I just want to address it directly. I take responsibility for what happened here. My RubyGems.org account was using an insecure, reu

Hey since this is blown up I just want to address it directly. I take responsibility for what happened here. My RubyGems.org account was using an insecure, reused password that has leaked to the internet in other breaches. I made that account probably over 10 years ago, so it predated my use of password managers and I haven't used it much lately, so I didn't catch it in a 1password audit or anything. Sometimes we miss things despite our best efforts. Rotate your passwords, kids.

donkeyd 7y ago
Wow, that's a pretty well executed and possibly targeted attack then. It blows my mind how easy it can be to perform a high impact attack by abusing popular libraries. Hopefully this was caught before it got into production in high profile implementations.
- mwmanning 7y ago
  Yeah I'm assuming the methodology is: 1) Find high-value target libraries 2) Grab the usernames of accounts with push access 3) Check those against password dumps I feel really stupid about this, but like I said it was an oversight. I apologize and will try to do better.
  - 0x0 7y ago
    Sounds like rubygems and other registries like npm should try to get ahold of those password dumps and check them against their own account databases somewhat frequently!
    - dspillett 7y ago
      If you find a reused password, how do you let the user know though? If I got a "your account is vulnerable" message I'd ignore it as junk like all the other ones I get pretty much daily. You could force a change next time the user logs to your interactive interface, but many users won't do that for some time. The best approach is probably to disable the account completely until an interactive login is made and a password reset can be forced but some would be up in arms about the inconvenience caused: you can't just allow a simple reset as the login could be coming from an attacker not the original user, an extra channel will need to be used to verify the identity. You might just have to leave the account locked forever and expect the user to create a new one - but now you have the old account and its content which may be used as a dependency of many projects which now break, unnecessarily if there hasn't been a login by a nefarious type.
      - eli 7y ago
        Presumably you'd use whatever procedure you use for a lost password? But simply forcing a password change at the next login after detecting an insecure password would not unduly burden anyone and would be better than doing nothing.
      - yebyen 7y ago
        You could send that notification, invalidate any client tokens, and also disable the compromised password forcing the user to re-authenticate through their email address, a-la password reset, and I guess also verify they aren't using the same password again. You wouldn't lock the account forever, the point is to establish that the person whose password was compromised knows, that the password is not the only factor which is used to regain access to the account, and to ensure that your service (rubygems) and its downstream users are not compromised as well as a result of the breach. Any groaning about the inconvenience caused by disabling account access until the password is changed, can be simply shrugged away in favor of security concerns, with a link to this story about rest-client. By the time you have learned the user's plaintext password, their account may already have been compromised. There's a case to make that you disable all downloads of any gems that might be compromised from the account until you've verified they aren't. That might be over the top, especially for popular projects as now we are talking serious inconvenience affecting potentially thousands or more of downstreams. It's a sticky situation, since you don't really know how long that password has been in the open for hackers to use and abuse once you've discovered it in a password dump.
      - marcus_holmes 7y ago
        If a gem maintainer is re-using a known-compromised password they have absolutely zero right to be annoyed at the "inconvenience" of having to reset their password to something that isn't compromised. RubyGems has a responsibility to its users and community here. It (like npm) needs to take this stuff seriously.
      - josephwegner 7y ago
        Heroku did this about a year ago. They have a list of known pwned passwords (probably haveibeenpwned, but honestly I'm not sure), and disallow accounts to use those passwords. When that change was implemented, any account using a pwned password had that password expired. https://status.heroku.com/incidents/1625 https://status.heroku.com/incidents/1625 (source: I work for Heroku Support)
    - kevin_thibedeau 7y ago
      That's not very practical if salted hashes are being stored.
      - marcus_holmes 7y ago
        but the salt is in the database. You can hash all the known-compromised passwords with the salt and see if any match.
        stevenwliao 7y ago
        The salts should be different for each user, specifically to deter brute forcing of this nature. The only time you would have access is when the user logs in, so for rarely logged in users you would have to proactively reset their password or cross your fingers.
        lamontcg 7y ago
        Hopefully you don't transmit the password and are doing challenge/response so that you don't even have it when the user logs in. But even with 12 round bcrypt hashing, you should be able to fairly cheaply attack a list of 2,000 bcrypted passwords with a million-entry database of leaked e-mail/password combos in a GPU-month. Probably easier to force a password reset on everyone and then do the checking on password change, although you need to be careful there not to be sending the password. EDIT: uhm, wait, so if you've got the e-mail address in the dump then there's only one user for that, so just grab their salt and hash the password and check it. So that million entry database should be checkable in a bit over half an hour...
        eneveu 7y ago
        > Hopefully you don't transmit the password and are doing challenge/response so that you don't even have it when the user logs in. Wasn't challenge/response / SRP authentication debunked ? https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2011/august/javascript-cryptography-considered-harmful/ https://www.nccgroup.trust/us/about-us/newsroom-and-events/b... https://news.ycombinator.com/item?id=2859470 https://news.ycombinator.com/item?id=2859470
    - febeling 7y ago
      Make 2FA mandatory. Apple did just that recently with their app stores. Someone authoring libraries should be able to handle that.
  - OskarS 7y ago
    It happens. You've taken reasonable precautions to safeguard your online identity, which is all one can really ask. Sometimes things slip through the cracks. The hacker is to blame, not you. The larger question is about if gem/npm/cargo-style package managers are such a terrific idea in the long run. The security implications are pretty serious.
- dspillett 7y ago
  I doubt the initial attack was targetted. That would have been a brute force testing-known-passwords-against-similarly-named-accounts. Once a useful account was found it could well have been sold on the appropriate black market rather than the finder using it themselves.
vorpalhex 7y ago
It can happen to us all.
tempguy9999 7y ago
> I take responsibility for what happened here that's.... rare. Well done.

donkeyd 7y ago

Wow, that's a pretty well executed and possibly targeted attack then. It blows my mind how easy it can be to perform a high impact attack by abusing popular libraries. Hopefully this was caught before it got into production in high profile implementations.

mwmanning 7y ago
Yeah I'm assuming the methodology is: 1) Find high-value target libraries 2) Grab the usernames of accounts with push access 3) Check those against password dumps I feel really stupid about this, but like I said it was an oversight. I apologize and will try to do better.
- 0x0 7y ago
  Sounds like rubygems and other registries like npm should try to get ahold of those password dumps and check them against their own account databases somewhat frequently!
  - dspillett 7y ago
    If you find a reused password, how do you let the user know though? If I got a "your account is vulnerable" message I'd ignore it as junk like all the other ones I get pretty much daily. You could force a change next time the user logs to your interactive interface, but many users won't do that for some time. The best approach is probably to disable the account completely until an interactive login is made and a password reset can be forced but some would be up in arms about the inconvenience caused: you can't just allow a simple reset as the login could be coming from an attacker not the original user, an extra channel will need to be used to verify the identity. You might just have to leave the account locked forever and expect the user to create a new one - but now you have the old account and its content which may be used as a dependency of many projects which now break, unnecessarily if there hasn't been a login by a nefarious type.
    - eli 7y ago
      Presumably you'd use whatever procedure you use for a lost password? But simply forcing a password change at the next login after detecting an insecure password would not unduly burden anyone and would be better than doing nothing.
    - yebyen 7y ago
      You could send that notification, invalidate any client tokens, and also disable the compromised password forcing the user to re-authenticate through their email address, a-la password reset, and I guess also verify they aren't using the same password again. You wouldn't lock the account forever, the point is to establish that the person whose password was compromised knows, that the password is not the only factor which is used to regain access to the account, and to ensure that your service (rubygems) and its downstream users are not compromised as well as a result of the breach. Any groaning about the inconvenience caused by disabling account access until the password is changed, can be simply shrugged away in favor of security concerns, with a link to this story about rest-client. By the time you have learned the user's plaintext password, their account may already have been compromised. There's a case to make that you disable all downloads of any gems that might be compromised from the account until you've verified they aren't. That might be over the top, especially for popular projects as now we are talking serious inconvenience affecting potentially thousands or more of downstreams. It's a sticky situation, since you don't really know how long that password has been in the open for hackers to use and abuse once you've discovered it in a password dump.
    - marcus_holmes 7y ago
      If a gem maintainer is re-using a known-compromised password they have absolutely zero right to be annoyed at the "inconvenience" of having to reset their password to something that isn't compromised. RubyGems has a responsibility to its users and community here. It (like npm) needs to take this stuff seriously.
    - josephwegner 7y ago
      Heroku did this about a year ago. They have a list of known pwned passwords (probably haveibeenpwned, but honestly I'm not sure), and disallow accounts to use those passwords. When that change was implemented, any account using a pwned password had that password expired. https://status.heroku.com/incidents/1625 https://status.heroku.com/incidents/1625 (source: I work for Heroku Support)
  - kevin_thibedeau 7y ago
    That's not very practical if salted hashes are being stored.
    - marcus_holmes 7y ago
      but the salt is in the database. You can hash all the known-compromised passwords with the salt and see if any match.
      - stevenwliao 7y ago
        The salts should be different for each user, specifically to deter brute forcing of this nature. The only time you would have access is when the user logs in, so for rarely logged in users you would have to proactively reset their password or cross your fingers.
        lamontcg 7y ago
        Hopefully you don't transmit the password and are doing challenge/response so that you don't even have it when the user logs in. But even with 12 round bcrypt hashing, you should be able to fairly cheaply attack a list of 2,000 bcrypted passwords with a million-entry database of leaked e-mail/password combos in a GPU-month. Probably easier to force a password reset on everyone and then do the checking on password change, although you need to be careful there not to be sending the password. EDIT: uhm, wait, so if you've got the e-mail address in the dump then there's only one user for that, so just grab their salt and hash the password and check it. So that million entry database should be checkable in a bit over half an hour...
        eneveu 7y ago
        > Hopefully you don't transmit the password and are doing challenge/response so that you don't even have it when the user logs in. Wasn't challenge/response / SRP authentication debunked ? https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2011/august/javascript-cryptography-considered-harmful/ https://www.nccgroup.trust/us/about-us/newsroom-and-events/b... https://news.ycombinator.com/item?id=2859470 https://news.ycombinator.com/item?id=2859470
  - febeling 7y ago
    Make 2FA mandatory. Apple did just that recently with their app stores. Someone authoring libraries should be able to handle that.
- OskarS 7y ago
  It happens. You've taken reasonable precautions to safeguard your online identity, which is all one can really ask. Sometimes things slip through the cracks. The hacker is to blame, not you. The larger question is about if gem/npm/cargo-style package managers are such a terrific idea in the long run. The security implications are pretty serious.
dspillett 7y ago
I doubt the initial attack was targetted. That would have been a brute force testing-known-passwords-against-similarly-named-accounts. Once a useful account was found it could well have been sold on the appropriate black market rather than the finder using it themselves.

vorpalhex 7y ago

It can happen to us all.

tempguy9999 7y ago

> I take responsibility for what happened here that's.... rare. Well done.