10 ms·

Actual twitter post: https://blog.twitter.com/official/en_us/topics/company/2018/keeping-your-account-secure.html https://blog.twitter.com/official/en_us/topics

Actual twitter post: https://blog.twitter.com/official/en_us/topics/company/2018/keeping-your-account-secure.html https://blog.twitter.com/official/en_us/topics/company/2018/... "Due to a bug, passwords were written to an internal log before completing the hashing process. We found this error ourselves, removed the passwords, and are implementing plans to prevent this bug from happening again." Exact same thing that github did just recently.

xd1936 8y ago
"[We] are implementing plans to prevent this bug from happening again" sure makes it sound like this bug is still happening. Should we wait a couple of days before changing passwords? Will it end up in this log right now, just like the old one?
- wlesieutre 8y ago
  That sounds more like "We're adding a more thorough testing and code-review process for our password systems to prevent developers from accidentally logging unhashed passwords in the future".
- SahAssar 8y ago
  ~~Sounds more like "we fixed this bug, and will ignore the processes that led to it happening" bullshit to me.~~
  - always_good 8y ago
    And this comment sounds more like "DAE hate twitter." Their response is acceptable and textbook. Doesn't really seem like the appropriate place to wage the battle.
    - SahAssar 8y ago
      Yeah, bad kneejerk response on my part. Sorry.
- reificator 8y ago
  No, it sounds like a reasonable bugfixing strategy. Identify the bug, identify the fastest way to resolve it, then once it's fixed figure out how to ensure it never happens again, and what to do if it does.
- bichiliad 8y ago
  I think you read "prevent this bug from happening again" to mean "prevent this particular problem from happening one more time", while the blogpost probably means something like "prevent this class of bug from occurring in the future"
jahvo 8y ago
It's funny, I wonder if hearing about that github bug made them check if they had committed the same mistake... only to find that they did :-)
- badloginagain 8y ago
  I think I, and everyone here, should check as well. If capable, security-minded companies can make such a mistake, so can you.
  - milesokeefe 8y ago
    Yep, glad I read this thread. We were making the same simple mistake.
    - bigiain 8y ago
      We aren't. Now. (We caught ourselves doing it 4-5 months back, and went through _everything_ checking... Only random accident that brought it to the attention of anyone who bothered to question it too... Two separate instances by different devs of 'if (DEBUG_LEVEL = 3){ }' instead of == 3 - both missed by code reviews too...)
      - jcoffland 8y ago
        This is why you should turn on compiler warnings and heed them. It would have caught this.
        bch 8y ago
        And consider “Yoda Notation”[0], which some people find annoying, but I found an easy hurdle to clear: if ( 3 = DEBUGLEVEL ) wouldn’t pass the the parser because you can’t assign to an rvalue. [0] https://en.wikipedia.org/wiki/Yoda_conditions https://en.wikipedia.org/wiki/Yoda_conditions
        8y ago
ams6110 8y ago
So best practice would be that the cleartext password is never sent to the server, so they could never log it even accidentally. That means the hashing needs to be done client side, probably with JavaScript. Is there any safe way to do that?
- jrochkind1 8y ago
  nah, that just makes the "hashed password" the equivalent of the cleartext password. Whatever it is your client sends to the server for auth is the thing that needs to be protected. If the client sends a "hashed password", that's just... the password. Which now needs to be protected. Since if someone has it, they can just send it to the server for auth. But you can do fancy cryptographic things where the server never sees the password and it's still secure. like the entire field of public key cryptography, diffie-hellman key exchange, etc.
  - ams6110 8y ago
    Makes sense, and then you're getting into something akin to SSH key pairs, and I know from experience that many users can't manage that especially across multiple client devices.
    - jrochkind1 8y ago
      There are probably ways to make it reasonable UX, but they probably require built-in browser (or other client) support. Someone in another part of this thread mentioned the "Web Authentication API" for browsers, which I'm not familiar with, but is possibly trying to approach this?
      - tehbeard 8y ago
        Web Auth API (authn) does try to make it usable. It ties in with the credential management API (A way to have the browser store login credentials for a site, a much less heuristic based approach than autocomplete on forms) and basic principle is generate a key pair, pass back public key to be sent to server during registration. On login generate a challenge value for the client to sign. I don't think iirc the JS code ever sees the private key, only the browser sees it.
theCricketer 8y ago
For people that know more about web security than I: Is there a reason it isn't good practice to hash the password client side so that the backends only ever see the hashed password and there is no chance for such mistakes?
- zackbloom 8y ago
  Realize the point of hashing the password is to make sure the thing users send to you is different than the thing you store. You'll still have to hash the hashes again on your end, otherwise anyone who gets accessed to your stored passwords could use them to login.
  - pishpash 8y ago
    But at least, with salt, it wouldn't be applicable to other sites, just one. Better to just never reuse a password though. Honestly sites should just standardize on a password changing protocol, that will go a long way towards making passwords actually disposable.
    - stingrae 8y ago
      I don't think a password changing protocol would help make passwords disposable. Making people change passwords often will result in people reusing more passwords.
      - cm2187 8y ago
        No the point is for password manager. The password manager would regularly reset all the password.... until someone accesses your password manager and locks you out of everything!
    - sjwright 8y ago
      If by protocol you mean a standard, consistent API that can be used by password managers to update passwords automatically, then I completely agree.
  - 8y ago
philip1209 8y ago
In the past, I've seen logs monitored for high-entropy strings that could be API keys or passwords. However, in a NoSQL/UUID-using environment, this could be really hard to implement.
- techdragon 8y ago
  Log line -> high entropy check -> false positive uuid check -> alerts I’m not seeing how it would be a challenge in a uuid based environment, unless there’s a nuanced detail I’m missing.
- maddyboo 8y ago
  Perhaps implement some type of “password canary” - some type of test account(s) with known high-entropy passwords. Have an automated system send periodic login requests (or any other requests which contain sensitive information that shouldn’t be logged) for this account, and have another system which searches log files for the password. If it’s ever found, you know something is leaking.
  - cm2187 8y ago
    And regularly check for that password on haveibeenpwned and other breached password databases.
    - AstralStorm 8y ago
      Do you trust the database to not have been hijacked to capture checked passwords? A better advice is to delete accounts you don't use. If not possible (illegal in EU now) scramble private data and the password. Download the databases yourself and check them locally. Changing passwords regularly also limits the damage.
hmwhy 8y ago
Genuine question—how would this bug be produced in the first place? My (limited) experience makes me think that cleartext passwords are somehow hard coded to be logged, perhaps through error logging or a feature that’s intended for testing during development. I personally would not code a backend that allows passwords (or any sensitive strings) to be logged in any shape or form in production, so it seems a little weird to me that this mistake is considered a “bug” instead of a very careless mistake. Am I missing something? EDIT: Thank you very much in advance!
- omarforgotpwd 8y ago
  Let's say you log requests and the POST body parameters that are sent along with them. Oops, forgot to explicitly blank out and fields known to contain passwords. Now they're saved in cleartext in the logs every time the user logs in.
  - ShabbosGoy 8y ago
    In the context of production, why would you need to log anything other than X-Forwarded-For/X-Real-IP, timestamp, and the endpoint that was hit?
    - methodover 8y ago
      Because when fatal exceptions happen you want to know what the request was. It helps debug what went wrong.
    - gutnor 8y ago
      Remember that the context is a bug. So sure you don't want to log everything in Prod, but maybe you do in Dev. In that case, a bug would be to push the dev logging configuration to Prod. Oops. If you have the clear text password at any point in your codebase, then there is no full-proof way to prevent to log it unintentionally as the result of a bug. You just have to be extra-careful ( code review, minimal amount code manipulating it, prod-like testing environment with log scanner, ...)
  - 8y ago
tzhenghao 8y ago
Hmm why should passwords (hashed or not) be stored in logs though? I don’t see a reason for doing that. You could unset it (and/or other sensitive data) before dumping them into logs.
- Zombieball 8y ago
  They shouldn’t. It was an unintentional bug
- patrickthebold 8y ago
  They shouldn't. It was a mistake.
- Leka74 8y ago
  Probably logging the HTTP/S requests, which included usernames & passwords in plaintext.
auscompgeek 8y ago
Wasn't this GitLab, not GitHub?
- m-p-3 8y ago
  From the email I received: "During the course of regular auditing, GitHub discovered that a recently introduced bug exposed a small number of users’ passwords to our internal logging system, including yours. We have corrected this, but you'll need to reset your password to regain access to your account."
TomK32 8y ago
Wouldn't have happend with Rails... http://api.rubyonrails.org/classes/ActionDispatch/Http/FilterParameters.html http://api.rubyonrails.org/classes/ActionDispatch/Http/Filte...
- mikesickler 8y ago
  Of course it could have! No API is foolproof
  - skolsuper 8y ago
    I think the joke is that both Github and Twitter are famous for being built on Rails (although Twitter just-as-famously required a move off of Rails in order to scale)
    - yebyen 8y ago
      There was a great keynote about this at this year's RailsConf The argument was essentially that if Twitter had instead chosen a language that was more natively inclined toward scalability, they would have necessarily hired 10x as many engineers and they would not have succeeded at building the product that people use to tell each other what bar they are at, simply, which ultimately was the braindead simple thing (that you can probably scale just fine in any language) which drove their success... it wasn't any great technological feat that made Twitter successful, it was pretty much just the "Bro" app that people loved. (The talk was called "Rails Doesn't Scale" and will be available soon, but RailsConf2018 evidently hasn't posted any of the videos yet.)
MichaelMoser123 8y ago
So the us commander in chief can now be impersonated on twitter? I am shocked!
mohitmun 8y ago
Is it there on the github blog? Any links would be appreciated
- Tepix 8y ago
  From what I've read it only applied to a small number of users and they were notified by email.
mtgx 8y ago
Which makes me wonder, is this really a bug or did someone make it look like a "bug"? Also they say they found no evidence of anyone stealing these passwords, but I wouldn't be surprised if some companies decide not to look too hard just so they can later say "they found no evidence of such an act."
Bromskloss 8y ago
Wouldn't it be better to never even send the password to the server, but instead performing a challenge-response procedure? Does HTTP have no such feature built in?
Bromskloss 8y ago
> implementing plans to prevent this bug from happening again How does one do that?
- rscafi 8y ago
  Being simplistic, perhaps an automated test with a known password on account creation or login (e.g. "dolphins") and then a search for the value on all generated logs.
bonyt 8y ago
Sounds like the same kind of thing that happened with APFS encryption passwords recently, too. https://www.mac4n6.com/blog/2018/3/30/omg-seriously-apfs-encrypted-plaintext-password-found-in-another-more-persistent-macos-log-file https://www.mac4n6.com/blog/2018/3/30/omg-seriously-apfs-enc...
- dkonofalski 8y ago
  Where is it showing a password there? I assume this has been fixed because I can't duplicate it on my machine and the screenshot posted on that article doesn't seem to show any plaintext passwords.
sjapkee 8y ago
>Due to a bug >Write passwords to a log Security level - Twitter.

xd1936 8y ago

"[We] are implementing plans to prevent this bug from happening again" sure makes it sound like this bug is still happening. Should we wait a couple of days before changing passwords? Will it end up in this log right now, just like the old one?

wlesieutre 8y ago
That sounds more like "We're adding a more thorough testing and code-review process for our password systems to prevent developers from accidentally logging unhashed passwords in the future".
SahAssar 8y ago
~~Sounds more like "we fixed this bug, and will ignore the processes that led to it happening" bullshit to me.~~
- always_good 8y ago
  And this comment sounds more like "DAE hate twitter." Their response is acceptable and textbook. Doesn't really seem like the appropriate place to wage the battle.
  - SahAssar 8y ago
    Yeah, bad kneejerk response on my part. Sorry.
reificator 8y ago
No, it sounds like a reasonable bugfixing strategy. Identify the bug, identify the fastest way to resolve it, then once it's fixed figure out how to ensure it never happens again, and what to do if it does.
bichiliad 8y ago
I think you read "prevent this bug from happening again" to mean "prevent this particular problem from happening one more time", while the blogpost probably means something like "prevent this class of bug from occurring in the future"

jahvo 8y ago

It's funny, I wonder if hearing about that github bug made them check if they had committed the same mistake... only to find that they did :-)

badloginagain 8y ago
I think I, and everyone here, should check as well. If capable, security-minded companies can make such a mistake, so can you.
- milesokeefe 8y ago
  Yep, glad I read this thread. We were making the same simple mistake.
  - bigiain 8y ago
    We aren't. Now. (We caught ourselves doing it 4-5 months back, and went through _everything_ checking... Only random accident that brought it to the attention of anyone who bothered to question it too... Two separate instances by different devs of 'if (DEBUG_LEVEL = 3){ }' instead of == 3 - both missed by code reviews too...)
    - jcoffland 8y ago
      This is why you should turn on compiler warnings and heed them. It would have caught this.
      - bch 8y ago
        And consider “Yoda Notation”[0], which some people find annoying, but I found an easy hurdle to clear: if ( 3 = DEBUGLEVEL ) wouldn’t pass the the parser because you can’t assign to an rvalue. [0] https://en.wikipedia.org/wiki/Yoda_conditions https://en.wikipedia.org/wiki/Yoda_conditions
        8y ago

ams6110 8y ago

So best practice would be that the cleartext password is never sent to the server, so they could never log it even accidentally. That means the hashing needs to be done client side, probably with JavaScript. Is there any safe way to do that?

jrochkind1 8y ago
nah, that just makes the "hashed password" the equivalent of the cleartext password. Whatever it is your client sends to the server for auth is the thing that needs to be protected. If the client sends a "hashed password", that's just... the password. Which now needs to be protected. Since if someone has it, they can just send it to the server for auth. But you can do fancy cryptographic things where the server never sees the password and it's still secure. like the entire field of public key cryptography, diffie-hellman key exchange, etc.
- ams6110 8y ago
  Makes sense, and then you're getting into something akin to SSH key pairs, and I know from experience that many users can't manage that especially across multiple client devices.
  - jrochkind1 8y ago
    There are probably ways to make it reasonable UX, but they probably require built-in browser (or other client) support. Someone in another part of this thread mentioned the "Web Authentication API" for browsers, which I'm not familiar with, but is possibly trying to approach this?
    - tehbeard 8y ago
      Web Auth API (authn) does try to make it usable. It ties in with the credential management API (A way to have the browser store login credentials for a site, a much less heuristic based approach than autocomplete on forms) and basic principle is generate a key pair, pass back public key to be sent to server during registration. On login generate a challenge value for the client to sign. I don't think iirc the JS code ever sees the private key, only the browser sees it.

theCricketer 8y ago

For people that know more about web security than I: Is there a reason it isn't good practice to hash the password client side so that the backends only ever see the hashed password and there is no chance for such mistakes?

zackbloom 8y ago
Realize the point of hashing the password is to make sure the thing users send to you is different than the thing you store. You'll still have to hash the hashes again on your end, otherwise anyone who gets accessed to your stored passwords could use them to login.
- pishpash 8y ago
  But at least, with salt, it wouldn't be applicable to other sites, just one. Better to just never reuse a password though. Honestly sites should just standardize on a password changing protocol, that will go a long way towards making passwords actually disposable.
  - stingrae 8y ago
    I don't think a password changing protocol would help make passwords disposable. Making people change passwords often will result in people reusing more passwords.
    - cm2187 8y ago
      No the point is for password manager. The password manager would regularly reset all the password.... until someone accesses your password manager and locks you out of everything!
  - sjwright 8y ago
    If by protocol you mean a standard, consistent API that can be used by password managers to update passwords automatically, then I completely agree.
- 8y ago

philip1209 8y ago

In the past, I've seen logs monitored for high-entropy strings that could be API keys or passwords. However, in a NoSQL/UUID-using environment, this could be really hard to implement.

techdragon 8y ago
Log line -> high entropy check -> false positive uuid check -> alerts I’m not seeing how it would be a challenge in a uuid based environment, unless there’s a nuanced detail I’m missing.
maddyboo 8y ago
Perhaps implement some type of “password canary” - some type of test account(s) with known high-entropy passwords. Have an automated system send periodic login requests (or any other requests which contain sensitive information that shouldn’t be logged) for this account, and have another system which searches log files for the password. If it’s ever found, you know something is leaking.
- cm2187 8y ago
  And regularly check for that password on haveibeenpwned and other breached password databases.
  - AstralStorm 8y ago
    Do you trust the database to not have been hijacked to capture checked passwords? A better advice is to delete accounts you don't use. If not possible (illegal in EU now) scramble private data and the password. Download the databases yourself and check them locally. Changing passwords regularly also limits the damage.

hmwhy 8y ago

Genuine question—how would this bug be produced in the first place? My (limited) experience makes me think that cleartext passwords are somehow hard coded to be logged, perhaps through error logging or a feature that’s intended for testing during development. I personally would not code a backend that allows passwords (or any sensitive strings) to be logged in any shape or form in production, so it seems a little weird to me that this mistake is considered a “bug” instead of a very careless mistake. Am I missing something? EDIT: Thank you very much in advance!

omarforgotpwd 8y ago
Let's say you log requests and the POST body parameters that are sent along with them. Oops, forgot to explicitly blank out and fields known to contain passwords. Now they're saved in cleartext in the logs every time the user logs in.
- ShabbosGoy 8y ago
  In the context of production, why would you need to log anything other than X-Forwarded-For/X-Real-IP, timestamp, and the endpoint that was hit?
  - methodover 8y ago
    Because when fatal exceptions happen you want to know what the request was. It helps debug what went wrong.
  - gutnor 8y ago
    Remember that the context is a bug. So sure you don't want to log everything in Prod, but maybe you do in Dev. In that case, a bug would be to push the dev logging configuration to Prod. Oops. If you have the clear text password at any point in your codebase, then there is no full-proof way to prevent to log it unintentionally as the result of a bug. You just have to be extra-careful ( code review, minimal amount code manipulating it, prod-like testing environment with log scanner, ...)
- 8y ago

tzhenghao 8y ago

Hmm why should passwords (hashed or not) be stored in logs though? I don’t see a reason for doing that. You could unset it (and/or other sensitive data) before dumping them into logs.

Zombieball 8y ago
They shouldn’t. It was an unintentional bug
patrickthebold 8y ago
They shouldn't. It was a mistake.
Leka74 8y ago
Probably logging the HTTP/S requests, which included usernames & passwords in plaintext.

auscompgeek 8y ago

Wasn't this GitLab, not GitHub?

m-p-3 8y ago
From the email I received: "During the course of regular auditing, GitHub discovered that a recently introduced bug exposed a small number of users’ passwords to our internal logging system, including yours. We have corrected this, but you'll need to reset your password to regain access to your account."

TomK32 8y ago

Wouldn't have happend with Rails... http://api.rubyonrails.org/classes/ActionDispatch/Http/FilterParameters.html http://api.rubyonrails.org/classes/ActionDispatch/Http/Filte...

mikesickler 8y ago
Of course it could have! No API is foolproof
- skolsuper 8y ago
  I think the joke is that both Github and Twitter are famous for being built on Rails (although Twitter just-as-famously required a move off of Rails in order to scale)
  - yebyen 8y ago
    There was a great keynote about this at this year's RailsConf The argument was essentially that if Twitter had instead chosen a language that was more natively inclined toward scalability, they would have necessarily hired 10x as many engineers and they would not have succeeded at building the product that people use to tell each other what bar they are at, simply, which ultimately was the braindead simple thing (that you can probably scale just fine in any language) which drove their success... it wasn't any great technological feat that made Twitter successful, it was pretty much just the "Bro" app that people loved. (The talk was called "Rails Doesn't Scale" and will be available soon, but RailsConf2018 evidently hasn't posted any of the videos yet.)

MichaelMoser123 8y ago

So the us commander in chief can now be impersonated on twitter? I am shocked!

mohitmun 8y ago

Is it there on the github blog? Any links would be appreciated

Tepix 8y ago
From what I've read it only applied to a small number of users and they were notified by email.

mtgx 8y ago

Which makes me wonder, is this really a bug or did someone make it look like a "bug"? Also they say they found no evidence of anyone stealing these passwords, but I wouldn't be surprised if some companies decide not to look too hard just so they can later say "they found no evidence of such an act."

Bromskloss 8y ago

Wouldn't it be better to never even send the password to the server, but instead performing a challenge-response procedure? Does HTTP have no such feature built in?

Bromskloss 8y ago

> implementing plans to prevent this bug from happening again How does one do that?

rscafi 8y ago
Being simplistic, perhaps an automated test with a known password on account creation or login (e.g. "dolphins") and then a search for the value on all generated logs.

bonyt 8y ago

Sounds like the same kind of thing that happened with APFS encryption passwords recently, too. https://www.mac4n6.com/blog/2018/3/30/omg-seriously-apfs-encrypted-plaintext-password-found-in-another-more-persistent-macos-log-file https://www.mac4n6.com/blog/2018/3/30/omg-seriously-apfs-enc...

dkonofalski 8y ago
Where is it showing a password there? I assume this has been fixed because I can't duplicate it on my machine and the screenshot posted on that article doesn't seem to show any plaintext passwords.

sjapkee 8y ago

>Due to a bug >Write passwords to a log Security level - Twitter.