9 ms·

Heroku CI and Review App Secrets Compromised

249 points by himeexcelanta 4y ago

Just got an email from Salesforce: "Action Required: Heroku security notification". Looks like the database that stores pipeline-level config variables for both Review Apps and Heroku CI were compromised. Per Heroku, "...any secrets you set in Review Apps and Heroku CI config may have been compromised and should be rotated". This...is really messed up :/

vmception 4y ago
oh shit. I'm surprised we haven't heard about major services getting hacked to oblivion right now, so much is stored in environment variables are there any mystery hacks occurring yet? is this database known to have been spread anywhere?
- jacobsenscott 4y ago
  Hopefully most people didn't have production creds stored in pipeline config. These vars were for review apps and ci.
oxff 4y ago
My sides, what a shitshow.
daudmalik06 4y ago
This is why we at vulert.com never access the customer's codebase or any installation, who doesn't know vulert, it's a service that notifies you for security issues in your software dependencies.
Mandatum 4y ago
Sounds like a customer's canary token triggered this based on the current reporting. "Trust is our Number 1 value."
- ubertaco 4y ago
  For folks employed by Salesforce, the phrase "Trust is our number 1 value" only comes up in two contexts: 1. company all-hands meetings, which are basically pep rallies with no actual content 2. when someone working at Salesforce brings up a glaring problem and says "if Trust is our number 1 value, why don't we do something about this huge problem?", which is usually met with either silence and bureaucratic obstacles or with excuses, usually something like "customers trust us to spend the money they pay us building the features and products they want", which is like...exactly not the definition used at any of the pep rallies.
  - hthrowaway5 4y ago
    that and dreamforce (over and over again)
mepiethree 4y ago
11 days ago they said "While we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets." I guess that was a lie?!
- himeexcelanta 4y ago
  HN with the quality security advice, with all the recommendations to rotate config vars just to be safe.
  - bigDinosaur 4y ago
    I can't think of a reason to not rotate credentials and variables the second you see a security incident of this scale even when it was in the very earliest stages. Better safe than sorry, and also a good time to review just how easy it is to update all the variables (is it automated & scripted, where are they stored and generated, etc.)
- bibinou 4y ago
  the subsequent blog post (https://blog.heroku.com/we-heard-your-feedback https://blog.heroku.com/we-heard-your-feedback) says: > Additionally, we have no evidence that the attacker has accessed any customer accounts or decrypted customers’ environment variables. which, as pointed out in its HN thread, means "we now know they got access to encrypted vars, and we don't know yet if they could have decrypted them." in BS-speak. The title "We've Heard Your Feedback" is also a red herring, usually means "we know we fucked up bad and we still have no idea of the whole impact of the breach".
hthrowaway5 4y ago
Yep, they outright lied about env vars. Incredible. It pains me to see even occasional defenders of Heroku. They're not the company they were 10 years ago. They've been gutted and left for dead years ago but the product was so good nobody noticed until now. They're not to be trusted as your platform. They simply don't have anywhere close to the manpower required to run such a platform. This was a when not if situation. If you're still on it, make your plans to move away now. Time is ticking until a major outage or another security incident like this one. See my comment history and related threads for more. Specifically this summary: https://news.ycombinator.com/item?id=31374048 https://news.ycombinator.com/item?id=31374048
- colesantiago 4y ago
  I would like to move but there are really no good alternatives that are even close to Heroku.
  - hthrowaway5 4y ago
    Well hopefully once it's gone the competition will be able to get more market share to build quality product. Heroku has been starving the entire ecosystem for years. I don't have experience with any other PaaS's so I can't recommend one, but what you say is what I commonly hear.
    - ezekg 4y ago
      Heroku hasn’t been starving the ecosystem. They simply haven’t had real competition on their caliber of (zero-)devops.
    - glenngillen 4y ago
      This makes no sense. Heroku have had no competition because nobody has built a better product. They’ve not been starving anyone or anything. Given the biggest and most common complaint most lay against Heroku is that it’s too expensive, if anything the lack of innovation for years has created a huge window for a competitor. And yet here we are. Still.
bradleybuda 4y ago
Text of the email: At Salesforce, we understand that the confidentiality, integrity, and availability of your data is vital to your business, and we take the protection of your data very seriously. We value transparency and wanted to notify you of an issue affecting your account. Based on current progress, we plan to complete our investigation by May 30, 2022. We are continuing with remediation activities and plan to publish additional information about the incident once it’s resolved. As reported on status.heroku.com, on April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. On that same day, the threat actor downloaded data from another database that stores pipeline-level config vars for Review Apps and Heroku CI. This was identified on May 16, 2022, after further forensic investigation. We have no evidence of any unauthorized access to Heroku systems since April 14, 2022. As a result, any secrets you set in Review Apps and Heroku CI config vars may have been compromised and should be rotated. In addition, any Heroku tokens stored in these pipeline config vars would potentially have allowed access to your Heroku account between April 7, 2022 and May 5, 2022, when your passwords were reset, invalidating all Heroku tokens as a result. Please note, these pipeline-level config vars are different from standard app config vars. App config vars were not stored in this database and we have no evidence to suggest app config vars were compromised.
- hthrowaway5 4y ago
  > At Salesforce, we understand that the confidentiality, integrity, and availability of your data is vital to your business [...] Hey Bob, why didn't you tell your customers a month ago to rotate their creds just to be safe? This is flat out insulting.
  - krono 4y ago
    > At salesforce.com, inc., trust is our #1 value Their legal pages[1] are filled to the brim with those ridiculous statements. I never understood why they'd even bother making it sound nice, especially not for B2B. Customers won't trust the message and likely can't use them in court, and they themselves must surely know they're creating expectations that they can't guarantee to meet. [1] https://www.salesforce.com/company/legal https://www.salesforce.com/company/legal
nameless912 4y ago
Yup, that's game, set, and match. I really feel bad for all the herokai still left holding the line, but damn am I glad I got out when I did.
hkhanna 4y ago
I spent the last two days migrating my company to Render from Heroku, and now I'm glad I did. Render is a little rough around the edges; Heroku is far more polished. But it's probably to Render's credit that, in my opinion, the most annoying thing about Render is that it's impossible to google about Render because "render" is such a common word in the tech world! Their support is good and responsive, and the developer experience was good enough. It has some warts, and there were definitely times I missed Heroku, but their speed of improvement gives me confidence in their future. Sad to leave Heroku after almost a decade with them. They were far ahead of their time.
- rychco 4y ago
  I also intend to deploy some new services on render (having previously used Heroku). I was debating between render & fly, which I've also had my eye on and may still try for something else in the future.
  - heartbreak 4y ago
    I think Fly will be great, but it's not there yet. I migrated to Render.
    - Rastonbury 4y ago
      I am considering moving, could you share why Fly wasn't adequate for you?
- tdfx 4y ago
  We just started planning our transition, as well. This was handled so poorly I can't imagine anyone would ever plan to start a new project on their platform.
  - wlll 4y ago
    What happened?
heartbreak 4y ago
I completely stopped getting their update emails when I deleted all of my running apps. This is the second one I've seen on HN that I haven't received (though I received the others). Is the impact limited to specific customer accounts, or are they just not updating me anymore?
- Ozzie_osman 4y ago
  I think it's specific to apps with Review Apps or Heroku CI (specifically the Config Vars). If you didn't have those, based on this email, you may not be impacted.
  - heartbreak 4y ago
    I had both of those, and I had config vars for my pipeline apps.
- hthrowaway5 4y ago
  From the email: > We value transparency and wanted to notify you of an issue affecting your account. My guess is they sent it to users with pipelines that have env vars. It's funny since this sentence demonstrates they don't value transparency by not telling the other users more information about the hack. They updated Heroku Status but surprisingly failed to mention anything about CI or pipelines.
gault8121 4y ago
Has anyone done a load test comparison for Heroku vs. Render.com? The "Pro Ultra" on Render is $450/month for 32 GB RAM + 8 CPU. The Heroku Performance L Dyno is $500 a month for 14 GB RAM. The Render server seems like a much better offering.
- oxff 4y ago
  Has anyone done a security review of Heroku alternatives? I see lots of comments about moving there and using them instead of Heroku because of this incident. But that's hardly a justifiable reason alone, since those new platforms are probably worse off in security.
  - beninsydney 4y ago
    Fly have https://news.ycombinator.com/user?id=tptacek https://news.ycombinator.com/user?id=tptacek working for them so it's hard to imagine they would get security wrong.
  - ryanSrich 4y ago
    Is render SOC 2 certified? ISO 27001? Those are fairly standard. If they don’t have either it’s going to be a hard time getting any serious business to migrate.
    - anurag 4y ago
      Render will have these certifications late summer (we're currently being audited for both).
  - deleted 4y ago
    [deleted]
  - njp77 4y ago
    Aptible has always prioritized security, and is a competitive alternative. https://www.aptible.com/heroku-alternative/ https://www.aptible.com/heroku-alternative/
kaycebasques 4y ago
Any guesses as to how the "threat actor" got access to the databases? I understand most guesses would be conjecture (unless someone here has an inside scoop). Just curious about how stuff like this usually gets compromised.
jamespetercook 4y ago
Slightly off-topic, but can anyone tell me how you’d know that your database has been accessed by a threat actor? Should I be periodically reviewing all my logs for something unusual?
- MarkMarine 4y ago
  Yep. Quick and dirty you could alert on large or slow queries, and check the logs periodically. I know it’s probably not effective but I grep logs and watch the terminal looking for aberrant shapes. I believe AWS offers a ML solution to watch your infra and alert for things that are out of the usual, and I’m sure (haven’t built it, but talked to people that worked in the systems) the big companies have sophisticated systems looking for threats that use everything above and far more.
boesboes 4y ago
Still havent had any notification on this. A+ for Heroku
itsmeste 4y ago
When Salesforce bought Heroku back in 2011, it was pretty clear Heroku would become yet another dead product that once was an absolute great piece of software. Why? Commercialism. Founders sell to the highest bidder to make their exit worthwhile for themselves, not caring about the future of the product (and customers). It's a no-brainer that a commercial company like Salesforce (it's in their name!) doesn't have what it takes to build AAA software, but focuses on maximizing their profit. They drove away their best staff, focused on the wrong features, and are seemingly overwhelmed by maintaining their purchased software, all while probably not even realizing their demise. We should all come to the agreement that takeovers of fundamental software by incompetent companies should be seen as a hostility towards every current user of said software.
- alx__ 4y ago
  > When Salesforce bought Heroku back in 2011, it was pretty clear Heroku would become yet another dead product that once was an absolute great piece of software. That feels like a angsty-tinted view. I recall the day it happened. The Ruby dev shop I was at was optimistically nervous. As Heroku had been a shiny new thing and only deployed Ruby. Acquisition allowed them to expand and support other languages. They didn't even have pipelines! https://techcrunch.com/2010/12/08/breaking-salesforce-buys-heroku-for-212-million-in-cash/ https://techcrunch.com/2010/12/08/breaking-salesforce-buys-h...
zevir 4y ago
For those looking for a great alternative to Review Apps - Livecycle is great (https://livecycle.io/ https://livecycle.io/). It offers the automated per-PR ephemeral environment and much more. It also includes a rich layer of built-in collaboration and annotation tools that allow all collaborators to join the PR review and leave their comments visually, on top of the product UI. The comments are maintained in Livecycle and also synced back to Git as review comments so that developers can see the issues faster, understand them better and address them sooner. There are setup templates that make it easy to simply copy over your docker file and get started within a few minutes. And the team is eager to help if you have any questions or issues.

249 points by himeexcelanta 4y ago

vmception 4y ago

oh shit. I'm surprised we haven't heard about major services getting hacked to oblivion right now, so much is stored in environment variables are there any mystery hacks occurring yet? is this database known to have been spread anywhere?

jacobsenscott 4y ago
Hopefully most people didn't have production creds stored in pipeline config. These vars were for review apps and ci.

oxff 4y ago

My sides, what a shitshow.

daudmalik06 4y ago

This is why we at vulert.com never access the customer's codebase or any installation, who doesn't know vulert, it's a service that notifies you for security issues in your software dependencies.

Mandatum 4y ago

Sounds like a customer's canary token triggered this based on the current reporting. "Trust is our Number 1 value."

ubertaco 4y ago
For folks employed by Salesforce, the phrase "Trust is our number 1 value" only comes up in two contexts: 1. company all-hands meetings, which are basically pep rallies with no actual content 2. when someone working at Salesforce brings up a glaring problem and says "if Trust is our number 1 value, why don't we do something about this huge problem?", which is usually met with either silence and bureaucratic obstacles or with excuses, usually something like "customers trust us to spend the money they pay us building the features and products they want", which is like...exactly not the definition used at any of the pep rallies.
- hthrowaway5 4y ago
  that and dreamforce (over and over again)

mepiethree 4y ago

11 days ago they said "While we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets." I guess that was a lie?!

himeexcelanta 4y ago
HN with the quality security advice, with all the recommendations to rotate config vars just to be safe.
- bigDinosaur 4y ago
  I can't think of a reason to not rotate credentials and variables the second you see a security incident of this scale even when it was in the very earliest stages. Better safe than sorry, and also a good time to review just how easy it is to update all the variables (is it automated & scripted, where are they stored and generated, etc.)
bibinou 4y ago
the subsequent blog post (https://blog.heroku.com/we-heard-your-feedback https://blog.heroku.com/we-heard-your-feedback) says: > Additionally, we have no evidence that the attacker has accessed any customer accounts or decrypted customers’ environment variables. which, as pointed out in its HN thread, means "we now know they got access to encrypted vars, and we don't know yet if they could have decrypted them." in BS-speak. The title "We've Heard Your Feedback" is also a red herring, usually means "we know we fucked up bad and we still have no idea of the whole impact of the breach".

hthrowaway5 4y ago

Yep, they outright lied about env vars. Incredible. It pains me to see even occasional defenders of Heroku. They're not the company they were 10 years ago. They've been gutted and left for dead years ago but the product was so good nobody noticed until now. They're not to be trusted as your platform. They simply don't have anywhere close to the manpower required to run such a platform. This was a when not if situation. If you're still on it, make your plans to move away now. Time is ticking until a major outage or another security incident like this one. See my comment history and related threads for more. Specifically this summary: https://news.ycombinator.com/item?id=31374048 https://news.ycombinator.com/item?id=31374048

colesantiago 4y ago
I would like to move but there are really no good alternatives that are even close to Heroku.
- hthrowaway5 4y ago
  Well hopefully once it's gone the competition will be able to get more market share to build quality product. Heroku has been starving the entire ecosystem for years. I don't have experience with any other PaaS's so I can't recommend one, but what you say is what I commonly hear.
  - ezekg 4y ago
    Heroku hasn’t been starving the ecosystem. They simply haven’t had real competition on their caliber of (zero-)devops.
  - glenngillen 4y ago
    This makes no sense. Heroku have had no competition because nobody has built a better product. They’ve not been starving anyone or anything. Given the biggest and most common complaint most lay against Heroku is that it’s too expensive, if anything the lack of innovation for years has created a huge window for a competitor. And yet here we are. Still.

bradleybuda 4y ago

Text of the email: At Salesforce, we understand that the confidentiality, integrity, and availability of your data is vital to your business, and we take the protection of your data very seriously. We value transparency and wanted to notify you of an issue affecting your account. Based on current progress, we plan to complete our investigation by May 30, 2022. We are continuing with remediation activities and plan to publish additional information about the incident once it’s resolved. As reported on status.heroku.com, on April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. On that same day, the threat actor downloaded data from another database that stores pipeline-level config vars for Review Apps and Heroku CI. This was identified on May 16, 2022, after further forensic investigation. We have no evidence of any unauthorized access to Heroku systems since April 14, 2022. As a result, any secrets you set in Review Apps and Heroku CI config vars may have been compromised and should be rotated. In addition, any Heroku tokens stored in these pipeline config vars would potentially have allowed access to your Heroku account between April 7, 2022 and May 5, 2022, when your passwords were reset, invalidating all Heroku tokens as a result. Please note, these pipeline-level config vars are different from standard app config vars. App config vars were not stored in this database and we have no evidence to suggest app config vars were compromised.

hthrowaway5 4y ago
> At Salesforce, we understand that the confidentiality, integrity, and availability of your data is vital to your business [...] Hey Bob, why didn't you tell your customers a month ago to rotate their creds just to be safe? This is flat out insulting.
- krono 4y ago
  > At salesforce.com, inc., trust is our #1 value Their legal pages[1] are filled to the brim with those ridiculous statements. I never understood why they'd even bother making it sound nice, especially not for B2B. Customers won't trust the message and likely can't use them in court, and they themselves must surely know they're creating expectations that they can't guarantee to meet. [1] https://www.salesforce.com/company/legal https://www.salesforce.com/company/legal

nameless912 4y ago

Yup, that's game, set, and match. I really feel bad for all the herokai still left holding the line, but damn am I glad I got out when I did.

hkhanna 4y ago

I spent the last two days migrating my company to Render from Heroku, and now I'm glad I did. Render is a little rough around the edges; Heroku is far more polished. But it's probably to Render's credit that, in my opinion, the most annoying thing about Render is that it's impossible to google about Render because "render" is such a common word in the tech world! Their support is good and responsive, and the developer experience was good enough. It has some warts, and there were definitely times I missed Heroku, but their speed of improvement gives me confidence in their future. Sad to leave Heroku after almost a decade with them. They were far ahead of their time.

rychco 4y ago
I also intend to deploy some new services on render (having previously used Heroku). I was debating between render & fly, which I've also had my eye on and may still try for something else in the future.
- heartbreak 4y ago
  I think Fly will be great, but it's not there yet. I migrated to Render.
  - Rastonbury 4y ago
    I am considering moving, could you share why Fly wasn't adequate for you?
tdfx 4y ago
We just started planning our transition, as well. This was handled so poorly I can't imagine anyone would ever plan to start a new project on their platform.
- wlll 4y ago
  What happened?

heartbreak 4y ago

I completely stopped getting their update emails when I deleted all of my running apps. This is the second one I've seen on HN that I haven't received (though I received the others). Is the impact limited to specific customer accounts, or are they just not updating me anymore?

Ozzie_osman 4y ago
I think it's specific to apps with Review Apps or Heroku CI (specifically the Config Vars). If you didn't have those, based on this email, you may not be impacted.
- heartbreak 4y ago
  I had both of those, and I had config vars for my pipeline apps.
hthrowaway5 4y ago
From the email: > We value transparency and wanted to notify you of an issue affecting your account. My guess is they sent it to users with pipelines that have env vars. It's funny since this sentence demonstrates they don't value transparency by not telling the other users more information about the hack. They updated Heroku Status but surprisingly failed to mention anything about CI or pipelines.

gault8121 4y ago

Has anyone done a load test comparison for Heroku vs. Render.com? The "Pro Ultra" on Render is $450/month for 32 GB RAM + 8 CPU. The Heroku Performance L Dyno is $500 a month for 14 GB RAM. The Render server seems like a much better offering.

oxff 4y ago
Has anyone done a security review of Heroku alternatives? I see lots of comments about moving there and using them instead of Heroku because of this incident. But that's hardly a justifiable reason alone, since those new platforms are probably worse off in security.
- beninsydney 4y ago
  Fly have https://news.ycombinator.com/user?id=tptacek https://news.ycombinator.com/user?id=tptacek working for them so it's hard to imagine they would get security wrong.
- ryanSrich 4y ago
  Is render SOC 2 certified? ISO 27001? Those are fairly standard. If they don’t have either it’s going to be a hard time getting any serious business to migrate.
  - anurag 4y ago
    Render will have these certifications late summer (we're currently being audited for both).
- deleted 4y ago
  [deleted]
- njp77 4y ago
  Aptible has always prioritized security, and is a competitive alternative. https://www.aptible.com/heroku-alternative/ https://www.aptible.com/heroku-alternative/

kaycebasques 4y ago

Any guesses as to how the "threat actor" got access to the databases? I understand most guesses would be conjecture (unless someone here has an inside scoop). Just curious about how stuff like this usually gets compromised.

jamespetercook 4y ago

Slightly off-topic, but can anyone tell me how you’d know that your database has been accessed by a threat actor? Should I be periodically reviewing all my logs for something unusual?

MarkMarine 4y ago
Yep. Quick and dirty you could alert on large or slow queries, and check the logs periodically. I know it’s probably not effective but I grep logs and watch the terminal looking for aberrant shapes. I believe AWS offers a ML solution to watch your infra and alert for things that are out of the usual, and I’m sure (haven’t built it, but talked to people that worked in the systems) the big companies have sophisticated systems looking for threats that use everything above and far more.

boesboes 4y ago

Still havent had any notification on this. A+ for Heroku

itsmeste 4y ago

When Salesforce bought Heroku back in 2011, it was pretty clear Heroku would become yet another dead product that once was an absolute great piece of software. Why? Commercialism. Founders sell to the highest bidder to make their exit worthwhile for themselves, not caring about the future of the product (and customers). It's a no-brainer that a commercial company like Salesforce (it's in their name!) doesn't have what it takes to build AAA software, but focuses on maximizing their profit. They drove away their best staff, focused on the wrong features, and are seemingly overwhelmed by maintaining their purchased software, all while probably not even realizing their demise. We should all come to the agreement that takeovers of fundamental software by incompetent companies should be seen as a hostility towards every current user of said software.

alx__ 4y ago
> When Salesforce bought Heroku back in 2011, it was pretty clear Heroku would become yet another dead product that once was an absolute great piece of software. That feels like a angsty-tinted view. I recall the day it happened. The Ruby dev shop I was at was optimistically nervous. As Heroku had been a shiny new thing and only deployed Ruby. Acquisition allowed them to expand and support other languages. They didn't even have pipelines! https://techcrunch.com/2010/12/08/breaking-salesforce-buys-heroku-for-212-million-in-cash/ https://techcrunch.com/2010/12/08/breaking-salesforce-buys-h...

zevir 4y ago

For those looking for a great alternative to Review Apps - Livecycle is great (https://livecycle.io/ https://livecycle.io/). It offers the automated per-PR ephemeral environment and much more. It also includes a rich layer of built-in collaboration and annotation tools that allow all collaborators to join the PR review and leave their comments visually, on top of the product UI. The comments are maintained in Livecycle and also synced back to Git as review comments so that developers can see the issues faster, understand them better and address them sooner. There are setup templates that make it easy to simply copy over your docker file and get started within a few minutes. And the team is eager to help if you have any questions or issues.