6 ms·

Git ransom campaign incident report

181 points by brntn 7y ago

rossdavidh 7y ago
"All of this has happened before, and it will all happen again." - Peter Pan movie, Battlestar Galactica, and should be every security incident report ever. Not saying they shouldn't have issued their analysis, of course they should have, it mostly looks on target. But...it will all happen again.
gumby 7y ago
Since literally everybody who has cloned a repo has a full copy of it, and since git is a decentralized revision control system, what on earth can it mean to hold a repo for ransom? The write up even says so: to recover, just push your code back up to our repo. I really don't understand what they are talking about. It's as if someone showed me a photo of my child and said, "pay me or I'll burn this photograph". What am I missing?
- empath75 7y ago
  There’s probably a lot of code sitting in private git repos where like one guy worked on it five years ago and then quit the company and gitlab/github might be the only place it exists.
- unilynx 7y ago
  Private repositories. You may not want their contents to become public
- NateEag 7y ago
  The threat cited in the article said not just that the code would remain deleted, but that it would be "leaked" - presumably many of these were private repos. You could never trust that the attacker actually deleted their copy of the repo, but then, the whole cryptolocking business model falls down if the attacker isn't at least moderately honest, so I can see why people would respond to that threat.
  - Someone 7y ago
    ”the whole cryptolocking business model falls down if the attacker isn't at least moderately honest” Nitpick: it only requires most attackers to be somewhat honest. Having a few unscrupulous ones may make life harder for the “honest” ones, but they themselves can be better of, e.g. by, after receiving payment, demanding more money.
    - NateEag 7y ago
      Very fair point.
    - cwkoss 7y ago
      Is it more unethical to release an "honest cryptolocker" or one that lies and never gives the files, degrading the trust the entire cryptolocker grift relies on?
      - johnday 7y ago
        It's pretty obvious that it's worse to be an actual criminal, than someone who goes around and pretends to be one. In the same way that it's worse to shoot someone with an actual gun than to threaten to shoot them with a Nerf gun. The negative network effects on other scammers are also nice.
        kekebo 7y ago
        In this case both are actual criminals but one returns your data after payment while the other doesn't
        deleted 7y ago
        [deleted]
        vokep 7y ago
        I'm not so sure. An "honest cryptolocker" helps support more cryptolocker use, as people trust that if they pay the criminal they'll get their stuff If dishonest ones were the norm, than maybe cryptolocking would cannibalize itself as nobody would pay since they know its useless. So in a sense the dishonest one while having less ethical intention has more ethical results. But only at scale. Hmmm.
- ajross 7y ago
  How certain are you that the photo archive you use had a copy of that photo? I mean, yeah, most people have local trees. Inevitably someone won't, and like spam this is a volume scam.
- p1necone 7y ago
  In my experience there's enough people who don't understand how git works for a shotgun approach to find plenty of marks to fool.
  - bregma 7y ago
    Your experience converges with mine.
- rileymat2 7y ago
  I don't keep all my git code local, only the projects I am currently working on.
  - bigiain 7y ago
    Do you rely on just one 3rd party like gitlab/bitbucket/github to keep your _only_ copy of your non-current projects? That seems unwise. I don't have many local repos on this 128GB MacBookAir, but as well as BitBucket all the projects I have ever worked on are on other several machines and/or hard drives I have locally, and also zipped up in S3 buckets and on tarsnap. Like they say, there's two kinds of people. People who've lost important data because they didn't back it up properly, and people who haven't yet.
    - rovr138 7y ago
      > Through immediate independent investigations, all three companies observed that user accounts were compromised using legitimate credentials including passwords, app passwords, API keys, and personal access tokens. Part of your backup strategy depends on external services. Not necessarily in your case, but people who only have their backups externally on a service could be affected. > all the projects I have ever worked on are on other several machines and/or hard drives And depending on your strategy, since they're so distributed it could mean they're outdated repos. If not, and they pull automatically, they could be affected. Local backups also have issues. The disk might die, the data might be corrupted or any other myriad of things could happen. > People who've lost important data because they didn't back it up properly, and people who haven't yet. Is there such a thing as a perfect backup strategy?
      - bigiain 7y ago
        > Is there such a thing as a perfect backup strategy? At work, there's "Can meet contractually agreed RPO and RTO with 99.99% certainty". Automate the standard setup, and sleep well at night. Perfect. At home, there's "I've done enough that I think the next improvement is an unfeasibly large amount of extra time&money for an unreasonably small improvement". I've, for myself at home, settled on Apple's Time Machine backing up my Macs (and their phone/ipad iTunes backups) to a raid 10 set, that raid 10 set rsynced to another one at the opposite end of the house, and a weekly backup of that stored on a single drive that only powers up for 6 hours every Sunday night then powers back down again - so if my whole network gets breached and cryptolockered (for example) I'll still have at most 7 day old data at home. I also push that weekly backup out to S3 and tarsnap for off-site in-case-my-house-burns-down, or I've set it all on fire and moved to Belize scenarios... I've been running most of that for ~8 years now. I've called it "done", while not "perfect", its certainly good enough against "not-Mossad threat models". If Mossad or The NSA want to delete my backups, so be it - I'll go be a carpenter or a gardener or something.
jedberg 7y ago
I just have to say, props to Gitlab for being included in this. For a lot of enterprises that use Github and Bitbucket, this may be their first into to Gitlab.
Ancient 7y ago
Repo's with remaining ransom file: https://github.com/search?q=1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9DA&type=Code https://github.com/search?q=1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9...
- sverhagen 7y ago
  Weird, weren't GitHub and friends restoring these repositories for these unlucky users?
  - ralph84 7y ago
    I assume they're trying to contact the repo owners to see how they want to proceed. Even if the change seems obvious, changing the contents of a repo is definitely a case where the git host should involve the owner of the repo (barring ToS violations).
    - jredmond 7y ago
      This is the safer bet. The host should offer to restore from their own backups, but some customers may have already taken care of their own stuff.
ralph84 7y ago
2FA is great for the web UI, but none of these vendors make it particularly easy to enforce 2FA on the command line.
- nathanaldensr 7y ago
  Exactly. PATs are designed to circumvent human intervention (MFA) for authentication in order to support automation. I am very curious if there's a better way than PATs.
  - tonny747 7y ago
    ssh keys with a passphrase?
  - ekarulf 7y ago
    SSH keys on a YubiKey
- deleted 7y ago
  [deleted]
- nhumrich 7y ago
  An ssh key is 2FA
  - michaelmior 7y ago
    How so? An SSH key is a single factor. You could argue that a password-protected private key provides a second factor, but that still falls in the category of "something you know."
    - will4274 7y ago
      How many people can recite their SSH key? Surely an SSH key is "something you have".
      - wnevets 7y ago
heelhook 7y ago
Seems like no one fell for this though. https://www.blockchain.com/btc/address/1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9DA https://www.blockchain.com/btc/address/1ES14c7qLb5CYhLMUekct...
- avian 7y ago
  Probably a unique wallet in each message. It seems to be that way in scam email ransoms.
  - kristofferc 7y ago
    Doesn't seem that way: https://github.com/search?q=1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9DA&type=Code https://github.com/search?q=1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9...
penagwin 7y ago
Is it common that companies share intelligence like this? I think it's a wonderful idea, given they all operate on essentially the same service (git) they share similar security concerns.
- smudgymcscmudge 7y ago
  It took exceptional circumstances, but it was the right call in this case. All were working toward the same goal. We can all go back to our war after the white walkers are dealt with (don’t bring Cersei up or it ruins the analogy)
- WrtCdEvrydy 7y ago
  In the Cybersecurity field, yes! You'd be surprised how often you'll be rolling up to your competition to compare the virus files you pulled from each of your networks and reverse engineering on VMs together... then next week you have to pretend you hate them until something else goes wrong again.
deleted 7y ago
[deleted]
shapov 7y ago
I didn't see it mentioned in the article, but did any of the 3 companies confirm that the repos have been actually cloned as the attackers suggest?
mlindner 7y ago
1. Stop using 'git add .' This is a bad habit I see people keep suggesting to new git users. Stop recommending it and stop doing it. 2. Never store your password in .git/config. Why are you doing that? That shouldn't be stored in .git/config.
- benburleson 7y ago
  Why are people using passwords instead of keys?
  - morpheuskafka 7y ago
    Why do people even have passwords that don't require 2FA?
  - deleted 7y ago
    [deleted]
  - jwilk 7y ago
    "Stupid corporate firewall blocks SSH connections" would be my guess.
- darklajid 7y ago
  Maybe I misunderstood something, but I read the text as people (or .. services like the three doing the announcement) offering URLs containing an access token? As in https://example.com/mlindner/project1/821372asd1786d21das https://example.com/mlindner/project1/821372asd1786d21das or something? If you use this approach to manage access to a repository, then .. that gets stored in the .git/config. No need to store a password or something. (Then again, maybe I didn't understand the explanation correctly?)
falsedan 7y ago
> Otherwise, you can still clone the repository and make use of: git reflog or git fsck to find your last commit and change the HEAD. I don't understand: when I clone a repo, I get a copy of all the branches/tags and the commits they point to & the trees/blobs from those commits. If the repo is wiped, I get a single master branch with a single commit with a single tree and a single blob, and no reflog because that is local to the repo, and I (as a fresh cloner) haven't updated any refs. Perhaps they are thinking about a mirror clone? That still won't include the reflog, but you can at least find dangling commits and guess which one was master.
- falsedan 7y ago
  Oops, mirror clone only includes reachable commits. No idea what the 'Otherwise' clarification means then, it sounds bogus
chrischen 7y ago
How does one withdraw bitcoin to fiat or even use it without it being traceable? Are there laundering or anonymizing services for bitcoin withdrawals to fiat?
- leethargo 7y ago
  AFAIK there have been bitcoin laundering services for years (bitcoin in, bitcoin out). As for spending, there used to be pre-loaded credit cards that you could from bitcoin.
  - chrischen 7y ago
    Wouldn't the credit cards be traceable (record of where you spent it). It wouldn't necessarily tie you exactly but it'd create a large paper-trail.
- Legogris 7y ago
  The BTC could be used as payments for services or products which don't require identification (and holding "dirty" BTC is now the problem of the sand celler if they ever want to take it to traditional businesses in jurisdictions that enforce KYC for BTC payments). Criminals can also trade BTC for physical cash. They could also by some means (for example permissionless decentralized exchanges) convert it to a cryptocurrency with private transaction properties such as ZCash, Monero, Beam or GRIN and then back again. The "laundering services" you refer to (generally called "mixers") are still around but most of them have been shown reversible with high degrees of confidence. CoinJoin is the state of the art here, with the most well-known implementations being JoinMarket and Wasabi Wallet. But in general law enforcement and investigators are definitely wising up to cryptocurrencies and to be fully untraceable one has to go through a lot of hoops and not make a single mistake in the process. Even the above mentioned approaches can leak information that can be used to tie an individual to the transactions if not executed properly. Most likely they will use the same tried and true approach they would have used for stolen fiat funds; identity theft. I personally know people who have been drawn into a criminal investigation for money laundering because they had been initiated a transaction selling BTC on LocalBitcoins via bank transfer (unwise unless you know and trust the person), turned out already stolen BTC had been converted into fiat on a compromised bank account, which was then supposed to be converted into "clean" BTC again. Fortunately the investigation was already underway when the transaction happened, my friends bank account was blocked as a result when the transfer was initiated and the whole thing was sorted out in the end.

181 points by brntn 7y ago

"All of this has happened before, and it will all happen again." - Peter Pan movie, Battlestar Galactica, and should be every security incident report ever. Not saying they shouldn't have issued their analysis, of course they should have, it mostly looks on target. But...it will all happen again.

gumby 7y ago

Since literally everybody who has cloned a repo has a full copy of it, and since git is a decentralized revision control system, what on earth can it mean to hold a repo for ransom? The write up even says so: to recover, just push your code back up to our repo. I really don't understand what they are talking about. It's as if someone showed me a photo of my child and said, "pay me or I'll burn this photograph". What am I missing?

empath75 7y ago
There’s probably a lot of code sitting in private git repos where like one guy worked on it five years ago and then quit the company and gitlab/github might be the only place it exists.
unilynx 7y ago
Private repositories. You may not want their contents to become public
NateEag 7y ago
The threat cited in the article said not just that the code would remain deleted, but that it would be "leaked" - presumably many of these were private repos. You could never trust that the attacker actually deleted their copy of the repo, but then, the whole cryptolocking business model falls down if the attacker isn't at least moderately honest, so I can see why people would respond to that threat.
- Someone 7y ago
  ”the whole cryptolocking business model falls down if the attacker isn't at least moderately honest” Nitpick: it only requires most attackers to be somewhat honest. Having a few unscrupulous ones may make life harder for the “honest” ones, but they themselves can be better of, e.g. by, after receiving payment, demanding more money.
  - NateEag 7y ago
    Very fair point.
  - cwkoss 7y ago
    Is it more unethical to release an "honest cryptolocker" or one that lies and never gives the files, degrading the trust the entire cryptolocker grift relies on?
    - johnday 7y ago
      It's pretty obvious that it's worse to be an actual criminal, than someone who goes around and pretends to be one. In the same way that it's worse to shoot someone with an actual gun than to threaten to shoot them with a Nerf gun. The negative network effects on other scammers are also nice.
      - kekebo 7y ago
        In this case both are actual criminals but one returns your data after payment while the other doesn't
        deleted 7y ago
        [deleted]
      - vokep 7y ago
        I'm not so sure. An "honest cryptolocker" helps support more cryptolocker use, as people trust that if they pay the criminal they'll get their stuff If dishonest ones were the norm, than maybe cryptolocking would cannibalize itself as nobody would pay since they know its useless. So in a sense the dishonest one while having less ethical intention has more ethical results. But only at scale. Hmmm.
ajross 7y ago
How certain are you that the photo archive you use had a copy of that photo? I mean, yeah, most people have local trees. Inevitably someone won't, and like spam this is a volume scam.
p1necone 7y ago
In my experience there's enough people who don't understand how git works for a shotgun approach to find plenty of marks to fool.
- bregma 7y ago
  Your experience converges with mine.
rileymat2 7y ago
I don't keep all my git code local, only the projects I am currently working on.
- bigiain 7y ago
  Do you rely on just one 3rd party like gitlab/bitbucket/github to keep your _only_ copy of your non-current projects? That seems unwise. I don't have many local repos on this 128GB MacBookAir, but as well as BitBucket all the projects I have ever worked on are on other several machines and/or hard drives I have locally, and also zipped up in S3 buckets and on tarsnap. Like they say, there's two kinds of people. People who've lost important data because they didn't back it up properly, and people who haven't yet.
  - rovr138 7y ago
    > Through immediate independent investigations, all three companies observed that user accounts were compromised using legitimate credentials including passwords, app passwords, API keys, and personal access tokens. Part of your backup strategy depends on external services. Not necessarily in your case, but people who only have their backups externally on a service could be affected. > all the projects I have ever worked on are on other several machines and/or hard drives And depending on your strategy, since they're so distributed it could mean they're outdated repos. If not, and they pull automatically, they could be affected. Local backups also have issues. The disk might die, the data might be corrupted or any other myriad of things could happen. > People who've lost important data because they didn't back it up properly, and people who haven't yet. Is there such a thing as a perfect backup strategy?
    - bigiain 7y ago
      > Is there such a thing as a perfect backup strategy? At work, there's "Can meet contractually agreed RPO and RTO with 99.99% certainty". Automate the standard setup, and sleep well at night. Perfect. At home, there's "I've done enough that I think the next improvement is an unfeasibly large amount of extra time&money for an unreasonably small improvement". I've, for myself at home, settled on Apple's Time Machine backing up my Macs (and their phone/ipad iTunes backups) to a raid 10 set, that raid 10 set rsynced to another one at the opposite end of the house, and a weekly backup of that stored on a single drive that only powers up for 6 hours every Sunday night then powers back down again - so if my whole network gets breached and cryptolockered (for example) I'll still have at most 7 day old data at home. I also push that weekly backup out to S3 and tarsnap for off-site in-case-my-house-burns-down, or I've set it all on fire and moved to Belize scenarios... I've been running most of that for ~8 years now. I've called it "done", while not "perfect", its certainly good enough against "not-Mossad threat models". If Mossad or The NSA want to delete my backups, so be it - I'll go be a carpenter or a gardener or something.

jedberg 7y ago

I just have to say, props to Gitlab for being included in this. For a lot of enterprises that use Github and Bitbucket, this may be their first into to Gitlab.

Ancient 7y ago

Repo's with remaining ransom file: https://github.com/search?q=1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9DA&type=Code https://github.com/search?q=1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9...

sverhagen 7y ago
Weird, weren't GitHub and friends restoring these repositories for these unlucky users?
- ralph84 7y ago
  I assume they're trying to contact the repo owners to see how they want to proceed. Even if the change seems obvious, changing the contents of a repo is definitely a case where the git host should involve the owner of the repo (barring ToS violations).
  - jredmond 7y ago
    This is the safer bet. The host should offer to restore from their own backups, but some customers may have already taken care of their own stuff.

ralph84 7y ago

2FA is great for the web UI, but none of these vendors make it particularly easy to enforce 2FA on the command line.

nathanaldensr 7y ago
Exactly. PATs are designed to circumvent human intervention (MFA) for authentication in order to support automation. I am very curious if there's a better way than PATs.
- tonny747 7y ago
  ssh keys with a passphrase?
- ekarulf 7y ago
  SSH keys on a YubiKey
deleted 7y ago
[deleted]
nhumrich 7y ago
An ssh key is 2FA
- michaelmior 7y ago
  How so? An SSH key is a single factor. You could argue that a password-protected private key provides a second factor, but that still falls in the category of "something you know."
  - will4274 7y ago
    How many people can recite their SSH key? Surely an SSH key is "something you have".
    - wnevets 7y ago

heelhook 7y ago

Seems like no one fell for this though. https://www.blockchain.com/btc/address/1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9DA https://www.blockchain.com/btc/address/1ES14c7qLb5CYhLMUekct...

avian 7y ago
Probably a unique wallet in each message. It seems to be that way in scam email ransoms.
- kristofferc 7y ago
  Doesn't seem that way: https://github.com/search?q=1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9DA&type=Code https://github.com/search?q=1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9...

penagwin 7y ago

Is it common that companies share intelligence like this? I think it's a wonderful idea, given they all operate on essentially the same service (git) they share similar security concerns.

smudgymcscmudge 7y ago
It took exceptional circumstances, but it was the right call in this case. All were working toward the same goal. We can all go back to our war after the white walkers are dealt with (don’t bring Cersei up or it ruins the analogy)
WrtCdEvrydy 7y ago
In the Cybersecurity field, yes! You'd be surprised how often you'll be rolling up to your competition to compare the virus files you pulled from each of your networks and reverse engineering on VMs together... then next week you have to pretend you hate them until something else goes wrong again.

deleted 7y ago

[deleted]

shapov 7y ago

I didn't see it mentioned in the article, but did any of the 3 companies confirm that the repos have been actually cloned as the attackers suggest?

mlindner 7y ago

1. Stop using 'git add .' This is a bad habit I see people keep suggesting to new git users. Stop recommending it and stop doing it. 2. Never store your password in .git/config. Why are you doing that? That shouldn't be stored in .git/config.

benburleson 7y ago
Why are people using passwords instead of keys?
- morpheuskafka 7y ago
  Why do people even have passwords that don't require 2FA?
- deleted 7y ago
  [deleted]
- jwilk 7y ago
  "Stupid corporate firewall blocks SSH connections" would be my guess.
darklajid 7y ago
Maybe I misunderstood something, but I read the text as people (or .. services like the three doing the announcement) offering URLs containing an access token? As in https://example.com/mlindner/project1/821372asd1786d21das https://example.com/mlindner/project1/821372asd1786d21das or something? If you use this approach to manage access to a repository, then .. that gets stored in the .git/config. No need to store a password or something. (Then again, maybe I didn't understand the explanation correctly?)

falsedan 7y ago

> Otherwise, you can still clone the repository and make use of: git reflog or git fsck to find your last commit and change the HEAD. I don't understand: when I clone a repo, I get a copy of all the branches/tags and the commits they point to & the trees/blobs from those commits. If the repo is wiped, I get a single master branch with a single commit with a single tree and a single blob, and no reflog because that is local to the repo, and I (as a fresh cloner) haven't updated any refs. Perhaps they are thinking about a mirror clone? That still won't include the reflog, but you can at least find dangling commits and guess which one was master.

falsedan 7y ago
Oops, mirror clone only includes reachable commits. No idea what the 'Otherwise' clarification means then, it sounds bogus

chrischen 7y ago

How does one withdraw bitcoin to fiat or even use it without it being traceable? Are there laundering or anonymizing services for bitcoin withdrawals to fiat?

leethargo 7y ago
AFAIK there have been bitcoin laundering services for years (bitcoin in, bitcoin out). As for spending, there used to be pre-loaded credit cards that you could from bitcoin.
- chrischen 7y ago
  Wouldn't the credit cards be traceable (record of where you spent it). It wouldn't necessarily tie you exactly but it'd create a large paper-trail.
Legogris 7y ago
The BTC could be used as payments for services or products which don't require identification (and holding "dirty" BTC is now the problem of the sand celler if they ever want to take it to traditional businesses in jurisdictions that enforce KYC for BTC payments). Criminals can also trade BTC for physical cash. They could also by some means (for example permissionless decentralized exchanges) convert it to a cryptocurrency with private transaction properties such as ZCash, Monero, Beam or GRIN and then back again. The "laundering services" you refer to (generally called "mixers") are still around but most of them have been shown reversible with high degrees of confidence. CoinJoin is the state of the art here, with the most well-known implementations being JoinMarket and Wasabi Wallet. But in general law enforcement and investigators are definitely wising up to cryptocurrencies and to be fully untraceable one has to go through a lot of hoops and not make a single mistake in the process. Even the above mentioned approaches can leak information that can be used to tie an individual to the transactions if not executed properly. Most likely they will use the same tried and true approach they would have used for stolen fiat funds; identity theft. I personally know people who have been drawn into a criminal investigation for money laundering because they had been initiated a transaction selling BTC on LocalBitcoins via bank transfer (unwise unless you know and trust the person), turned out already stolen BTC had been converted into fiat on a compromised bank account, which was then supposed to be converted into "clean" BTC again. Fortunately the investigation was already underway when the transaction happened, my friends bank account was blocked as a result when the transfer was initiated and the whole thing was sorted out in the end.