10 ms·

Data accidentally exposed by Microsoft AI researchers

721 points by deepersprout 3y ago

mola 3y ago
It's always funny that wiz's big security revelations are almost always about Microsoft. When wiz's founder was the highest ranking in charge of cyber security at Microsoft in his previous job .
- alphabetting 3y ago
  Would be kind of surprising if that weren't the case.
h1fra 3y ago
The article is focusing on AI and teams messages for some reason, but the exposed bucket had password, ssh keys, credentials, .env and most probably a lot of proprietary code. I can't even imagine the nightmare it has created internally.
anon1199022 3y ago
Just proves how hard it cloud security now. 1-2 mistake and you expose TB's. Insane.
- photoGrant 3y ago
  Hard coded secrets in shareable URL’s with almost infinite time windows and an untraceable ability to audit what’s made and shared and at what level? Sounds like it’s as hard as it’s always been. Pretty basic and filled with humans
  - LeifCarrotson 3y ago
    I feel like it's harder. It's no longer hierarchical, with organization schemes limited to folders and files. People no longer talk about network paths, or server names. Mobile and desktop apps alike go to enormous effort to abstract and hide the location at which a document gets stored, instead everything is tagged and shared across buckets and accounts and domains... I expect that the people at this organization working on cutting-edge AI are pretty sharp, but it's no surprise that they don't entirely understand the implications of "SAS tokens" and "storage containers" and "permissive access scope" on Azure, and the differences between Account SAS, Service SAS, and User Delegation SAS. Maybe the people at Wiz.io are sharper, but unless I missed the sarcasm, they may be wrong when they say [1] "Generating an Account SAS is a simple process." That looks like a really complicated process! We just traced back an issue where a bunch of information was missing from a previous employee's projects when we changed his account to a shared mailbox. Turns out that he'd inadvertently been saving and sharing documents from his individual OneDrive on O365 (There's not one drive! There are many! Stop trying to pretend there's only one drive!) instead of the "official" organization-level project folder, and had weird settings on his laptop that pointed every "Save" operation at that personal folder, requiring a byzantine procedure to input a real path to get back to the project folder. [1]: https://i.imgur.com/6V7VLLd.png https://i.imgur.com/6V7VLLd.png
    - eitland 3y ago
      > but unless I missed the sarcasm, they may be wrong when they say [1] "Generating an Account SAS is a simple process." That looks like a really complicated process! No, unless I understand actually it is intended to be understood the other way: It is too easy to create a to broad token. And in the next paragraph, after the image, they explain that in addition to it being easy to create, these tokens are impossible to audit.
- tombert 3y ago
  My wife and I just rewatched WarGames for the millionth time a few nights ago. The level of cybersecurity incompetency in the early 80's makes sense; computers (and in particular networked computers) were still relatively new, and there weren't that many external users to begin with, so while the potential impact of a mistake was huge (which of course was the plot of the movie), the likelihood of a horrible thing happening was fairly low just because computers were an expensive, somewhat niche thing. Fast forward to 2023, and now everyone owns bunches of computers, all of which are connected to a network, and all of which are oodles more powerful than anything in the 80s. Cybersecurity protocols are of course much more mature now, but there's also several orders of magnitude more potential attackers than there were in the 80s.
  - viraptor 3y ago
    > Cybersecurity protocols are of course much more mature now At technical level, sure. At the deployment, configuration and management level, not quite. Overall things are so bad that news aren't even reporting the hospitals taken over by ransomware anymore. It's still happening almost every week and we're just... used to it.
  - LinuxBender 3y ago
    That modem setup in Wargames is still a thing for many organizations including some banks and telcos. Not naming names but I suspect the modems will be around for a very long time. Some have a password on their modem but they are usually very simple. Their only saving grace is that they are usually in front of a mainframe speaking proprietary MML that only old fuddy duddies like me would remember. There are a few of us here
    - heywhatupboys 3y ago
      what does this have to do with a "modem" per se?
      - LinuxBender 3y ago
        The parent comment was about the movie Wargames and the questionable security of the 80's that is still in use today. That security in Wargames was a modem that provided access to a subsystem of the WOPR mainframe named "Joshua". Joshua had super-user privs on the mainframe. It was likely meant to be a temporary means for the system architect to monitor and improve the system after it was deployed but then life changing circumstances may have distracted his attention away from decommissioning the modem. The movie still holds up today and is worth a watch. Actually it may be more pertinent now than ever.
buro9 3y ago
Part of me thought "this is fine as very few could actually download 38TB". But that's not true as it's just so cheap to spin up a machine and some storage on a Cloud provider and deal with it later. It's also not true as I've got a 1Gbps internet connection and 112TB usable in my local NAS. All of a sudden (over a decade) all the numbers got big and massive data exfiltration just looks to be trivial. I mean, obviously that's the sales pitch... you need this vendor's monitoring and security, but that's not a bad sales pitch as you need to be able to imagine and think of the risk to monitor for it and most engineers aren't thinking that way.
- byteknight 3y ago
  Trivial in a technical sense but monitoring capabilities (hopefully) have increased in kind.
- koolba 3y ago
  At the rack rates of $.05/GB, that’d come out to $1,945 per copy that’s downloaded. So not only do you have the breach, you also have a fat bill too.
  - redox99 3y ago
    > $.05/GB That's just a scam rate by AWS. The true price is 1/100th of that, if that.
- sillysaurusx 3y ago
  How do you have your NAS configured? The more specifics, the better; I’ve wanted one. Do you worry about failure? In your hardware life I mean, not your personal life.
  - aftbit 3y ago
    Not the OP but I have a pair of Chenbro NR12000 1U rack mount servers, bought for about $120 each on eBay a few years ago. Each has 12 internal 3.5" mounting points and 14 SATA cables. In one server, I have 12 4TB used enterprise drives. In the other, I have 12 8TB drives. Both have 16 GB of RAM (should probably be more) and two 2.5" SATA SSDs. They are configured with two ZFS raidz1 vdevs, each made up of 6 disks. This gives me 10 usable disks and 2 used for parity, and the ability to survive at least one failure but maybe two (if I'm lucky). I back up critical data from the 80TB NAS to the 40TB NAS, and the most critical data gets backed up nightly to a single hard drive in my friend's NAS box (offsite). Twice a year, I back up the full thing to external hard drives and take them out of state to a different friend's house. Don't worry, be happy.
endisneigh 3y ago
how is this sort of stuff not at least encrypted at rest?
- Smaug123 3y ago
  Per the article, the Azure bucket was explicitly shared. Azure Storage is generally encrypted at rest (https://learn.microsoft.com/en-us/azure/storage/common/storage-service-encryption https://learn.microsoft.com/en-us/azure/storage/common/stora...).
- tremon 3y ago
  Encryption at rest does nothing to prevent online access to data. It's only useful if you leave your storage cabinet standing on the side of the road.
  - quickthrower2 3y ago
    Your laptop backup could be encrypted. New problem: where to out the keys. Maybe another storage account with different access controls.
    - pixl97 3y ago
      > New problem: where to out the keys. If it's windows, Active Directory.
- nightpool 3y ago
  What do you think "encryption at rest" means
bkm 3y ago
Would be insane if the GPT4 model is in there somewhere (as its served by Azure).
- albert_e 3y ago
  Also imagine all such exposed data sources including those that are not yet discovered... are crawled and trained on by GPT5. Meanwhile a big enterprise provider like MS suffers a bigger leak and exposes MS Teams/ OneDrive / SharePoint data of all its North America customers say. Boom we have GPT model that can autonomously run whole businesses.
- naillo 3y ago
  Well there is that "transformers" folder at the bottom of the screenshot...
hdesh 3y ago
On a lighter note - I saw a chat message that started with "Hey dude! How is it going". I'm disappointed that the response was not https://nohello.net/en/ https://nohello.net/en/.
- monkpit 3y ago
  I strongly support the “no hello” concept but I also fear being seen as “that guy” so I never mention it. Sigh
  - monkpit 3y ago
    I should have a slack bot that replies automatically to generic greetings… that way they’ll get on with whatever the issue is and I won’t have to reply.
    - bornfreddy 3y ago
      Ha ha, that's a great idea! A: Hello! B's bot: Hello to you too! I am a chatty bot which loves responding to greetings. Is there a message I can forward to B?
  - hiddencost 3y ago
    I make it my status message.
    - version_five 3y ago
      I tried that on slack for a while, it made no difference. I don't think most people read the status message. The medium lends itself to the "Hi" type messages unfortunately, there's not really a way go constrain human nature, other than to not use instant messaging at all (I also tried changing my status to a note telling people to phone me, that didn't work either)
    - deleted 3y ago
      [deleted]
    - 3y ago
munchler 3y ago
> This case is an example of the new risks organizations face when starting to leverage the power of AI more broadly, as more of their engineers now work with massive amounts of training data. It seems like a stretch to associate this risk with AI specifically. The era of "big data" started several years before the current AI boom.
- numbsafari 3y ago
  This is the risk of using, checks notes, Azure and working with Microsoft. Except there is no risk for them. They've proven time and again they have major security snafus and not be held accountable.
  - robertlagrant 3y ago
    They have all the regulatory paperwork in place, so it must be fine.
    - datavirtue 3y ago
      They are also the top line investment for the majority of mutual and pension funds. Don't crab too much, they are funding your retirement.
  - eddythompson80 3y ago
    Virtual networks are a nightmare to setup and manage in Azure which is why everyone just takes the easy path and not bother. Almost every Azure service we deal with has virtual networks as an after thought because they want to get to market as quickly as possible, and even to them managing vnets is a nightmare. Not to excuse developers/users though. There are plenty of unsecured S3 buckets, docker containers, and Github repos that expose too much "because it's easier". I've had a developer checkin their ftp creds into a repo the whole company has access to. He even broke the keys up and concat them in shell to work around the static checks "because it's easier" for their dev/test flow.
- mavhc 3y ago
  With big data comes big responsibility
junglistguy 3y ago
[dead]
bt1a 3y ago
Don't get pickled, friends!
sillysaurusx 3y ago
The article tries to play up the AI angle, but this was a pretty standard misconfiguration of a storage token. This kind of thing happens shockingly often, and it’s why frequent pentests are important.
- sneak 3y ago
  It was so common that S3 added several features to make it really, really hard to accidentally leave a whole bucket public. Looks like Azure hasn't done similarly.
  - mcast 3y ago
    Is there any valid use case for when it's a good idea to publicly expose a S3 bucket?
    - sneak 3y ago
      Sharing of datasets, disk images, ISOs, ML models, etc, as well as public websites.
- cj 3y ago
  > it’s why frequent pentests are important. Unfortunately a lot of pen testing services have devolved into "We know you need a report for SOC 2, but don't worry, we can do some light security testing and generate a report for you in a few days and you'll be able to check the box for compliance" Which is guess is better than nothing. If anyone works at a company that does pen tests for compliance purposes, I'd recommend advocating internally for doing a "quick, easy, and cheap" pen test to "check the box" for compliance, _alongside_ a more comprehensive pen test (maybe call it something other than a "pen test" to convince internal stakeholders who might be afraid that a 2nd in depth pen test might weaken their compliance posture since the report is typically shared with sales prospects) Ideally grey box or white box testing (provide access to codebase / infrastructure to make finding bugs easier). Most pen tests done for compliance purposes are black-box and limit their findings as a result.
stevanl 3y ago
Looks like it was up for 2 years with that old link[1]. Fixed two months ago. [1] https://github.com/microsoft/robust-models-transfer/blame/a9e0e80bcd49bd8651c0b3198c7dc89179b2c0ac/README.md#L37 https://github.com/microsoft/robust-models-transfer/blame/a9...
Nischalj10 3y ago
zsh, any way to download the stuff?
avereveard 3y ago
Oof. Is that containing code from GitHub private repos?
saurik 3y ago
A number of replies here are noting (correctly) how this doesn't have much to do with AI (despite some sentences in this article kind of implicating it; the title doesn't really, fwiw) and is more of an issue with cloud providers, confusing ways in which security tokens apply to data being shared publicly, and dealing with big data downloads (which isn't terribly new)... ...but one notable way in which it does implicate an AI-specific risk is how prevalent it is to use serialized Python objects to store these large opaque AI models, given how the Python serialization format was never exactly intended for untrusted data distribution and so is kind of effectively code... but stored in a way where both what that code says as well as that it is there at all is extremely obfuscated to people who download it. > This is particularly interesting considering the repository’s original purpose: providing AI models for use in training code. The repository instructs users to download a model data file from the SAS link and feed it into a script. The file’s format is ckpt, a format produced by the TensorFlow library. It’s formatted using Python’s pickle formatter, which is prone to arbitrary code execution by design. Meaning, an attacker could have injected malicious code into all the AI models in this storage account, and every user who trusts Microsoft’s GitHub repository would’ve been infected by it.
- hedora 3y ago
  Occasionally, I’ll talk to someone suggesting a dynamically typed language (or stringly-typed java) for a very large scale (in developer count) security or mission critical application. This incident is a good one to point back to.
  - chinchilla2020 3y ago
    What is the best practice? I'm assuming something that isn't a programming language object...
  - sillysaurusx 3y ago
    laughs in log4j vuln A good fraction of the flaws we found at Matasano involved pentests against statically typed languages. If an adversary has root access to your storage box, they can likely find ways to pivot their access. Netpens were designed to do that, and those were the most fun; they’d parachute us into a random network, give us non-root creds, and say “try to find as many other servers that you can get to.” It was hard, but we’d find ways, and it almost never involved modifying existing files. It wasn’t necessary — the bash history always had so many useful points of interest. It’s true that the dynamics are a little different there, since that’s a running server rather than a storage box. But those two employees’ hard drive backups have an almost 100% chance of containing at least one pivot vector. Sadly choice of technology turns out to be irrelevant, and can even lead to overconfidence. The solution is to pay for regular security testing, and not just the automated kind. Get someone in there to try to sleuth out attack vectors by hand. It’s expensive, but it pays off.
rickette 3y ago
At this point MS might as well aquire Wiz, given the number of azure security findings they have found.
mymac 3y ago
Fortunately not a whole of of data and for sure with a little bit like that there wasn't anything important, confidential or embarrassing in there. Looking forward to Microsoft's itemised list of what was taken, as well as their GDPR related filing.
HumblyTossed 3y ago
Microsoft, too big to fa.. care.
formerly_proven 3y ago
This stands out > Our scan shows that this account contained 38TB of additional data — including Microsoft employees’ personal computer backups. Not even Microsoft has functioning corporate IT any more, with employees not just being able to make their own image-based backups, but also having to store them in some random A3 bucket that they're using for work files.
- croes 3y ago
  Why not even? Security was never a strong part of Microsoft.
riwsky 3y ago
If only Microsoft hadn’t named the project “robust” models transfer, they could have dodged this Hubrisbleed attack.
wodenokoto 3y ago
I really dislike how Azure makes you juggle keys in order to make any two Azure things talk together. Even more so, you only have two keys for the entire storage account. Would have made much more sense if you could have unlimited, named keys for each container.
- PretzelPirate 3y ago
  > if you could have unlimited, named keys for each container. These exist and are called Shared Access Tokens. People are too lazy to use them and just use the account-wide keys instead.
- quickthrower2 3y ago
  https://learn.microsoft.com/en-us/azure/role-based-access-control/overview https://learn.microsoft.com/en-us/azure/role-based-access-co...
- unoti 3y ago
  > I really dislike how Azure makes you juggle keys in order to make any two Azure things talk together. Actually there is a better way. Look into “Managed Identity”. This allows you to grant access from one service to another, for example grant access to allow a specific VM to work with your storage account.
  - bob1029 3y ago
    This is what we are using for everything. It makes life so much easier. So far, our new Azure tenant has absolutely zero passwords or shared secrets to keep track of. Granting a function app access to SQL Server by way of the app's name felt like some kind of BS magic trick to me at first. But it absolutely works. Experiences like this give me hope for the future.
quickthrower2 3y ago
Two of the things that make me cringe are mentioned. Pickle files and SAS tokens. I get nervous dealing with Azure storage. Use RBAC. They should depreciate SAS and account keys IMO. SOC2 type auditing should have been done here so I am surprised of the reach. Having the SAS with no expiry and then the deep level of access it gave including machine backups with their own tokens. A lot of lack of defence in depth going on there. My view is burn all secrets. Burn all environment variables. I think most systems can work based on roles. Important humans access via username password and other factors. If you are working in one cloud you don’t in theory need secrets. If not I had the idea the other day that proxies tightly couples to vaults could be used as api adaptors to convert then into RBAC too. But I am not a security expert just paranoid lol.
- ana_winters 3y ago
  [flagged]
- bunderbunder 3y ago
  Pickle files are cringe, but they're also basically unavoidable when working with Python machine learning infrastructure. None of the major ML packages provide a proper model serialization/deserialization mechanism. In the case of scikit-learn, the code implementing some components does so much crazy dynamic shit that it might not even be feasible to provide a well-engineered serde mechanism without a major rewrite. Or at least, that's roughly what the project's maintainers say whenever they close tickets requesting such a thing.
  - jklehm 3y ago
    ONNX[0], model-as-protosbufs, continuing to gain adoption will hopefully solve this issue. [0] https://github.com/onnx/onnx https://github.com/onnx/onnx
    - bunderbunder 3y ago
      ONNX is cool, but it still only supports a minority of scikit-learn components. Some of them simply aren't compatible with ONNX's basic design.
svaha1728 3y ago
Embrace, extend, and extinguish cybersecurity with AI. It's the Microsoft way.
EGreg 3y ago
This seems to be a common occurrence with Big Tech and Big Government, so we better get used to it: https://qbix.com/blog/2023/06/12/no-way-to-prevent-this-says-only-industry-where-this-regularly-happens-2/ https://qbix.com/blog/2023/06/12/no-way-to-prevent-this-says... https://qbix.com/blog/2021/01/25/no-way-to-prevent-this-says-only-industry-where-this-regularly-happens/ https://qbix.com/blog/2021/01/25/no-way-to-prevent-this-says...
- alphabetting 3y ago
  Is this stuff regularly happening to AWS and GCP? This is like the 3rd insane security incident from Microsoft in the past year.
  - EGreg 3y ago
    https://www.bleepingcomputer.com/news/security/top-secret-us-army-and-nsa-files-left-exposed-online-on-amazon-s3-server/ https://www.bleepingcomputer.com/news/security/top-secret-us... https://www.engadget.com/amp/2018-07-18-robocall-exposes-voter-records.html https://www.engadget.com/amp/2018-07-18-robocall-exposes-vot... Ok so it’s not Microsoft exposing Microsoft, but government exposing its S3 buckets. The question should be — why is all that data and power concentrated in one place? Because of the capitalist system and Big Tech, or Big Government. Personally I am rather happy when “top secret information” is exposed, because that I s the type of thing that harms people around the world more than it helps. The government wants to know who is sending you $600 but doesnt want to tell you how they spent trillions on shadowy “defense” contractors. https://community.qbix.com/t/transparency-in-government/234 https://community.qbix.com/t/transparency-in-government/234
anyoneamous 3y ago
Straight to jail.
- 1-6 3y ago
  Nah, Microsoft probably has a blameless culture
  - croes 3y ago
    It was hackers, for sure.
jl6 3y ago
Kind of incredible that someone managed to export Teams messages out from Teams…
naikrovek 3y ago
Amazing how ingrained it is in some people to just go around security controls. someone chose to make that SAS have a long expiry and someone chose to make it read-write.
- JohnMakin 3y ago
  It’s easy. “ugh, this thing needs to get out by end of week and I can’t scope this key properly, nothing’s working with it.” “just give it admin privileges and we’ll fix it later” sometimes they’ll put a short TTL on it, aware of the risk. Then something major breaks a few months later, gets a 15 year expiry, never is remediated. It’s common because it’s tempting and easy to tell yourself you’ll fix it later, refactor, etc. But then people leave, stuff gets dropped, and security is very rarely a priority in most orgs - let alone remediation of old security issues.
EMCymatics 3y ago
That's a lot of data.
gumballindie 3y ago
Would be cool if someone analysed - i am fairly certain it has proprietary code and data laying around. Would be useful for future lawsuits against microsoft and others that steal people’s ip for “training” purposes.
34679 3y ago
@4mm character width: 4e-6 * 3.8e+13 = 152 million kilometers of text. Nearly 200 round trips to the moon.
madelyn-goodman 3y ago
This is so unfortunate but a clear illustration of something I've been thinking about a lot when it comes to LLMs and AI. It seems like we're forgetting that we are just handing our data over to these companies on a solver platter in the form of our prompts. Disclosure that I do work for Tonic.ai and we are working on a way to automatically redact any information you send to an LLM - https://www.tonic.ai/solar https://www.tonic.ai/solar
Jamesmoorez 3y ago
[dead]
baz00 3y ago
What's that, the second major data loss / leak event from MSFT recently. Is your data really safe there?
pradn 3y ago
It's not reasonable to expect human security token generation to be perfectly secure all the time. The system needs to be safe overall. The organization should have set an OrgPolicy on this entire project to prevent blanket sharing of auth tokens/credentials like this. Ideally blanket access tokens should be opt-in, not opt-out. Google banned generation of service account keys for internally-used projects. So an awry JSON file doesn't allow access to Google data/code. This is enforced at the highest level by OrgPolicy. There's a bunch more restrictions, too.
davidf18 3y ago
[dead]
lijok 3y ago
I wouldn't trust MSFT with my glass of chocolate milk at this point. I would come back to lipstick all over the rim and somehow multiple leaks in the glass
kevinsundar 3y ago
This is very similar to how some security researchers got access to TikTok's S3 bucket: https://medium.com/berkeleyischool/cloudsquatting-taking-over-abandoned-cloud-storage-locations-8466290fb8de https://medium.com/berkeleyischool/cloudsquatting-taking-ove... They used the same mechanism of using common crawl or other publicly available web crawler data to source dns records for s3 buckets.
fithisux 3y ago
My opinion is that it was not an "accident", but they prepare us for the era where powerful companies will "own" our data in the name of security. Should have been sent to prison.

721 points by deepersprout 3y ago

mola 3y ago

It's always funny that wiz's big security revelations are almost always about Microsoft. When wiz's founder was the highest ranking in charge of cyber security at Microsoft in his previous job .

alphabetting 3y ago
Would be kind of surprising if that weren't the case.

h1fra 3y ago

The article is focusing on AI and teams messages for some reason, but the exposed bucket had password, ssh keys, credentials, .env and most probably a lot of proprietary code. I can't even imagine the nightmare it has created internally.

anon1199022 3y ago

Just proves how hard it cloud security now. 1-2 mistake and you expose TB's. Insane.

photoGrant 3y ago
Hard coded secrets in shareable URL’s with almost infinite time windows and an untraceable ability to audit what’s made and shared and at what level? Sounds like it’s as hard as it’s always been. Pretty basic and filled with humans
- LeifCarrotson 3y ago
  I feel like it's harder. It's no longer hierarchical, with organization schemes limited to folders and files. People no longer talk about network paths, or server names. Mobile and desktop apps alike go to enormous effort to abstract and hide the location at which a document gets stored, instead everything is tagged and shared across buckets and accounts and domains... I expect that the people at this organization working on cutting-edge AI are pretty sharp, but it's no surprise that they don't entirely understand the implications of "SAS tokens" and "storage containers" and "permissive access scope" on Azure, and the differences between Account SAS, Service SAS, and User Delegation SAS. Maybe the people at Wiz.io are sharper, but unless I missed the sarcasm, they may be wrong when they say [1] "Generating an Account SAS is a simple process." That looks like a really complicated process! We just traced back an issue where a bunch of information was missing from a previous employee's projects when we changed his account to a shared mailbox. Turns out that he'd inadvertently been saving and sharing documents from his individual OneDrive on O365 (There's not one drive! There are many! Stop trying to pretend there's only one drive!) instead of the "official" organization-level project folder, and had weird settings on his laptop that pointed every "Save" operation at that personal folder, requiring a byzantine procedure to input a real path to get back to the project folder. [1]: https://i.imgur.com/6V7VLLd.png https://i.imgur.com/6V7VLLd.png
  - eitland 3y ago
    > but unless I missed the sarcasm, they may be wrong when they say [1] "Generating an Account SAS is a simple process." That looks like a really complicated process! No, unless I understand actually it is intended to be understood the other way: It is too easy to create a to broad token. And in the next paragraph, after the image, they explain that in addition to it being easy to create, these tokens are impossible to audit.
tombert 3y ago
My wife and I just rewatched WarGames for the millionth time a few nights ago. The level of cybersecurity incompetency in the early 80's makes sense; computers (and in particular networked computers) were still relatively new, and there weren't that many external users to begin with, so while the potential impact of a mistake was huge (which of course was the plot of the movie), the likelihood of a horrible thing happening was fairly low just because computers were an expensive, somewhat niche thing. Fast forward to 2023, and now everyone owns bunches of computers, all of which are connected to a network, and all of which are oodles more powerful than anything in the 80s. Cybersecurity protocols are of course much more mature now, but there's also several orders of magnitude more potential attackers than there were in the 80s.
- viraptor 3y ago
  > Cybersecurity protocols are of course much more mature now At technical level, sure. At the deployment, configuration and management level, not quite. Overall things are so bad that news aren't even reporting the hospitals taken over by ransomware anymore. It's still happening almost every week and we're just... used to it.
- LinuxBender 3y ago
  That modem setup in Wargames is still a thing for many organizations including some banks and telcos. Not naming names but I suspect the modems will be around for a very long time. Some have a password on their modem but they are usually very simple. Their only saving grace is that they are usually in front of a mainframe speaking proprietary MML that only old fuddy duddies like me would remember. There are a few of us here
  - heywhatupboys 3y ago
    what does this have to do with a "modem" per se?
    - LinuxBender 3y ago
      The parent comment was about the movie Wargames and the questionable security of the 80's that is still in use today. That security in Wargames was a modem that provided access to a subsystem of the WOPR mainframe named "Joshua". Joshua had super-user privs on the mainframe. It was likely meant to be a temporary means for the system architect to monitor and improve the system after it was deployed but then life changing circumstances may have distracted his attention away from decommissioning the modem. The movie still holds up today and is worth a watch. Actually it may be more pertinent now than ever.

buro9 3y ago

Part of me thought "this is fine as very few could actually download 38TB". But that's not true as it's just so cheap to spin up a machine and some storage on a Cloud provider and deal with it later. It's also not true as I've got a 1Gbps internet connection and 112TB usable in my local NAS. All of a sudden (over a decade) all the numbers got big and massive data exfiltration just looks to be trivial. I mean, obviously that's the sales pitch... you need this vendor's monitoring and security, but that's not a bad sales pitch as you need to be able to imagine and think of the risk to monitor for it and most engineers aren't thinking that way.

byteknight 3y ago
Trivial in a technical sense but monitoring capabilities (hopefully) have increased in kind.
koolba 3y ago
At the rack rates of $.05/GB, that’d come out to $1,945 per copy that’s downloaded. So not only do you have the breach, you also have a fat bill too.
- redox99 3y ago
  > $.05/GB That's just a scam rate by AWS. The true price is 1/100th of that, if that.
sillysaurusx 3y ago
How do you have your NAS configured? The more specifics, the better; I’ve wanted one. Do you worry about failure? In your hardware life I mean, not your personal life.
- aftbit 3y ago
  Not the OP but I have a pair of Chenbro NR12000 1U rack mount servers, bought for about $120 each on eBay a few years ago. Each has 12 internal 3.5" mounting points and 14 SATA cables. In one server, I have 12 4TB used enterprise drives. In the other, I have 12 8TB drives. Both have 16 GB of RAM (should probably be more) and two 2.5" SATA SSDs. They are configured with two ZFS raidz1 vdevs, each made up of 6 disks. This gives me 10 usable disks and 2 used for parity, and the ability to survive at least one failure but maybe two (if I'm lucky). I back up critical data from the 80TB NAS to the 40TB NAS, and the most critical data gets backed up nightly to a single hard drive in my friend's NAS box (offsite). Twice a year, I back up the full thing to external hard drives and take them out of state to a different friend's house. Don't worry, be happy.

endisneigh 3y ago

how is this sort of stuff not at least encrypted at rest?

Smaug123 3y ago
Per the article, the Azure bucket was explicitly shared. Azure Storage is generally encrypted at rest (https://learn.microsoft.com/en-us/azure/storage/common/storage-service-encryption https://learn.microsoft.com/en-us/azure/storage/common/stora...).
tremon 3y ago
Encryption at rest does nothing to prevent online access to data. It's only useful if you leave your storage cabinet standing on the side of the road.
- quickthrower2 3y ago
  Your laptop backup could be encrypted. New problem: where to out the keys. Maybe another storage account with different access controls.
  - pixl97 3y ago
    > New problem: where to out the keys. If it's windows, Active Directory.
nightpool 3y ago
What do you think "encryption at rest" means

bkm 3y ago

Would be insane if the GPT4 model is in there somewhere (as its served by Azure).

albert_e 3y ago
Also imagine all such exposed data sources including those that are not yet discovered... are crawled and trained on by GPT5. Meanwhile a big enterprise provider like MS suffers a bigger leak and exposes MS Teams/ OneDrive / SharePoint data of all its North America customers say. Boom we have GPT model that can autonomously run whole businesses.
naillo 3y ago
Well there is that "transformers" folder at the bottom of the screenshot...

hdesh 3y ago

On a lighter note - I saw a chat message that started with "Hey dude! How is it going". I'm disappointed that the response was not https://nohello.net/en/ https://nohello.net/en/.

monkpit 3y ago
I strongly support the “no hello” concept but I also fear being seen as “that guy” so I never mention it. Sigh
- monkpit 3y ago
  I should have a slack bot that replies automatically to generic greetings… that way they’ll get on with whatever the issue is and I won’t have to reply.
  - bornfreddy 3y ago
    Ha ha, that's a great idea! A: Hello! B's bot: Hello to you too! I am a chatty bot which loves responding to greetings. Is there a message I can forward to B?
- hiddencost 3y ago
  I make it my status message.
  - version_five 3y ago
    I tried that on slack for a while, it made no difference. I don't think most people read the status message. The medium lends itself to the "Hi" type messages unfortunately, there's not really a way go constrain human nature, other than to not use instant messaging at all (I also tried changing my status to a note telling people to phone me, that didn't work either)
  - deleted 3y ago
    [deleted]
  - 3y ago

munchler 3y ago

> This case is an example of the new risks organizations face when starting to leverage the power of AI more broadly, as more of their engineers now work with massive amounts of training data. It seems like a stretch to associate this risk with AI specifically. The era of "big data" started several years before the current AI boom.

numbsafari 3y ago
This is the risk of using, checks notes, Azure and working with Microsoft. Except there is no risk for them. They've proven time and again they have major security snafus and not be held accountable.
- robertlagrant 3y ago
  They have all the regulatory paperwork in place, so it must be fine.
  - datavirtue 3y ago
    They are also the top line investment for the majority of mutual and pension funds. Don't crab too much, they are funding your retirement.
- eddythompson80 3y ago
  Virtual networks are a nightmare to setup and manage in Azure which is why everyone just takes the easy path and not bother. Almost every Azure service we deal with has virtual networks as an after thought because they want to get to market as quickly as possible, and even to them managing vnets is a nightmare. Not to excuse developers/users though. There are plenty of unsecured S3 buckets, docker containers, and Github repos that expose too much "because it's easier". I've had a developer checkin their ftp creds into a repo the whole company has access to. He even broke the keys up and concat them in shell to work around the static checks "because it's easier" for their dev/test flow.
mavhc 3y ago
With big data comes big responsibility

junglistguy 3y ago

[dead]

bt1a 3y ago

Don't get pickled, friends!

sillysaurusx 3y ago

The article tries to play up the AI angle, but this was a pretty standard misconfiguration of a storage token. This kind of thing happens shockingly often, and it’s why frequent pentests are important.

sneak 3y ago
It was so common that S3 added several features to make it really, really hard to accidentally leave a whole bucket public. Looks like Azure hasn't done similarly.
- mcast 3y ago
  Is there any valid use case for when it's a good idea to publicly expose a S3 bucket?
  - sneak 3y ago
    Sharing of datasets, disk images, ISOs, ML models, etc, as well as public websites.
cj 3y ago
> it’s why frequent pentests are important. Unfortunately a lot of pen testing services have devolved into "We know you need a report for SOC 2, but don't worry, we can do some light security testing and generate a report for you in a few days and you'll be able to check the box for compliance" Which is guess is better than nothing. If anyone works at a company that does pen tests for compliance purposes, I'd recommend advocating internally for doing a "quick, easy, and cheap" pen test to "check the box" for compliance, _alongside_ a more comprehensive pen test (maybe call it something other than a "pen test" to convince internal stakeholders who might be afraid that a 2nd in depth pen test might weaken their compliance posture since the report is typically shared with sales prospects) Ideally grey box or white box testing (provide access to codebase / infrastructure to make finding bugs easier). Most pen tests done for compliance purposes are black-box and limit their findings as a result.

stevanl 3y ago

Looks like it was up for 2 years with that old link[1]. Fixed two months ago. [1] https://github.com/microsoft/robust-models-transfer/blame/a9e0e80bcd49bd8651c0b3198c7dc89179b2c0ac/README.md#L37 https://github.com/microsoft/robust-models-transfer/blame/a9...

Nischalj10 3y ago

zsh, any way to download the stuff?

avereveard 3y ago

Oof. Is that containing code from GitHub private repos?

saurik 3y ago

A number of replies here are noting (correctly) how this doesn't have much to do with AI (despite some sentences in this article kind of implicating it; the title doesn't really, fwiw) and is more of an issue with cloud providers, confusing ways in which security tokens apply to data being shared publicly, and dealing with big data downloads (which isn't terribly new)... ...but one notable way in which it does implicate an AI-specific risk is how prevalent it is to use serialized Python objects to store these large opaque AI models, given how the Python serialization format was never exactly intended for untrusted data distribution and so is kind of effectively code... but stored in a way where both what that code says as well as that it is there at all is extremely obfuscated to people who download it. > This is particularly interesting considering the repository’s original purpose: providing AI models for use in training code. The repository instructs users to download a model data file from the SAS link and feed it into a script. The file’s format is ckpt, a format produced by the TensorFlow library. It’s formatted using Python’s pickle formatter, which is prone to arbitrary code execution by design. Meaning, an attacker could have injected malicious code into all the AI models in this storage account, and every user who trusts Microsoft’s GitHub repository would’ve been infected by it.

hedora 3y ago
Occasionally, I’ll talk to someone suggesting a dynamically typed language (or stringly-typed java) for a very large scale (in developer count) security or mission critical application. This incident is a good one to point back to.
- chinchilla2020 3y ago
  What is the best practice? I'm assuming something that isn't a programming language object...
- sillysaurusx 3y ago
  laughs in log4j vuln A good fraction of the flaws we found at Matasano involved pentests against statically typed languages. If an adversary has root access to your storage box, they can likely find ways to pivot their access. Netpens were designed to do that, and those were the most fun; they’d parachute us into a random network, give us non-root creds, and say “try to find as many other servers that you can get to.” It was hard, but we’d find ways, and it almost never involved modifying existing files. It wasn’t necessary — the bash history always had so many useful points of interest. It’s true that the dynamics are a little different there, since that’s a running server rather than a storage box. But those two employees’ hard drive backups have an almost 100% chance of containing at least one pivot vector. Sadly choice of technology turns out to be irrelevant, and can even lead to overconfidence. The solution is to pay for regular security testing, and not just the automated kind. Get someone in there to try to sleuth out attack vectors by hand. It’s expensive, but it pays off.

rickette 3y ago

At this point MS might as well aquire Wiz, given the number of azure security findings they have found.

mymac 3y ago

Fortunately not a whole of of data and for sure with a little bit like that there wasn't anything important, confidential or embarrassing in there. Looking forward to Microsoft's itemised list of what was taken, as well as their GDPR related filing.

HumblyTossed 3y ago

Microsoft, too big to fa.. care.

formerly_proven 3y ago

This stands out > Our scan shows that this account contained 38TB of additional data — including Microsoft employees’ personal computer backups. Not even Microsoft has functioning corporate IT any more, with employees not just being able to make their own image-based backups, but also having to store them in some random A3 bucket that they're using for work files.

croes 3y ago
Why not even? Security was never a strong part of Microsoft.

riwsky 3y ago

If only Microsoft hadn’t named the project “robust” models transfer, they could have dodged this Hubrisbleed attack.

wodenokoto 3y ago

I really dislike how Azure makes you juggle keys in order to make any two Azure things talk together. Even more so, you only have two keys for the entire storage account. Would have made much more sense if you could have unlimited, named keys for each container.

PretzelPirate 3y ago
> if you could have unlimited, named keys for each container. These exist and are called Shared Access Tokens. People are too lazy to use them and just use the account-wide keys instead.
quickthrower2 3y ago
https://learn.microsoft.com/en-us/azure/role-based-access-control/overview https://learn.microsoft.com/en-us/azure/role-based-access-co...
unoti 3y ago
> I really dislike how Azure makes you juggle keys in order to make any two Azure things talk together. Actually there is a better way. Look into “Managed Identity”. This allows you to grant access from one service to another, for example grant access to allow a specific VM to work with your storage account.
- bob1029 3y ago
  This is what we are using for everything. It makes life so much easier. So far, our new Azure tenant has absolutely zero passwords or shared secrets to keep track of. Granting a function app access to SQL Server by way of the app's name felt like some kind of BS magic trick to me at first. But it absolutely works. Experiences like this give me hope for the future.

quickthrower2 3y ago

Two of the things that make me cringe are mentioned. Pickle files and SAS tokens. I get nervous dealing with Azure storage. Use RBAC. They should depreciate SAS and account keys IMO. SOC2 type auditing should have been done here so I am surprised of the reach. Having the SAS with no expiry and then the deep level of access it gave including machine backups with their own tokens. A lot of lack of defence in depth going on there. My view is burn all secrets. Burn all environment variables. I think most systems can work based on roles. Important humans access via username password and other factors. If you are working in one cloud you don’t in theory need secrets. If not I had the idea the other day that proxies tightly couples to vaults could be used as api adaptors to convert then into RBAC too. But I am not a security expert just paranoid lol.

ana_winters 3y ago
[flagged]
bunderbunder 3y ago
Pickle files are cringe, but they're also basically unavoidable when working with Python machine learning infrastructure. None of the major ML packages provide a proper model serialization/deserialization mechanism. In the case of scikit-learn, the code implementing some components does so much crazy dynamic shit that it might not even be feasible to provide a well-engineered serde mechanism without a major rewrite. Or at least, that's roughly what the project's maintainers say whenever they close tickets requesting such a thing.
- jklehm 3y ago
  ONNX[0], model-as-protosbufs, continuing to gain adoption will hopefully solve this issue. [0] https://github.com/onnx/onnx https://github.com/onnx/onnx
  - bunderbunder 3y ago
    ONNX is cool, but it still only supports a minority of scikit-learn components. Some of them simply aren't compatible with ONNX's basic design.

svaha1728 3y ago

Embrace, extend, and extinguish cybersecurity with AI. It's the Microsoft way.

EGreg 3y ago

This seems to be a common occurrence with Big Tech and Big Government, so we better get used to it: https://qbix.com/blog/2023/06/12/no-way-to-prevent-this-says-only-industry-where-this-regularly-happens-2/ https://qbix.com/blog/2023/06/12/no-way-to-prevent-this-says... https://qbix.com/blog/2021/01/25/no-way-to-prevent-this-says-only-industry-where-this-regularly-happens/ https://qbix.com/blog/2021/01/25/no-way-to-prevent-this-says...

alphabetting 3y ago
Is this stuff regularly happening to AWS and GCP? This is like the 3rd insane security incident from Microsoft in the past year.
- EGreg 3y ago
  https://www.bleepingcomputer.com/news/security/top-secret-us-army-and-nsa-files-left-exposed-online-on-amazon-s3-server/ https://www.bleepingcomputer.com/news/security/top-secret-us... https://www.engadget.com/amp/2018-07-18-robocall-exposes-voter-records.html https://www.engadget.com/amp/2018-07-18-robocall-exposes-vot... Ok so it’s not Microsoft exposing Microsoft, but government exposing its S3 buckets. The question should be — why is all that data and power concentrated in one place? Because of the capitalist system and Big Tech, or Big Government. Personally I am rather happy when “top secret information” is exposed, because that I s the type of thing that harms people around the world more than it helps. The government wants to know who is sending you $600 but doesnt want to tell you how they spent trillions on shadowy “defense” contractors. https://community.qbix.com/t/transparency-in-government/234 https://community.qbix.com/t/transparency-in-government/234

anyoneamous 3y ago

Straight to jail.

1-6 3y ago
Nah, Microsoft probably has a blameless culture
- croes 3y ago
  It was hackers, for sure.

jl6 3y ago

Kind of incredible that someone managed to export Teams messages out from Teams…

naikrovek 3y ago

Amazing how ingrained it is in some people to just go around security controls. someone chose to make that SAS have a long expiry and someone chose to make it read-write.

JohnMakin 3y ago
It’s easy. “ugh, this thing needs to get out by end of week and I can’t scope this key properly, nothing’s working with it.” “just give it admin privileges and we’ll fix it later” sometimes they’ll put a short TTL on it, aware of the risk. Then something major breaks a few months later, gets a 15 year expiry, never is remediated. It’s common because it’s tempting and easy to tell yourself you’ll fix it later, refactor, etc. But then people leave, stuff gets dropped, and security is very rarely a priority in most orgs - let alone remediation of old security issues.

EMCymatics 3y ago

That's a lot of data.

gumballindie 3y ago

Would be cool if someone analysed - i am fairly certain it has proprietary code and data laying around. Would be useful for future lawsuits against microsoft and others that steal people’s ip for “training” purposes.

34679 3y ago

@4mm character width: 4e-6 * 3.8e+13 = 152 million kilometers of text. Nearly 200 round trips to the moon.

madelyn-goodman 3y ago

This is so unfortunate but a clear illustration of something I've been thinking about a lot when it comes to LLMs and AI. It seems like we're forgetting that we are just handing our data over to these companies on a solver platter in the form of our prompts. Disclosure that I do work for Tonic.ai and we are working on a way to automatically redact any information you send to an LLM - https://www.tonic.ai/solar https://www.tonic.ai/solar

Jamesmoorez 3y ago

[dead]

baz00 3y ago

What's that, the second major data loss / leak event from MSFT recently. Is your data really safe there?

pradn 3y ago

It's not reasonable to expect human security token generation to be perfectly secure all the time. The system needs to be safe overall. The organization should have set an OrgPolicy on this entire project to prevent blanket sharing of auth tokens/credentials like this. Ideally blanket access tokens should be opt-in, not opt-out. Google banned generation of service account keys for internally-used projects. So an awry JSON file doesn't allow access to Google data/code. This is enforced at the highest level by OrgPolicy. There's a bunch more restrictions, too.

davidf18 3y ago

[dead]

lijok 3y ago

I wouldn't trust MSFT with my glass of chocolate milk at this point. I would come back to lipstick all over the rim and somehow multiple leaks in the glass

kevinsundar 3y ago

This is very similar to how some security researchers got access to TikTok's S3 bucket: https://medium.com/berkeleyischool/cloudsquatting-taking-over-abandoned-cloud-storage-locations-8466290fb8de https://medium.com/berkeleyischool/cloudsquatting-taking-ove... They used the same mechanism of using common crawl or other publicly available web crawler data to source dns records for s3 buckets.

fithisux 3y ago

My opinion is that it was not an "accident", but they prepare us for the era where powerful companies will "own" our data in the name of security. Should have been sent to prison.