11 ms·

Rubygems.org AWS Root Access Event – September 2025

280 points by ilikepi 11mo ago

xaxaxa123 11mo ago
interesting timing, isnt it?!
voidfunc 11mo ago
This is an absolute disaster for Ruby and the integrity of the Ruby ecosystem... I dont know if you can build a product with Ruby beyond this without basically having your own highly restricted gem cache.
- deleted 11mo ago
  [deleted]
xer0x 11mo ago
The Rubygems take over drama continues!
- mylons 11mo ago
  [flagged]
eutropia 11mo ago
They buried the lede... Arko wanted a copy of the HTTP Access logs from rubygems.org so his consultancy could monetize the data, after RC determined they didn't really have the budget for secondary on-call. Then after they removed him as a maintainer he logged in and changed the AWS root password.
- deleted 11mo ago
  [deleted]
- JeremyNT 11mo ago
  What a truly wild situation. In a certain sense this post justifies why RC wanted so badly to take ownership - I mean, here you have a maintainer who clearly has a desire to sell user data to make a buck - but the way it all played out with terrible communication and rookie mistakes on revoking access undermines faith in RC's ability to secure the service going forward. Not to mention no explanation here of who legally "owned" the rubygems repo (not just the infra) and why they thought they had the right to claim it, which is something disputed by the "other" side. Just a mess all around, nobody comes off looking very good here!
  - anon84873628 11mo ago
    I can give benefit of the doubt that making a proposal to monetize user data is a poorly-considered, bottom-scraping effort to find a replacement funding source for the on call work. Most of us would not consider it, but I think it should be ok to occasionally pitch some bad ideas, all else being equal and lacking full context. But messing with the credentials crosses an ethical line that isn't excused no matter how much you disagree with the other party's actions.
    - JeremyNT 11mo ago
      I totally agree, assuming all this is accurate he immediately proved that RC was right all along to be concerned about him!
    - fouc 11mo ago
      I can only assume it is silly revenge seeking behavior. Look at how symmetrical it is: 1. RC takes over GitHub Repository and locks everyone out 2. Arko takes over RubyGems server and locks everyone out. He was an authorized actor right up until they tried to remove him, but they forgot to revoke his access credentials. I wonder if legally-speaking he was even considered unauthorized. EDIT: Missed their email notification revoking his production access. Yeah looks like they could have a legal basis.
      - heartbreak 11mo ago
- picadi 11mo ago
  really disappointing. it's such a huge security concern and privacy/ethical lapse, i am super disappointed in him, despite his contributions to the world of Ruby package management he's now started a competing gem.coop package manager, and while they haven't released a privacy policy it does make me suspicious about how they were planning to fund it no single person should have Github owner + AWS root password for a major language's package manager and ecosystem just sitting around on their laptop while they fly around to different conferences in Japan e.g. (as Andre did while hacking rubygem's AWS root account to show off)
- case 11mo ago
  [citation needed]
  - evolve2k 11mo ago
    From the article: “Following these budget adjustments, Mr. Arko’s consultancy, which had been receiving approximately $50,000 per year for providing the secondary on-call service, submitted a proposal offering to provide secondary on-call services at no cost in exchange for access to production HTTP access logs, containing IP addresses and other personally identifiable information (PII).”
- deleted 11mo ago
  [deleted]
terracatta 11mo ago
That email screenshot is pretty bad for Arko. It clearly shows intent to sell PII data to a third party during a time when Ruby Central had diminished funds and needed help affording basic services. What the fuck.
- mikey_p 11mo ago
  Why do they need money? What happened to their funding?
  - dismalaf 11mo ago
    It was after a big sponsor pulled out and presumably before Shopify stepped in...
  - krowek 11mo ago
    One of the major sources of funding was cut because they sided with the devil...
    - baggy_trough 11mo ago
      The dastardly devil who made the whole thing popular in the first place! Quite a devil indeed.
      - ioasuncvinvaer 11mo ago
        Yes? What's your point?
        baggy_trough 11mo ago
        To make fun of the idea that DHH is a devil.
  - fareesh 11mo ago
    they were funded by a company run by a guy who read DHH's blog and cut funding unless they excommunicated him
andrewguenther 11mo ago
In 2025 there's no reason for anyone to be logging into an AWS account via the root credentials and this should have been addressed in the preventative measures. There's no actual control improvements here, just "we'll follow our procedures better next time" which imo is effectively doing nothing. Also this is really lacking in detail about how it was determined that no PII was accessed. What audit logs were checked? Where was this data stored? Overall this is a super disappointing postmortem...
- nerdjon 11mo ago
  > In 2025 there's no reason for anyone to be logging into an AWS account via the root credentials and this should have been addressed in the preventative measures. I am curious what preventative measures you expect in this situation? To my knowledge it is not actually possible to disable the root account. They also had it restricted to only 3 people with MFA which also seems pretty reasonable. It is not unheard of that there could be a situation where your ability to login through normal means (like lets say it relies on Okta and Okta goes down) and you need to get into the account, root may be your only option in a disaster situation. Given this was specifically for oncall someone having that makes sense. Not saying there were not failures because there clearly are, but there have been times I have had to use root when I had no other option to get into an account.
  - flumpcakes 11mo ago
    I'm now questioning my sanity but I thought you could disable login for the root account in AWS.
    - the_mitsuhiko 11mo ago
      Since there are certain operations that can only be done with the root account, there is no way to disable access to it.
      - jhealy 11mo ago
        Since 2024 you can disable the root credentials on all accounts except the Organization management account: https://aws.amazon.com/blogs/aws/centrally-managing-root-access-for-customers-using-aws-organizations/ https://aws.amazon.com/blogs/aws/centrally-managing-root-acc... I don't think the post mortem details whether the root access was on the org management account or an org member account.
codegeek 11mo ago
"The root account credentials, essentially the highest level of administrative control, are stored in a shared enterprise password manager in a shared vault to which only three individuals had access: two current Ruby Central staff members and one former maintainer, André Arko" I am wondering. Did they at least have MFA enabled on the root login or not ?
- terracatta 11mo ago
  Yes because they state under the section "Root Cause Analysis" > Ruby Central failed to rotate the AWS root account credentials (password and MFA) after the departure of personnel with access to the shared vault.
  - sersi 11mo ago
    If both password and MFA are stored in the same shared vault then MFA's purpose is compromised. Anyone getting access to that shared vault has the full keys to the kingdom the same as if MFA wasn't enabled. Also in this day and age, there's no reason to have the root account creds in a shared vault, no-one should ever need to access the root account, everyone should have IAM accounts with only the necessary permissions.
    - deleted 11mo ago
      [deleted]
    - blibble 11mo ago
      > If both password and MFA are stored in the same shared vault then MFA's purpose is compromised. Anyone getting access to that shared vault has the full keys to the kingdom the same as if MFA wasn't enabled. absolutely > no-one should ever need to access the root account someone has to be able to access it (rarely) if you're a micro-org having three people with the ability to get it doesn't seem that bad everything else they did is however terrible practice
- mikey_p 11mo ago
  So they failed to properly protect their credentials? This sure doesn't reflect all this supposed professionalism and improvements RC was supposed to make. Years ago, I decided with all the DHH drama, that using Rails was too much of a liability and this shit just makes the whole Ruby ecosystem a liability to anything build in that ecosystem.
deleted 11mo ago
[deleted]
andrewmcwatters 11mo ago
Ethical and legal boundaries? RubyGems Privacy Notice already tells you that they share information with a number of large firms and notably ClickHouse... for "Customer Data Processing." All this proposal does is request from one of the maintainers/on-call providers? another entry in this Privacy Notice as a part of a payment deal. This is a mess, but it also unnecessary smears both sides. It calls out that RubyCentral had poor cloud management in place, and it trashes an on-call provider. This is a terrible postmortem and all it does is advertise to users that RubyCentral doesn't know what it's doing.
- RhythmFox 11mo ago
  [flagged]
  - jaredcwhite 11mo ago
    I have no doubt a certain contingent who tend to flock on a certain former bird-themed network are happily spreading their votes around.
  - andrewmcwatters 11mo ago
    I think the point being that whatever was going on with access controls here was blatant gross negligence, because what is this guy doing with access who doesn't literally own the organization or the intellectual property? Set that aside, which obviously stinks, but then why is said obviously incompetent organization sharing confidential corporate emails with the public, saying this guy proposing a corporate data access plan in exchange for consulting fees is crossing "ethical and legal boundaries?" Are you serious? This reeks of someone or a number of people who have no idea what they're doing. Have these people never worked with a business before? Never spoken with a software firm? A marketing firm? Any consultancy whatsoever? Who owns the registry? What is going on here? This is insane. It makes the entire Ruby ecosystem look like it's run by children.
    - RhythmFox 11mo ago
      If only the Ruby ecosystem was really run by children, that sounds way more for than whatever this is. Joking aside though I think you are very right to question the professionalism behind this post. The narrative spin here is so toxic to good faith arguments around how we could change the 'supply chain' of Gems for the better. Wasn't that what this was all about Ruby Central? Can we get an update on that instead of this attack on your former admin?
ttfvjktesd 11mo ago
> failed to rotate the AWS root account credentials ... stored in a shared enterprise password manager Unfortunately, many enterprises follow the poor practice of storing shared credentials in a shared password manager without rotating them when an employee with prior access leaves the company.
- kjs3 11mo ago
  You might be surprised/horrified at the number of shops I run into that use shared creds without a password manager, still use creds from ex-employees because changing them smells too much like work, and ask "why would I do that?" when you ask about rotation.
phoronixrly 11mo ago
So we have DHH with his unhinged posts on one side, and Arko wanting to sell PII on the other. Great! I think we need an f-droid-like project for Rubygems that builds the gems from source, and takes care of signing, and is backed by a non-profit that is independent from Rails/Shopify
- dismalaf 11mo ago
  Gem can pull in gems from any repository, even straight from a git server like GitHub. And most of the time gems are built from scratch on your computer, Nokogiri is the only one I can think of that isn't.
  - zrail 11mo ago
    The problem, as with every package manager, is transitive dependencies. It's all well and good to set up direct dependencies to only pull from git repositories, but bundler still needs a way to resolve those gems' dependencies. You could pre-resolve every dependency in your chain to a git repository, even to a fork under your own control, but that will end up being a maintenance nightmare.
    - Imustaskforhelp 11mo ago
      Can't a middle compromise happen as it happens in something like golang? Can some vps/serverless provider not do this like fly.io as an recent example with kurt got got? or hetzner? I think that golang's model can actually be sort of cheaper/ more cost effective for servers as compared to how ruby might be doing it right now and so cheaper might mean that a new non profit can be created which can work on less money/outside funding/drama overall
      - zrail 11mo ago
        Retrofitting Go's dependency model into Ruby is not trivial. Go has used full URLs for dependencies from the jump, making a central package repository irrelevant. Ruby doesn't have that. At best you might have a source code URL in the gem source that you can access from a gem server, but that doesn't really anything. Someone still has to provide the index.
brunosutic 11mo ago
Does anyone know if Arko can have legal troubles (like being sued) if it can be proven he "removed authorized users" from RubyCentral AWS account?
- dismalaf 11mo ago
  RubyCentral is a legal entity and they paid him for his work so they definitely could take action against him if he was shown to have harmed users or the company in any way.
  - heartbreak 11mo ago
    Legal recourse is an interesting omission from their update, especially considering this was surely approved by their attorney(s).
- hokumguru 11mo ago
  https://www.justice.gov/jm/jm-9-48000-computer-fraud https://www.justice.gov/jm/jm-9-48000-computer-fraud at least a misdemeanor. most of the time its prosecuted, a felony.
hokumguru 11mo ago
Even if Arko wasn't the unauthorized access, this absolutely tarnishes his reputation for the rest of his career, damning email notwithstanding. Horrible time to own/run a consultancy. Can't imagine what his other customers are thinking right now.
dismalaf 11mo ago
Lol and in previous posts everyone was acting like Arko is above board... I brought up multiple times that his actions were suspicious, was downvoted. Now proof of that plus an email trying to low-key extort RubyCentral into allowing him to sell user data...
- ameliaquining 11mo ago
  It's not extortion, because refusing his offer would have left them no worse off than if he hadn't gotten involved in the first place. You can still object on other grounds, of course.
busterarm 11mo ago
Props to Ruby Central for taking all of the smears and reputational damage on the chin silently while they mitigated an actual security incident, made absolutely sure and wrote up a proper post-mortem. All of that in line with their original statement that their actions taken were in the interest of security/integrity of their platform. If there's any evidence that you need to know who the proper stewards of Ruby's gems are, it's this.
- software_writer 11mo ago
  100%
- deleted 11mo ago
  [deleted]
- xaxaxa123 11mo ago
  odd timing for such an incident..
  - busterarm 11mo ago
    Not really. Someone abusing their access after being removed is exactly when such events occur. Your post suggests conspiratorial thinking when there shouldn't be.
    - xaxaxa123 11mo ago
      my post suggests critical thinking when there should be..your post is just defamatory. all of this looks like a "problem, action, solution" scheme. what was the takeover then, a honest and transparent move by RC? you must be kidding. looked pretty much conspiratorial.
- Mystery-Machine 11mo ago
  An entity that promised security had a security incident due of their incompetence to properly secure their production environment root access?
deleted 11mo ago
[deleted]
ctoth 11mo ago
AWS account root access on a language package registry for 11 days. Not EC2 root - AWS account root. Complete control over IAM, S3, CloudTrail, every-damn-thing. They're claiming "no evidence of compromise" based on CloudTrail logs that AWS root could have deleted or modified. They even admit they "Enabled AWS CloudTrail" after regaining control - meaning CloudTrail wasn't running during the compromise window. You cannot verify supply chain integrity from logs on a system where root was compromised, and you definitely can't verify it when the logs didn't exist (they enabled them during remediation?). So basically, somebody correct me here if I'm wrong but ... Every gem published Sept 19-30 is suspect. Production Ruby applications running code from that window have no way to verify it wasn't backdoored. The correct response is to freeze publishing, rebuild from scratch (including re-publishing any packages published at the time? Ugh I don't even know how to do this! ) , and verify against offline backups. Instead they rotated passwords and called it done.
- yargzeblog 11mo ago
  IMO the only way to avoid doing a total rebuild is to have Andre Arko: 1. Admit that he was the unauthorized actor (which means he's probably admitting to a crime?) 2. Have him attest he didn't exfil or modify the integrity of service while committing a crime. If I was Ruby Central I would give clemency on #1 in exchange for #2 and I think #2 helps Andre Arko.
  - ctoth 11mo ago
    So you would expect people to accept that the entire root chain of custody for the Ruby supply chain is attested by ... A guy saying he didn't do anything bad? I have a cool cryptocurrency you might wanna check out that I definitely don't have a backdoor to!
  - LamaOfRuin 11mo ago
    If Andre doing that was criminal, it seems quite possible that their original takeover of the github organization was also criminal? I have been waiting to hear if there would be any civil action on it since it's not at all clear they had any rights to do most of what they did.
deng 11mo ago
If they really have ethical concerns regarding sharing data with third parties, maybe they should update their privacy policies accordingly? "We collect information related to web traffic such as IP addresses and geolocation data for security-relevant events and to analyze how and where RubyGems.org is used." (https://rubygems.org/policies/privacy https://rubygems.org/policies/privacy) "We may share aggregate or de-identified information with third parties for research, marketing, analytics, and other purposes, provided such information does not identify a particular individual." (https://rubycentral.org/privacy-notice/ https://rubycentral.org/privacy-notice/)
- deleted 11mo ago
  [deleted]
its-summertime 11mo ago
They do seem to have found the perfect scapegoat to point everyone towards to deflect from all the other issues with their actions
- RhythmFox 11mo ago
  Not only that but they get to use an individual who they have philosophical differences with. You can say it was 'good security practice', tarnish his reputation, and get to switch the narrative to something sympathetic to yourself all in one go. Very convenient for them. I think they make a lot of overly strong claims here, even though there are plenty of alternative explanations possible. The mere fact that 3 people had AWS root access during this period but they only identify one and never question that it could have been one of the others is telling. They reallllly want you to just take it as obvious that 1) all these actions were taken by 1 individual and 2) that individual was malicious. Then they sprinkle in enough nasty sounding activities and info about Andre to get you to draw the conclusion that he is bad, and did bad things, and they had to do these things the way they did. Using what reads like a business strategy email as a 'nefarious backstory' is so bad faith. I bet if you got access to all the board's emails you would see a ton of proposals for ways to support RubyGems that may not all sound great in isolation. They are being just transparent enough to bad mouth Andre while hiding any motivations from their end as purely 'security' related.
jmuguy 11mo ago
Well this certainly clarifies a lot. I'm not sure I'm confident that they know nothing was compromised further or the extent of any data exfil, it sounds like a lot of logging was enabled after the incident occurred.
jnewland 11mo ago
This is a pretty hilarious and long-winded way to say "we have no idea how to lock someone out of a web service:" > 1. While Ruby Central correctly removed access to shared credentials through its enterprise password manager prior to the incident, our staff did not consider the possibility that this credential may have been copied or exfiltrated to other password managers outside of Ruby Central’s visibility or control. > 2. Ruby Central failed to rotate the AWS root account credentials (password and MFA) after the departure of personnel with access to the shared vault.
- TehCorwiz 11mo ago
  Right?! Did nobody there think to actually disable the accounts? These are the people who are harping about "security" being the reason for the ham-fisted takeover of the source repos, but they didn't secure the production infrastructure?
- colonwqbang 11mo ago
  It didn't occur to them that he might have written the password down? That's wild.
  - TehCorwiz 11mo ago
    Either profoundly naive, tech illiterate, or it's a bad faith argument.
  - baobun 11mo ago
    Or more realistically, accessed the accounts via IAM token and/or service account. Something they also failed to consider, reading between the lines.
  - jeffwask 11mo ago
    No matter how you slice it this is miserable root password security. Why do maintainers need root access? No one in my org has root access but me and all those creds are tied to hardware MFA locked in my MDF.
deleted 11mo ago
[deleted]
shevy-java 11mo ago
I'd recommend to people to wait for a response - RubyCentral spins up a gazillion accusations right now and has been in the last days (and, it is also incomplete, because why did they fire every dev here and placed Marty Haught in charge specifically? They never were able to logically explain this; plus, why didn't they release this write-up before? It feels very strange to wait here; they could have clarified things before, but to me it seems they kind of waited and then tried to come up with some explanation that, to me, makes no real sense). I also highly recommend to not accept RubyCentral's current strategy to post very isolated emails and insinuate that "this is the ultimate, final proof". We all know that email conversation often requires lots of emails. So doing a piecemail release really feels strange. Plus, there also were in-person meetings - why does RubyCentral not release what was discussed here? Was there a conflict of interest due to financial pressure? Also, as was already pointed out, RubyCentral went lawyering up already - see discussions on reddit. Is this really the transparency we as users and developers want to see? This is blowing up by the day and no matter from which side you want to look at it, RubyCentral sits at the center; or, at the very least, made numerous mistakes, tries to cover past mistakes by ... making more mistakes. I think it would be better to dissolve RubyCentral. Let's start from a clean state here; let's find rules of engagement that doesn't put rich corporations atop the whole ecosystem. Last but not least - this tactical slandering is really annoying. If they have factual evidence, they need to bring the matter to a court; if they don't, they need to stop slandering people. To my knowledge RubyCentral hasn't yet started a court case, and I have a slight suspicious that they also will not, because we, as the general public, would then demand COMPLETE transparency, including ALL of RubyCentral's members and their activities here. So my recommendation is: wait for a while, let those accused respond.
- dismalaf 11mo ago
  > let those accused respond. Literally all we've heard so far is from the other side... > If they have factual evidence, they need to bring the matter to a court I'd be surprised if they aren't. This post feels very much like the amount of disclosure a lawyer would recommend to reassure stakeholders. > rules of engagement that doesn't put rich corporations atop the whole ecosystem Right now the only thing stopping us all from being held hostage by rogue maintainers is a rich corporation.
rgreeko42 11mo ago
So is this a smear of Arko (and by extension Ruby Gems' sloppy security) but dressed up like a Security disclosure? If I'm reading it right, it seems quite petty (and a bit cowardly). Arko was a maintainer was he not? How is that a breach? Presumably his credentials were not misbegotten, or is that the accusation?
- fatbird 11mo ago
  After Arko's direct access was revoked, Arko retained access via possession of the root password (which RC should have rotated at the same time). Arko then changed the root password, locking RC out of their AWS account, waited a couple weeks, and then Joel Drapper blogged about the situation with proof that the now-fired Arko controlled the account, in order to make RC look bad.
- picadi 11mo ago
  one assumes he copied the AWS root password out of the RC-provided enterprise password manager / vault onto his own personally controlled password manager before he was locked out, which might be forgivable if it wasn't the root login for a major language's package registry
- PapaPalpatine 11mo ago
  It’s only a smear if all of the public comments thus far against Shopify and Ruby Central are also smears.
deleted 11mo ago
[deleted]
nomdep 11mo ago
> “Following these budget adjustments, Mr. Arko’s consultancy, (…), submitted a proposal offering to provide secondary on-call services at no cost in exchange for access to production HTTP access logs, containing IP addresses and other personally identifiable information (PII). The offer would have given Mr. Arko’s consultancy access to that data, so that they could monetize it by analyzing access patterns and potentially sharing it with unrelated third-parties.” WTF. This is the same guy that is launched gems.coop, a competing index for Ruby gems recently. On the other hand, RubyCentral actions were truly incompetent, I don’t know anymore who is worse
- heartbreak 11mo ago
  Anyone who remembers how Ruby Together was managed knows who is worse.
tptacek 11mo ago
Presuming, as a group full of security peers kibitzing about this in a chat right now all do, that the "unauthorized actor" here is Andre Arko, this is Ruby Central pretty directly accusing Arko of having hacked Rubygems.org; it depicts what seems to be a black letter 18 USC 1030 violation. Any part of this narrative could be false, but I don't see a way to read it and take it as true where Arko's actions would be OK.
- saltypal 11mo ago
  Putting myself in Arko’s shoes, I can imagine (charitably!) the following choice, realizing that I still have access and shouldn’t: 1. Try to get in touch, quickly, with someone with the power to fix it and explain what needs to be rotated. 2. Absent 1, especially if it cannot be done quickly, rotate the credentials personally to get them back to a controlled state (by someone who actually understands the security implications) with the intent to hand them off. Especially if you still _think_ of yourself as responsible for the infrastructure, this is a no-brainer compared to letting anyone else who might be in the same “should have lost access but didn’t, due to negligence” maintain access. Not a legal defense, but let’s not be too hasty to judge.
  - saltypal 11mo ago
    I hadn't yet seen it when I wrote this, but 2 is pretty much exactly what Arko says: > Worried about the possibility of hacked accounts or some sort of social engineering, I took action as the primary on-call engineer to lock down the AWS account and prevent any actions by possible attackers. https://andre.arko.net/2025/10/09/the-rubygems-security-incident/ https://andre.arko.net/2025/10/09/the-rubygems-security-inci...
skywhopper 11mo ago
The most notable thing about this post is the great lengths to which they are willing to go to detail every step of this process, something they never came close to doing in the fallout from the original stripping of rights from the longtime volunteers.
fijiaarone 11mo ago
At least it’s not npm. Or cargo.
raggi 11mo ago
Honestly I think everyone involved here, and I know most of them, needs to step back and grow up. Despite the formal tone, and the potential legal issues at play here, this situation seems to be descending into needless childishness with potential life damaging side effects. All sides are being disrespectful.
fareesh 11mo ago
I've read both sides of the story and it does not make sense why he would change the root password and not communicate this with RubyCentral or make any attempt to contact them about the change. That in itself suggests malice.
- neverrroot 11mo ago
  This, cutting through the noise, this is it.
notpushkin 11mo ago
Update from Andre Arko: https://andre.arko.net/2025/10/09/the-rubygems-security-incident/ https://andre.arko.net/2025/10/09/the-rubygems-security-inci... Discussion: https://news.ycombinator.com/item?id=45535149 https://news.ycombinator.com/item?id=45535149 Not a good look for Ruby Central IMO.
- ycombinatrix 11mo ago
  Yikes.
shivenigma 11mo ago
RC seems to be incompetent and malicious, the original hostile take over is explained as a security measure. But in reality, you can simply just take away the ability to deploy and leave access to contribute to the repo untouched for people outside of the RC organization. But claiming that you care about security while missing a basic step of rotating credentials once a member has moved away is pathetic. God help Rubygems.org and the users.

280 points by ilikepi 11mo ago

interesting timing, isnt it?!

This is an absolute disaster for Ruby and the integrity of the Ruby ecosystem... I dont know if you can build a product with Ruby beyond this without basically having your own highly restricted gem cache.

deleted 11mo ago
[deleted]

xer0x 11mo ago

The Rubygems take over drama continues!

mylons 11mo ago
[flagged]

eutropia 11mo ago

They buried the lede... Arko wanted a copy of the HTTP Access logs from rubygems.org so his consultancy could monetize the data, after RC determined they didn't really have the budget for secondary on-call. Then after they removed him as a maintainer he logged in and changed the AWS root password.

deleted 11mo ago
[deleted]
JeremyNT 11mo ago
What a truly wild situation. In a certain sense this post justifies why RC wanted so badly to take ownership - I mean, here you have a maintainer who clearly has a desire to sell user data to make a buck - but the way it all played out with terrible communication and rookie mistakes on revoking access undermines faith in RC's ability to secure the service going forward. Not to mention no explanation here of who legally "owned" the rubygems repo (not just the infra) and why they thought they had the right to claim it, which is something disputed by the "other" side. Just a mess all around, nobody comes off looking very good here!
- anon84873628 11mo ago
  I can give benefit of the doubt that making a proposal to monetize user data is a poorly-considered, bottom-scraping effort to find a replacement funding source for the on call work. Most of us would not consider it, but I think it should be ok to occasionally pitch some bad ideas, all else being equal and lacking full context. But messing with the credentials crosses an ethical line that isn't excused no matter how much you disagree with the other party's actions.
  - JeremyNT 11mo ago
    I totally agree, assuming all this is accurate he immediately proved that RC was right all along to be concerned about him!
  - fouc 11mo ago
    I can only assume it is silly revenge seeking behavior. Look at how symmetrical it is: 1. RC takes over GitHub Repository and locks everyone out 2. Arko takes over RubyGems server and locks everyone out. He was an authorized actor right up until they tried to remove him, but they forgot to revoke his access credentials. I wonder if legally-speaking he was even considered unauthorized. EDIT: Missed their email notification revoking his production access. Yeah looks like they could have a legal basis.
    - heartbreak 11mo ago
picadi 11mo ago
really disappointing. it's such a huge security concern and privacy/ethical lapse, i am super disappointed in him, despite his contributions to the world of Ruby package management he's now started a competing gem.coop package manager, and while they haven't released a privacy policy it does make me suspicious about how they were planning to fund it no single person should have Github owner + AWS root password for a major language's package manager and ecosystem just sitting around on their laptop while they fly around to different conferences in Japan e.g. (as Andre did while hacking rubygem's AWS root account to show off)
case 11mo ago
[citation needed]
- evolve2k 11mo ago
  From the article: “Following these budget adjustments, Mr. Arko’s consultancy, which had been receiving approximately $50,000 per year for providing the secondary on-call service, submitted a proposal offering to provide secondary on-call services at no cost in exchange for access to production HTTP access logs, containing IP addresses and other personally identifiable information (PII).”
deleted 11mo ago
[deleted]

terracatta 11mo ago

That email screenshot is pretty bad for Arko. It clearly shows intent to sell PII data to a third party during a time when Ruby Central had diminished funds and needed help affording basic services. What the fuck.

mikey_p 11mo ago
Why do they need money? What happened to their funding?
- dismalaf 11mo ago
  It was after a big sponsor pulled out and presumably before Shopify stepped in...
- krowek 11mo ago
  One of the major sources of funding was cut because they sided with the devil...
  - baggy_trough 11mo ago
    The dastardly devil who made the whole thing popular in the first place! Quite a devil indeed.
    - ioasuncvinvaer 11mo ago
      Yes? What's your point?
      - baggy_trough 11mo ago
        To make fun of the idea that DHH is a devil.
- fareesh 11mo ago
  they were funded by a company run by a guy who read DHH's blog and cut funding unless they excommunicated him

andrewguenther 11mo ago

In 2025 there's no reason for anyone to be logging into an AWS account via the root credentials and this should have been addressed in the preventative measures. There's no actual control improvements here, just "we'll follow our procedures better next time" which imo is effectively doing nothing. Also this is really lacking in detail about how it was determined that no PII was accessed. What audit logs were checked? Where was this data stored? Overall this is a super disappointing postmortem...

nerdjon 11mo ago
> In 2025 there's no reason for anyone to be logging into an AWS account via the root credentials and this should have been addressed in the preventative measures. I am curious what preventative measures you expect in this situation? To my knowledge it is not actually possible to disable the root account. They also had it restricted to only 3 people with MFA which also seems pretty reasonable. It is not unheard of that there could be a situation where your ability to login through normal means (like lets say it relies on Okta and Okta goes down) and you need to get into the account, root may be your only option in a disaster situation. Given this was specifically for oncall someone having that makes sense. Not saying there were not failures because there clearly are, but there have been times I have had to use root when I had no other option to get into an account.
- flumpcakes 11mo ago
  I'm now questioning my sanity but I thought you could disable login for the root account in AWS.
  - the_mitsuhiko 11mo ago
    Since there are certain operations that can only be done with the root account, there is no way to disable access to it.
    - jhealy 11mo ago
      Since 2024 you can disable the root credentials on all accounts except the Organization management account: https://aws.amazon.com/blogs/aws/centrally-managing-root-access-for-customers-using-aws-organizations/ https://aws.amazon.com/blogs/aws/centrally-managing-root-acc... I don't think the post mortem details whether the root access was on the org management account or an org member account.

codegeek 11mo ago

"The root account credentials, essentially the highest level of administrative control, are stored in a shared enterprise password manager in a shared vault to which only three individuals had access: two current Ruby Central staff members and one former maintainer, André Arko" I am wondering. Did they at least have MFA enabled on the root login or not ?

terracatta 11mo ago
Yes because they state under the section "Root Cause Analysis" > Ruby Central failed to rotate the AWS root account credentials (password and MFA) after the departure of personnel with access to the shared vault.
- sersi 11mo ago
  If both password and MFA are stored in the same shared vault then MFA's purpose is compromised. Anyone getting access to that shared vault has the full keys to the kingdom the same as if MFA wasn't enabled. Also in this day and age, there's no reason to have the root account creds in a shared vault, no-one should ever need to access the root account, everyone should have IAM accounts with only the necessary permissions.
  - deleted 11mo ago
    [deleted]
  - blibble 11mo ago
    > If both password and MFA are stored in the same shared vault then MFA's purpose is compromised. Anyone getting access to that shared vault has the full keys to the kingdom the same as if MFA wasn't enabled. absolutely > no-one should ever need to access the root account someone has to be able to access it (rarely) if you're a micro-org having three people with the ability to get it doesn't seem that bad everything else they did is however terrible practice
mikey_p 11mo ago
So they failed to properly protect their credentials? This sure doesn't reflect all this supposed professionalism and improvements RC was supposed to make. Years ago, I decided with all the DHH drama, that using Rails was too much of a liability and this shit just makes the whole Ruby ecosystem a liability to anything build in that ecosystem.

deleted 11mo ago

[deleted]

andrewmcwatters 11mo ago

Ethical and legal boundaries? RubyGems Privacy Notice already tells you that they share information with a number of large firms and notably ClickHouse... for "Customer Data Processing." All this proposal does is request from one of the maintainers/on-call providers? another entry in this Privacy Notice as a part of a payment deal. This is a mess, but it also unnecessary smears both sides. It calls out that RubyCentral had poor cloud management in place, and it trashes an on-call provider. This is a terrible postmortem and all it does is advertise to users that RubyCentral doesn't know what it's doing.

RhythmFox 11mo ago
[flagged]
- jaredcwhite 11mo ago
  I have no doubt a certain contingent who tend to flock on a certain former bird-themed network are happily spreading their votes around.
- andrewmcwatters 11mo ago
  I think the point being that whatever was going on with access controls here was blatant gross negligence, because what is this guy doing with access who doesn't literally own the organization or the intellectual property? Set that aside, which obviously stinks, but then why is said obviously incompetent organization sharing confidential corporate emails with the public, saying this guy proposing a corporate data access plan in exchange for consulting fees is crossing "ethical and legal boundaries?" Are you serious? This reeks of someone or a number of people who have no idea what they're doing. Have these people never worked with a business before? Never spoken with a software firm? A marketing firm? Any consultancy whatsoever? Who owns the registry? What is going on here? This is insane. It makes the entire Ruby ecosystem look like it's run by children.
  - RhythmFox 11mo ago
    If only the Ruby ecosystem was really run by children, that sounds way more for than whatever this is. Joking aside though I think you are very right to question the professionalism behind this post. The narrative spin here is so toxic to good faith arguments around how we could change the 'supply chain' of Gems for the better. Wasn't that what this was all about Ruby Central? Can we get an update on that instead of this attack on your former admin?

ttfvjktesd 11mo ago

> failed to rotate the AWS root account credentials ... stored in a shared enterprise password manager Unfortunately, many enterprises follow the poor practice of storing shared credentials in a shared password manager without rotating them when an employee with prior access leaves the company.

kjs3 11mo ago
You might be surprised/horrified at the number of shops I run into that use shared creds without a password manager, still use creds from ex-employees because changing them smells too much like work, and ask "why would I do that?" when you ask about rotation.

phoronixrly 11mo ago

So we have DHH with his unhinged posts on one side, and Arko wanting to sell PII on the other. Great! I think we need an f-droid-like project for Rubygems that builds the gems from source, and takes care of signing, and is backed by a non-profit that is independent from Rails/Shopify

dismalaf 11mo ago
Gem can pull in gems from any repository, even straight from a git server like GitHub. And most of the time gems are built from scratch on your computer, Nokogiri is the only one I can think of that isn't.
- zrail 11mo ago
  The problem, as with every package manager, is transitive dependencies. It's all well and good to set up direct dependencies to only pull from git repositories, but bundler still needs a way to resolve those gems' dependencies. You could pre-resolve every dependency in your chain to a git repository, even to a fork under your own control, but that will end up being a maintenance nightmare.
  - Imustaskforhelp 11mo ago
    Can't a middle compromise happen as it happens in something like golang? Can some vps/serverless provider not do this like fly.io as an recent example with kurt got got? or hetzner? I think that golang's model can actually be sort of cheaper/ more cost effective for servers as compared to how ruby might be doing it right now and so cheaper might mean that a new non profit can be created which can work on less money/outside funding/drama overall
    - zrail 11mo ago
      Retrofitting Go's dependency model into Ruby is not trivial. Go has used full URLs for dependencies from the jump, making a central package repository irrelevant. Ruby doesn't have that. At best you might have a source code URL in the gem source that you can access from a gem server, but that doesn't really anything. Someone still has to provide the index.

brunosutic 11mo ago

Does anyone know if Arko can have legal troubles (like being sued) if it can be proven he "removed authorized users" from RubyCentral AWS account?

dismalaf 11mo ago
RubyCentral is a legal entity and they paid him for his work so they definitely could take action against him if he was shown to have harmed users or the company in any way.
- heartbreak 11mo ago
  Legal recourse is an interesting omission from their update, especially considering this was surely approved by their attorney(s).
hokumguru 11mo ago
https://www.justice.gov/jm/jm-9-48000-computer-fraud https://www.justice.gov/jm/jm-9-48000-computer-fraud at least a misdemeanor. most of the time its prosecuted, a felony.

hokumguru 11mo ago

Even if Arko wasn't the unauthorized access, this absolutely tarnishes his reputation for the rest of his career, damning email notwithstanding. Horrible time to own/run a consultancy. Can't imagine what his other customers are thinking right now.

dismalaf 11mo ago

Lol and in previous posts everyone was acting like Arko is above board... I brought up multiple times that his actions were suspicious, was downvoted. Now proof of that plus an email trying to low-key extort RubyCentral into allowing him to sell user data...

ameliaquining 11mo ago
It's not extortion, because refusing his offer would have left them no worse off than if he hadn't gotten involved in the first place. You can still object on other grounds, of course.

busterarm 11mo ago

Props to Ruby Central for taking all of the smears and reputational damage on the chin silently while they mitigated an actual security incident, made absolutely sure and wrote up a proper post-mortem. All of that in line with their original statement that their actions taken were in the interest of security/integrity of their platform. If there's any evidence that you need to know who the proper stewards of Ruby's gems are, it's this.

software_writer 11mo ago
100%
deleted 11mo ago
[deleted]
xaxaxa123 11mo ago
odd timing for such an incident..
- busterarm 11mo ago
  Not really. Someone abusing their access after being removed is exactly when such events occur. Your post suggests conspiratorial thinking when there shouldn't be.
  - xaxaxa123 11mo ago
    my post suggests critical thinking when there should be..your post is just defamatory. all of this looks like a "problem, action, solution" scheme. what was the takeover then, a honest and transparent move by RC? you must be kidding. looked pretty much conspiratorial.
Mystery-Machine 11mo ago
An entity that promised security had a security incident due of their incompetence to properly secure their production environment root access?

deleted 11mo ago

[deleted]

ctoth 11mo ago

AWS account root access on a language package registry for 11 days. Not EC2 root - AWS account root. Complete control over IAM, S3, CloudTrail, every-damn-thing. They're claiming "no evidence of compromise" based on CloudTrail logs that AWS root could have deleted or modified. They even admit they "Enabled AWS CloudTrail" after regaining control - meaning CloudTrail wasn't running during the compromise window. You cannot verify supply chain integrity from logs on a system where root was compromised, and you definitely can't verify it when the logs didn't exist (they enabled them during remediation?). So basically, somebody correct me here if I'm wrong but ... Every gem published Sept 19-30 is suspect. Production Ruby applications running code from that window have no way to verify it wasn't backdoored. The correct response is to freeze publishing, rebuild from scratch (including re-publishing any packages published at the time? Ugh I don't even know how to do this! ) , and verify against offline backups. Instead they rotated passwords and called it done.

yargzeblog 11mo ago
IMO the only way to avoid doing a total rebuild is to have Andre Arko: 1. Admit that he was the unauthorized actor (which means he's probably admitting to a crime?) 2. Have him attest he didn't exfil or modify the integrity of service while committing a crime. If I was Ruby Central I would give clemency on #1 in exchange for #2 and I think #2 helps Andre Arko.
- ctoth 11mo ago
  So you would expect people to accept that the entire root chain of custody for the Ruby supply chain is attested by ... A guy saying he didn't do anything bad? I have a cool cryptocurrency you might wanna check out that I definitely don't have a backdoor to!
- LamaOfRuin 11mo ago
  If Andre doing that was criminal, it seems quite possible that their original takeover of the github organization was also criminal? I have been waiting to hear if there would be any civil action on it since it's not at all clear they had any rights to do most of what they did.

deng 11mo ago

If they really have ethical concerns regarding sharing data with third parties, maybe they should update their privacy policies accordingly? "We collect information related to web traffic such as IP addresses and geolocation data for security-relevant events and to analyze how and where RubyGems.org is used." (https://rubygems.org/policies/privacy https://rubygems.org/policies/privacy) "We may share aggregate or de-identified information with third parties for research, marketing, analytics, and other purposes, provided such information does not identify a particular individual." (https://rubycentral.org/privacy-notice/ https://rubycentral.org/privacy-notice/)

deleted 11mo ago
[deleted]

its-summertime 11mo ago

They do seem to have found the perfect scapegoat to point everyone towards to deflect from all the other issues with their actions

RhythmFox 11mo ago
Not only that but they get to use an individual who they have philosophical differences with. You can say it was 'good security practice', tarnish his reputation, and get to switch the narrative to something sympathetic to yourself all in one go. Very convenient for them. I think they make a lot of overly strong claims here, even though there are plenty of alternative explanations possible. The mere fact that 3 people had AWS root access during this period but they only identify one and never question that it could have been one of the others is telling. They reallllly want you to just take it as obvious that 1) all these actions were taken by 1 individual and 2) that individual was malicious. Then they sprinkle in enough nasty sounding activities and info about Andre to get you to draw the conclusion that he is bad, and did bad things, and they had to do these things the way they did. Using what reads like a business strategy email as a 'nefarious backstory' is so bad faith. I bet if you got access to all the board's emails you would see a ton of proposals for ways to support RubyGems that may not all sound great in isolation. They are being just transparent enough to bad mouth Andre while hiding any motivations from their end as purely 'security' related.

jmuguy 11mo ago

Well this certainly clarifies a lot. I'm not sure I'm confident that they know nothing was compromised further or the extent of any data exfil, it sounds like a lot of logging was enabled after the incident occurred.

jnewland 11mo ago

This is a pretty hilarious and long-winded way to say "we have no idea how to lock someone out of a web service:" > 1. While Ruby Central correctly removed access to shared credentials through its enterprise password manager prior to the incident, our staff did not consider the possibility that this credential may have been copied or exfiltrated to other password managers outside of Ruby Central’s visibility or control. > 2. Ruby Central failed to rotate the AWS root account credentials (password and MFA) after the departure of personnel with access to the shared vault.

TehCorwiz 11mo ago
Right?! Did nobody there think to actually disable the accounts? These are the people who are harping about "security" being the reason for the ham-fisted takeover of the source repos, but they didn't secure the production infrastructure?
colonwqbang 11mo ago
It didn't occur to them that he might have written the password down? That's wild.
- TehCorwiz 11mo ago
  Either profoundly naive, tech illiterate, or it's a bad faith argument.
- baobun 11mo ago
  Or more realistically, accessed the accounts via IAM token and/or service account. Something they also failed to consider, reading between the lines.
- jeffwask 11mo ago
  No matter how you slice it this is miserable root password security. Why do maintainers need root access? No one in my org has root access but me and all those creds are tied to hardware MFA locked in my MDF.

deleted 11mo ago

[deleted]

shevy-java 11mo ago

I'd recommend to people to wait for a response - RubyCentral spins up a gazillion accusations right now and has been in the last days (and, it is also incomplete, because why did they fire every dev here and placed Marty Haught in charge specifically? They never were able to logically explain this; plus, why didn't they release this write-up before? It feels very strange to wait here; they could have clarified things before, but to me it seems they kind of waited and then tried to come up with some explanation that, to me, makes no real sense). I also highly recommend to not accept RubyCentral's current strategy to post very isolated emails and insinuate that "this is the ultimate, final proof". We all know that email conversation often requires lots of emails. So doing a piecemail release really feels strange. Plus, there also were in-person meetings - why does RubyCentral not release what was discussed here? Was there a conflict of interest due to financial pressure? Also, as was already pointed out, RubyCentral went lawyering up already - see discussions on reddit. Is this really the transparency we as users and developers want to see? This is blowing up by the day and no matter from which side you want to look at it, RubyCentral sits at the center; or, at the very least, made numerous mistakes, tries to cover past mistakes by ... making more mistakes. I think it would be better to dissolve RubyCentral. Let's start from a clean state here; let's find rules of engagement that doesn't put rich corporations atop the whole ecosystem. Last but not least - this tactical slandering is really annoying. If they have factual evidence, they need to bring the matter to a court; if they don't, they need to stop slandering people. To my knowledge RubyCentral hasn't yet started a court case, and I have a slight suspicious that they also will not, because we, as the general public, would then demand COMPLETE transparency, including ALL of RubyCentral's members and their activities here. So my recommendation is: wait for a while, let those accused respond.

dismalaf 11mo ago
> let those accused respond. Literally all we've heard so far is from the other side... > If they have factual evidence, they need to bring the matter to a court I'd be surprised if they aren't. This post feels very much like the amount of disclosure a lawyer would recommend to reassure stakeholders. > rules of engagement that doesn't put rich corporations atop the whole ecosystem Right now the only thing stopping us all from being held hostage by rogue maintainers is a rich corporation.

rgreeko42 11mo ago

So is this a smear of Arko (and by extension Ruby Gems' sloppy security) but dressed up like a Security disclosure? If I'm reading it right, it seems quite petty (and a bit cowardly). Arko was a maintainer was he not? How is that a breach? Presumably his credentials were not misbegotten, or is that the accusation?

fatbird 11mo ago
After Arko's direct access was revoked, Arko retained access via possession of the root password (which RC should have rotated at the same time). Arko then changed the root password, locking RC out of their AWS account, waited a couple weeks, and then Joel Drapper blogged about the situation with proof that the now-fired Arko controlled the account, in order to make RC look bad.
picadi 11mo ago
one assumes he copied the AWS root password out of the RC-provided enterprise password manager / vault onto his own personally controlled password manager before he was locked out, which might be forgivable if it wasn't the root login for a major language's package registry
PapaPalpatine 11mo ago
It’s only a smear if all of the public comments thus far against Shopify and Ruby Central are also smears.

deleted 11mo ago

[deleted]

nomdep 11mo ago

> “Following these budget adjustments, Mr. Arko’s consultancy, (…), submitted a proposal offering to provide secondary on-call services at no cost in exchange for access to production HTTP access logs, containing IP addresses and other personally identifiable information (PII). The offer would have given Mr. Arko’s consultancy access to that data, so that they could monetize it by analyzing access patterns and potentially sharing it with unrelated third-parties.” WTF. This is the same guy that is launched gems.coop, a competing index for Ruby gems recently. On the other hand, RubyCentral actions were truly incompetent, I don’t know anymore who is worse

heartbreak 11mo ago
Anyone who remembers how Ruby Together was managed knows who is worse.

tptacek 11mo ago

Presuming, as a group full of security peers kibitzing about this in a chat right now all do, that the "unauthorized actor" here is Andre Arko, this is Ruby Central pretty directly accusing Arko of having hacked Rubygems.org; it depicts what seems to be a black letter 18 USC 1030 violation. Any part of this narrative could be false, but I don't see a way to read it and take it as true where Arko's actions would be OK.

saltypal 11mo ago
Putting myself in Arko’s shoes, I can imagine (charitably!) the following choice, realizing that I still have access and shouldn’t: 1. Try to get in touch, quickly, with someone with the power to fix it and explain what needs to be rotated. 2. Absent 1, especially if it cannot be done quickly, rotate the credentials personally to get them back to a controlled state (by someone who actually understands the security implications) with the intent to hand them off. Especially if you still _think_ of yourself as responsible for the infrastructure, this is a no-brainer compared to letting anyone else who might be in the same “should have lost access but didn’t, due to negligence” maintain access. Not a legal defense, but let’s not be too hasty to judge.
- saltypal 11mo ago
  I hadn't yet seen it when I wrote this, but 2 is pretty much exactly what Arko says: > Worried about the possibility of hacked accounts or some sort of social engineering, I took action as the primary on-call engineer to lock down the AWS account and prevent any actions by possible attackers. https://andre.arko.net/2025/10/09/the-rubygems-security-incident/ https://andre.arko.net/2025/10/09/the-rubygems-security-inci...

skywhopper 11mo ago

The most notable thing about this post is the great lengths to which they are willing to go to detail every step of this process, something they never came close to doing in the fallout from the original stripping of rights from the longtime volunteers.

fijiaarone 11mo ago

At least it’s not npm. Or cargo.

raggi 11mo ago

Honestly I think everyone involved here, and I know most of them, needs to step back and grow up. Despite the formal tone, and the potential legal issues at play here, this situation seems to be descending into needless childishness with potential life damaging side effects. All sides are being disrespectful.

fareesh 11mo ago

I've read both sides of the story and it does not make sense why he would change the root password and not communicate this with RubyCentral or make any attempt to contact them about the change. That in itself suggests malice.

neverrroot 11mo ago
This, cutting through the noise, this is it.

notpushkin 11mo ago

Update from Andre Arko: https://andre.arko.net/2025/10/09/the-rubygems-security-incident/ https://andre.arko.net/2025/10/09/the-rubygems-security-inci... Discussion: https://news.ycombinator.com/item?id=45535149 https://news.ycombinator.com/item?id=45535149 Not a good look for Ruby Central IMO.

ycombinatrix 11mo ago
Yikes.

shivenigma 11mo ago

RC seems to be incompetent and malicious, the original hostile take over is explained as a security measure. But in reality, you can simply just take away the ability to deploy and leave access to contribute to the repo untouched for people outside of the RC organization. But claiming that you care about security while missing a basic step of rotating credentials once a member has moved away is pathetic. God help Rubygems.org and the users.