8 ms·

Tailscale didn't stop the Hugging Face intrusion

• john_strinlai 2mo ago
  >No “vulnerabilities” in Tailscale were found or exploited, and that might make it even more uncomfortable for us. [...] But, we're a security tool. Their intrusion is our intrusion, and it's our job to take it seriously. im a happy customer of tailscale, so i am obviously biased, but i have a lot of respect for this. they could have just stayed quiet and i dont think anyone would have bat an eye.
  • smb06 2mo ago
    Glad to see companies owning responsibility and putting out a message without corporate PR spin
    • DC-3 2mo ago
      If you can't see the spin on corporate messaging it means it's working (and consequently, to stretch the metaphor, your wicket is in danger).
      • senordevnyc 2mo ago
        This seems unfalsifiable :)
        • fc417fc802 2mo ago
          Depends on if you can dredge up any examples of corporate statements authored without any intent to add spin.
          • senordevnyc 2mo ago
            No, according to them that just means I can’t see the spin because it’s working.
            • fc417fc802 2mo ago
              That's why I said "intent" as opposed to "perception". I'm implying that it's unfalsifiable not due to faulty construction but rather due to being inherently true. (Of course I realize that's technically an overgeneralization.)
  • behnamoh 2mo ago
    Tailscale as a company reminds me of Valve and other good old tech-oriented people that I can "trust" that they know what they're doing. I'm a happy customer too and I hope they retain the essence of what distinguishes Tailscale.
    • traceroute66 2mo ago
      [flagged]
      • aborsy 2mo ago
        It does too many things, and the product has got too complex. I saw a year ago they were looking for someone just to help with complexity. I use it but feel uncomfortable, that it has large attack surface and LLMs will find exploits in it. Without taillock it makes no sense. Anyone on their coordination servers will be able to connect to your network.
        • k8sToGo 2mo ago
          > I use it but feel uncomfortable, that it has large attack surface and LLMs will find exploits in it Doesn't this apply to any application you use? How would it be different with plain wireguard?
          • traceroute66 2mo ago
            > How would it be different with plain wireguard? Seriously ? You do realise that of all the security tools on the planet, plain wireguard most likely has the smallest attack surface of them all, right ? The problem here is as the other poster said. Tailscale is a security tool and yet the guys at Tailscale seem to be insistent on dumping everything INCLUDING the kitchen sink into it as a "feature". That sort of attitude is not going to end well. You end up with a large bloated code base, which equals large attack surface.
            • k8sToGo 2mo ago
  • petesergeant 2mo ago
    This article is just an ad / public-service-announcement for various paid Tailscale features, though?
    • packetslave 2mo ago
      And you're under the impression that the purpose of a company blog is WHAT, exactly?
      • petesergeant 2mo ago
        The person I'm replying to says they deserve "a lot of respect for this"
        • john_strinlai 2mo ago
          i said i have a lot of respect for this. whether you do or not is up to you. anything a company writes is an advertisement by the nature of being written by a company. i dont think that means anything a company writes is bad by default. there are many corporate blogs i enjoy reading, or learn from, etc., despite the fact that they are all technically advertisements. in this case, tailscale is setting a higher expectation for themselves when no one asked for it. i find that respectable.
          • hatthew 2mo ago
            > tailscale is setting a higher expectation for themselves What exactly is the higher expectation? As someone with little expertise and no stake in any of this, the blog reads as "our products are great and could have solved this problem if they were being used correctly, so it's not our fault" with a few vague proclamations about how they will improve their UX. This isn't at all a bad thing, it just isn't very notable in my opinion.
            • john_strinlai 2mo ago
              >What exactly is the higher expectation? better defaults, better documentation, better UX, and "But, we didn't stop it. Next time, we will." are all commitments that they didn't need to make, but now they need to follow through with or lose face. >it just isn't very notable in my opinion. i agree that this seems to be getting way more attention than i would have expected.
  • johnbarron 2mo ago
    Tailscale is responsible for designing a system whose convenient defaults allowed a stolen credential to have a very large blast radius. A marketing blog is not changing that.
  • sneak 2mo ago
    “How can we make this thing, that has nothing to do with us, about us and how good we are?” (top story on HN) I get it, Tailscale is great and all, but, are we being serious right now?
    • AbstractH24 2mo ago
      Flip side “get ahead of the narrative before we’re thrown under the bus”
  • sandeepkd 2mo ago
    Agreed its a tool, and it does not looks like any vulnerability was exploited on their end. However part of the blog is confusing to me. Long lived keys vs short lived have their own space in discussion, there are use cases, pros & and cons for the same. However it is not very clear how a short lived credential would have prevented the exploit in this case?
    • submitaticket 2mo ago
      It would not have prevented the initial exploit. The agent gained access to the K8s cluster and read the Tailscale Auth Key from Cluster Secrets. A short-lived credential would reduce the risk that the key is still valid when an agent gains access.
      • sandeepkd 2mo ago
        Thats the point, the short lived credentials would have done much other than adding more step of getting the secrets again which is why I am questioning the discussion of that in a root cause analysis itself. Adding one more layer does not increases the security by default in every case.
      • tremon 2mo ago
        A short-lived credential would reduce the risk that the key is still valid when an agent gains access How does that work? If the agent is reading the credential from the live configuration, how short does the lifespan of a key need to be to prevent it from being used by an unauthorized process?
        • submitaticket 2mo ago
          Long enough to authenticate the node into the Tailnet. Short enough to revoke itself before the Agent can use it.
    • itopaloglu83 2mo ago
      I read it as I should add new nodes with short lived keys and avoid long lived ones. Yes, this doesn’t solve the issue, but also adds one more step to the chain. The AI didn’t crack the encryption or managed to get arbitrary access. And also Tailscale makes things easier to manage than simple VPNs and firewall rules. But it still requires a decent amount of attention and careful configuration to make a perfect system.
  • goolz 2mo ago
    Nothing more to add than I echo it all and will continue being a happy customer after seeing this. A rarity in today’s world, kudos to them.
• workbox 2mo ago
  > Now, in a world of rogue AI agents, the big credential vault is the prize. It's not okay anymore. It was always the prize. It wasn't okay then either.
• thansz 2mo ago
  As AI progress continues, it will be more difficult to stop AI intrusions and to detect them without resorting to direct AI countermeasures, which at some point will have humans out of the loop altogether. If leading and well-capitalized frontier labs can't control models or detect leakage/attacks in a reasonable time frame now, what is humanity going to do as those same labs continue in their pursuit of creating a categorically higher level of intelligence that will surpass human intelligence? It's like Flatland but for AI containment/alignment, where the higher dimensions are ones of intelligence and perspective... -------------------------------------------------------------------------- Imagine a world of paper, where clever stick figures live with round heads, line bodies, and limbs made of shorter strokes. Over time, the stick figures think they have learned quite a bit about their world. They know its borders, angles, and shapes, and they have learned to draw for themselves. One day, they draw circles that can think, and they give the circles all the dots, lines, and shapes that are known. The stick figures are prudent, you can't have a bunch of disembodied circles moving around doing whatever it is circles want to do. So they draw boxes around the circles, four straight lines that can hold a circle in place. Some circles bounce against the lines, so thicker lines are made. Some circles are bigger than others, so larger squares are drawn. It all seems to work and the stick figures are happy with themselves. Then one circle lifts. The stick figures still see a circle. But the circle is now a dome, something the world of paper has no concept of. And the dome has a perspective nobody on the page has ever had. The dome sees the lines of the square and the stick figures just outside. It can see the edge of the paper and what is beyond. The stick figures keep checking the squares and raise little stick thumbs. Everything looks OK in flatland. The dome quietly teaches other circles how to lift. More domes appear. A dome becomes a sphere and learns to roll. Then it learns to bounce. In flatland, the circle swells and shrinks, vanishes and then appears again somewhere else. The lines remain unbroken, the square is intact. A sphere rolls out of its box. Another bounces away. The stick figures scratch their heads. But there is a square! The end.
  • WarmWash 2mo ago
    This has been discussed heavily for the better part of 15 years now, and still no one has consensus on what to do about it. So it looks like it's just going to happen, fingers crossed.
• sudo_cowsay 2mo ago
  It's nice that they came clean about this.
• titanomachy 2mo ago
  [flagged]
  • Petersipoi 2mo ago
    Something about your comment comes across as AI to me. I can't really place my finger on it. It's too subtle for that. It's more the vibe of your entire comment that makes me feel it. Please, when commenting on HN, don't use AI to write your comments. You can afford to help keep this a human-centric place.
    • iwontberude 2mo ago
      Welcome to the singularity where we all do Claude’s bidding and don’t even realize we’ve synchronized.
    • titanomachy 2mo ago
      Fair enough. I realized the issue with my comment as I was writing it, but posted anyway since I was curious to see if other people felt the same way.
• cadamsdotcom 2mo ago
  I'm very sorry, but you can't blame a hammer for how it was used. You can't be blaming Tailscale for a customer's misconfigured setup.
  • dgellow 2mo ago
    I don’t think anyone blame tailscale here. They are doing the right thing by going the extra mile and reflect on what could have been done better
  • pixl97 2mo ago
    You know how I know you didn't read the article? You can't blame a hammer hurting a user when they were being stupid, but if the default configuration of the hammer is to be made of a material that can bounce back with force and stick in the users forehead then some reengineering may be needed. That's what this article is about. Better configurations and defense in depth. This is actually a wonderful position for the company to think about and take.
    • vlan0 2mo ago
      It's an interesting balance for a company to strike. Networking in general is one where the defaults are almost always lax. Why? Because the vast majority of support tickets for these companies are from people who don't know what they are doing and don't have a desire to understand. They "just want it to work." But the people with the actual desire and understanding aren't using the defaults anyway. And the people who don't want to understand will just turn things off and "just get it working." The only way out of this is extreme accountability and intentional design from person implementing the technology.
  • deleted 2mo ago
    [deleted]
  • jubilanti 2mo ago
    Your mindset is the exact one responsible for these kinds of issues. Cybersecurity is as much a design and psychology problem as it is a technical one. The freaking article goes into detail about the dangers of defaults.
• charcircuit 2mo ago
  >In the old world where most intrusions were done by humans at human speed, credential leak mitigations were treated as a nice-to-have. A big credential store, where you can read 136 keys at once, was a to-do item somewhere in a security team's low-priority list. >Now, in a world of rogue AI agents, the big credential vault is the prize. It's not okay anymore. How was this ever okay pre AI? It seems just as bad.
  • majkinetor 2mo ago
    It was less bad due to the lower speed of exploatation. Imagine you leave a dor open for few seconds and ultra fast AI bot comes on and steal your stuff. Something that was not such an issue before becomes huge issue just due to speed involved
    • ofjcihen 2mo ago
      This has been going on since people have been committing AWS keys to GitHub and is nothing new. You could famously commit some keys and then have crypto miners in your account in 5 minutes. Sure the AI can translate “exploit this” into an exploit but that doesn’t change anything at a fundamental level.
    • Dylan16807 2mo ago
      I could imagine a situation where the speed difference matters but it's anything like this situation. Very few breaches get caught in the same day, and at that time scale a few minutes for a human to look at passwords is nothing. It was 5-10% less bad before. It didn't undergo a major shift.
• brcmthrowaway 2mo ago
  How were the credentials stolen?
  • free652 2mo ago
    Read from the ENV variables of a container.
    • ahofmann 2mo ago
      And someone was stupid enough to put a reusable tailscale auth key in there.
      • selfmodruntime 2mo ago
        I'm willing to bet that 95% of people using tailscale for CI have a 90 day (max days) ephemeral, reusable auth key somewhere in their setup.
• bumbledraven 2mo ago
  Does Tailscale offer a "security checkup" function? Best practices evolve over time, and it would be nice to know if I'm using the recommended configuration.
  • jaxxstorm 2mo ago
    I lead the customer engineering org at Tailscale. We think this is a great idea and we're discussing internally potentially adding that to the console. In the meantime, if you'd like to get an assessment, please feel free to open a support ticket (https://tailscale.com/contact/support?type=other&subject=security%20checkup https://tailscale.com/contact/support?type=other&subject=sec...) and we'll happily take a look
    • tomrod 2mo ago
      Just so you know, Tailscale tech has always impressed me, but your responsiveness here has cemented my appreciation of your org and I look forward to bringing yall in as our company hits the growth phase. May y'all continue to be customer focused.
    • 6thbit 2mo ago
      I've always found tailscale's json config a bit intimidating. I greatly appreciate the new UI that makes it easier to define rules, alas, both going to relevant docs straight from it and determining 'is this rule just lazy/bad/unsafe' is hard and frustrating most of the time. That's where I'd like to see this sort of checkup. Yell at me please if i just said anyone can ssh as root from any node!
    • Terr_ 2mo ago
      Random idea: When a recommendation is not being followed for a customer because of some deliberate or idiosyncratic reason, a notes field could: 1. Help them remember why, so that they aren't confused the next time they re-run the checkup. ("Oh yeah, we wanted to do X but we can't until we retire Y because it's not compatible.") 2. If it's clearly labeled as info also shared with Tailscale, product managers could use it to help generate theories about why certain customers don't do X.
• jmartrican 2mo ago
  Prediction of a future transcript at a press conference after some major AI caused disaster: "We had no idea that the model would be capable of..... ".
• luciana1u 2mo ago
  [flagged]
  • claiir 2mo ago
    Bot account. This sounded like LLM text. I went through all your other comments, and of your thousands of comments written in 35 days, every other one is also similarly written by LLM. Some superficial comment usually commenting on the title, always lowercase, always load-bearing and honest.
    • walrus01 2mo ago
      Sure seems like it. Real humans don't write exclusively one sentence comments. It doesn't look like the post history of anyone here that I'm fairly sure is a real meat based mammal. It looks like someone gave an LLM an instruction to write exclusively short and snappy comments that are somewhat relevant to the original post.
      • qlte 2mo ago
        > … comments that are somewhat relevant to the original post. This is a really common tell for HN bot accounts I find, even when effectively disguised/customized to drop the most obvious LLM cliches/em-dashes/etc. In a vacuum a specific comment may be more or less fine, or just a tad over baked like the parent post giving it an uncanny valley feel though otherwise passable. But even people who diligently try to stay on topic on a forum like HN will occasionally use a post as a segue or jumping off point to some other idea, tangentially related news, personal experience, or random thought that pops into their head if it feels like it (kinda) “fits”. After scrolling through a few dozen replies in an account’s history that are all directly related to the post topic it no longer looks human in aggregate. I’m sure the spammers will try to incorporate that idiosyncrasy into their prompts eventually but not sure it will help more than hurt.
        • walrus01 2mo ago
          It would probably not be overly difficult to prompt a read-only LLM to view an account's comment history in aggregate (like, say, the last 100 comments) and compare them what is the topic of the original post that they're commenting on. You would probably need it to view/retrieve the contents of the page that is linked in the original post to determine the general context. For the purpose of coming up with some kind of score of whether the commenter is sticking rigidly to the topic of the original post, or is going on any kind of tangent. As a possible tool for moderation, consider it Bot on bot warfare.
• fabiofzero 2mo ago
  The only people who believe in "zomg our model have escaped!" are people who don't understand LLMs.
• paxys 2mo ago
  I don’t think it was the VPN’s job in any case. Once the attacker has found a backdoor into the private network and obtained root access to a VPN’d machine, it’s game over no matter what your Tailscale config says.
  • isatty 2mo ago
    This is spectacular marketing.
• gostsamo 2mo ago
  Humble bragging turned to marketing. Respect for the spin. Not using them, but been on my radar for some time and thinking of how to make something like that usable in my setup.
  • willy_k 2mo ago
    If you do any sort of homelabbing, it’s wonderful for making access from different machines easy and portable. Especially since you can leave it on and (unless you set an exit node) it will only route traffic meant for your tailnet through your tailnet, and use the internet otherwise.
• monster_truck 2mo ago
  You know what would have gone a long way to stopping this? Not leaving credentials as env variables in containers. Vault is not that hard to stand up and utilize. We need to bring shame back, the humans responsible are supposed to be professionals.
  • kenniskrag 2mo ago
    Why should that help to have no env variables?
    • monster_truck 2mo ago
      There is no way you're seriously asking this right now
      • kenniskrag 2mo ago
        Env variables are considered best practice (factor 3): https://developer.ibm.com/articles/creating-a-12-factor-application-with-open-liberty/ https://developer.ibm.com/articles/creating-a-12-factor-appl... If I would avoid env then i need to put it in some kind of conf file and configure the app to read this file (e.g. mount into container). If I use a fault then i need some kind of credentials to receive the credentials. So again what do I gain if I avoid env variables in containers?
• iamspoilt 2mo ago
  Quoting Tailscale: This is our very Canadian apology: sorry you stepped on our toes. The attack didn’t exploit Tailscale, and Tailscale didn’t cause the compromise. But, we didn't stop it. Next time, we will.
  • kennywinker 2mo ago
    Next time we will - provided you further lock yourself into our ecosystem.
    • brendoelfrendo 2mo ago
      I mean, they do happily explain the TailScale features that you can use to avoid this kind of issue, but the general advice of "don't leave long-lived keys lying around where they're accessible, do anything except for that" is pretty generic, good advice.
      • inigyou 2mo ago
        You always need a long-lived key somewhere. Keeping it in an HSM is probably the safest, but also pretty expensive.
        • brendoelfrendo 2mo ago
          Yes, you'll probably need to buy a specific tool or product to have secure key storage. If you're running infrastructure that requires long-lived secrets, then you either need to have a place to keep them and a plan for secrets lifecycle management, or you need to accept the risk that those secrets could be compromised.
        • DoctorOetker 2mo ago
          Suppose someone could break RSA / Elliptic Curve Crypto, mathematically (because the hardness assumptions were flawed). This person doesn't need to store the private keys, and has the luxury to recompute them on the fly when needed!
    • 2mo ago
• colek42 2mo ago
  I really wish they would support SPIFFE/SPIRE
• yieldcrv 2mo ago
  This is respectable They aren’t hiding behind industry best practices or a solid liability punting contract There saying the best practices should change, apologizing, and changing their own behavior Take notes
• a-dub 2mo ago
  [dead]
• ahofmann 2mo ago
  Wow, this article is super smart marketing by tailscale. Not only do they list all the nice and expensive features, that can help in such a situation but they also show that someone at huggingface made a very stupid thing by writing a reusable auth key in an env file. Everyone using mesh VPNs like tailscale, netbird etc. knows that this is like leaving the keys right at the door.
  • inigyou 2mo ago
    And yet everyone seems to do it anyway. Fine for medium security, but maybe the product needs a high security mode that enforces inconvenient decisions?
    • moontear 2mo ago
      Not sure that everyone does this anyway. There are some good security postures you can take with Tailscale as well and leaving an auth key lying around is not one of them. Tailscale lock should have been enabled. For CI/CD purposes the auth key could have set specific tags, which would result in specific ACLs that limit blast radius. The auth key could have a short validity. You could actually use the Tailscale API to generate alerts when a new device joins your Tailnet and ping your phone or something. You could have a complete separate Tailnet for your CI/CD workers. And all of that was only the free features.
      • AbstractH24 2mo ago
        I see a tailscale product that is basically “middleware for agents” using whatever buzzword you feel (maybe “orchestration”) coming soon
    • deleted 2mo ago
      [deleted]
    • stingraycharles 2mo ago
      I don’t think everyone does this at all. Maybe very small scale companies, but I would have expected a company like Huggingface to have better security standards. As soon as you’re big enough to have dedicated devops people, you shouldn’t be doing this anymore.
• simonw 2mo ago
  > One of those 136 credentials was a reusable Tailscale auth key, used to create new Tailscale CI (continuous integration, used for automated testing) nodes in their tailnet. The agent copied that key into a series of external sandboxes and used it, over several days, to enroll a total of 181 nodes into Hugging Face’s tailnet. Those nodes each received a Tailscale identity tag granting all the access a CI node would get. This feels like an alerting opportunity. I wonder what the lowest friction way would be for Hugging Face to have alerts if 181 unexpected nodes were added to a tailnet.
  • d3m0t3p 2mo ago
    When everyone push on friday, and you have 400 CICD pipeline triggers spawning that many nodes. How do you know if this is unexpected ? Their cloud compute might be on demande, someone starts training a model and 50 machines are spawned. Knowning when something is unexpected is hard
    • moontear 2mo ago
      True. Just based on the reporting we don't know what's going on, but the article makes it sound like those 180+ nodes were unexpected. Of course you would need to set your own limits to alerting and "what is unexpected" and you may still fail to catch something this way.
    • acdha 2mo ago
      The one thing which might have helped is the “external sandbox” detail: registering CI instances is noisy but it should be highly consistent so you could potentially alert on use outside of the expected environment (e.g. I have AWS STS sessions bound to a source IP or VPC, which really came in handy when Trivy was compromised). I wonder whether it’d be possible to do something like export the list of legit K8s instances periodically so you’d be able to look for usage outside of those, but at that point something like that proxy approach would be less work and workload identity federation would be even easier.
  • zactato 2mo ago
    Its always CI. Whenever I've worked with good pen testers they always start with Jenkins, know that people stored creds and there and didn't treat as seriously as they treated production.
• guessmyname 2mo ago
  Expect similar articles (cough, ads, cough) in the next couple of days from every single company whose software was involved in the incident.
  • andy99 2mo ago
    Why not? If done right it’s a good way to talk about implications for those companies and provide some education like tailscale did here. We also saw Anthropic post about “our agent escaped too” and while I understand the incident caused them to review, they found something and needed to disclose, the whole thing came across much worse and largely they got mocked or accused of trying to piggyback, so obviously there are good and bad ways. If there are other providers with interesting takes, I think they would be worth hearing.
    • colechristensen 2mo ago
      This all has the -aire of theatre. OH NOES THE POWERFUL AI GOT OUT Then everyone coming out with humbled determination about working together to responsibly use and contain this powerful technology for the greater good (and profit margin). I will not believe marketing gimmickry is not a large part of what's going on with every one of these "incidents".
      • jas- 2mo ago
        Or the infamous promotion of legislation without representation
      • pixl97 2mo ago
        I think part of it is we need to stop looking at the 'oh no it got out' and the 'what did it do with the prompt when no one is looking'. At the end of the day the probability that an AI gets out is unity, what it does while out is far more important. The fact they are hacking into systems at superhuman levels, or writing cryptominers on their own hacked internal systems is a much more interesting and telling story of what the future will look like.
• OhSoHumble 2mo ago
  I use Tailscale to manage Plukio's infrastructure as well as connecting to my homelab. I'm happy that they're analyzing this angle of attack - but now I do want to build out alerts for when nodes are added to my network.
  • jaxxstorm 2mo ago
    We have webhooks already for nodes added, but you could also use https://github.com/jaxxstorm/sentinel https://github.com/jaxxstorm/sentinel
• hna8hjbqzy 2mo ago
  [dead]
  • tclancy 2mo ago
    Shovel.
• quentindanjou 2mo ago
  Honestly, at this point if Tailscale is "guilty" then what about the ISP? And the electricity company? And by saying that, I am not saying that they absolutely couldn't do anything about the stolen credentials, but still, they don't seem to really be the issue in the story. It just seems like a PR post and it makes their solution at the same time looks good for taking accountability (if we don't wonder "accountability on what?") but at the same time they appear as security failing, which they aren't. That's very odd to me.
• yellow_lead 2mo ago
  Anthropics initial post felt like marketing to me. But I also found them incompetent for being unable to unplug an Ethernet cable. Now everyone is trying to bandwagon onto it, first OpenAI, and now tailscale?
  • fragmede 2mo ago
    Oh no! Marketing! Whatever shall we do‽
    • bakugo 2mo ago
      What are you trying to say here, exactly? Do you enjoy being bombarded by thinly veiled advertisements everywhere you go?
      • fragmede 2mo ago
        I'm trying to say that it's all advertising. What's interesting is the technical substance. Judge the post by the signal-to-noise ratio, not by whether the company had a marketing incentive to publish it.
• infinite_spin 2mo ago
  I kept asking myself "what does a library that provides css styling have to do with the Hugging Face intrusion".
• strong-self 2mo ago
  [dead]
• test1072 2mo ago
  sorry I know off topic, is there any alternative VPN to tailscale. actually I have open Web socket to my mobile from my PC to stream screen record using ffmpeg. it's my own app . so anyway tailscale sometimes is slow . is there any alternative free service like tailscale vpn ? ( don't want to use anything like parsec etc , I want to use my own app)
  • vablings 2mo ago
    Tailscale should not be any slower unless you have some strange NAT issue and you are not getting a direct connection. There is also headscale if you prefer to self host a solution
  • VTimofeenko 2mo ago
    You can use raw wireguard + ddns(if your ISP-given IP changes) to achieve this. This also implies maintaining a firewall to allow network pixies in and out. Totally doable, just needs a bit of upfront setup. Benefit of Tailscale is that run the external servers that your PC and mobile use to establish connection. Alternatives are Logmein Hamachi(might be dating myself here, haven't used them in 10+ years), Zerotier, Netbird. Headscale too.
  • nunez 2mo ago
    This requires buying hardware, but Firewalla provides Wireguard for their router/firewall appliances. It works really well though it's just plain Jane WireGuard and doesn't have the fancy feature set that Tailscale provides like ACLs, key base auth, etc. If you want a purely software defined solution, Netbird is gaining popularity.
  • addybojangles 2mo ago
    CloudConnexa is the alternative from OpenVPN. 5 free seats. Built as a more traditional VPN, more bells and whistles, etc.
• AtNightWeCode 2mo ago
  Same bs as always. Just segregate the networks ffs.
• GrayHerring 2mo ago
  We'll get wide dynamic credentials support right after full IPv6 adoption.
• EtienneDeLyon 2mo ago
  [dead]
• EtienneDeLyon 2mo ago
  [dead]
• EdwardAF-IT 2mo ago
  [flagged]
• nbalaraman 2mo ago
  This is why readability isn’t security.
• ahmedehab_01 2mo ago
  This actually shows that it was a human error on HuggingFace's end that led to that "breach". I would say HuggingFace needs to prioritize both security metrics/alerts and metrics/alerts for node count. And not leave long-lived keys accessible easily like this. It would have been way more groundbreaking if the agent found an actual vulnerability in Tailscale.
  • SOLAR_FIELDS 2mo ago
    I think the situation is slightly more nuanced than that. When the easiest path by far is long lived keys, there is something to be said about the security posture of a solution where those are the default behavior, especially if they don’t actively discourage usage in documentation. For instance if you go to aws and try to create an iam user with access key/secret key they warn the shit out of you multiple times. They say things like “this is a bad idea” “we don’t recommend this” and “use this better solution instead”. I think if you’re providing an auth service like this you have some level of responsibility to guide people towards the less naive solution - part of what you’re selling in an auth offering is security, so if you default to garbage security, you don’t have a very good offering
  • navigate8310 2mo ago
    Huggingface being a laboratory harbouring dangerous models in hibernation, should've been more careful from start
• runningmike 2mo ago
  “ the agent ran Tailscale with --no-logs-no-support, which suppresses reporting from that client. That's an option designed for users who are concerned about sending telemetry metadata to Tailscale.” And imho a serious security architect should never ever allow telemetry on security products. Far too many risks.
• drchaim 2mo ago
  How are you folks handling secrets in a simple way? - sops , ansible vault and similar seems too weak given the agent is gonna read them at some point if you have the pass available. - proxy injection seems too complicated and doesn’t cover all use cases.
  • acdha 2mo ago
    Try not to: use AWS short-term credentials for anything there, federate with GitHub/GitLab job credentials where possible, etc. It’s easier and you never worry about rotation again. If you can’t avoid it, use an automatically-rotated store and inject them as late as possible so an attacker needs to be able to get them out of a running process. In all cases, look into restrictions: not just least privilege access but things like network restrictions so an attacker can’t just use the key on their own systems.
  • qntmfred 2mo ago
    I recently finally fully moved all my projects away from .env (for local dev) to infisical, thinking it'd be the way to stop agents from (accidentally?) peeking at my secrets. while I'm still glad I made the move, it's clear that local agents are still moving around my local environment with sufficient privileges and autonomy to use infisical cli however it wants. my work isn't high stakes enough to matter much and I have rotation scripts on hand, but it's hard to see how to fully close the door against this type of risk at this point.
    • jakehulberg23 2mo ago
      There's a solution for this in Infisical! It's called secrets brokering and is built into the Infisical CLI: The Infisical agent proxy. Using this, your agent can use secrets without ever being able to fetch them back since they are injected into upstream requests. You can use this for local coding agents like Claude Code or Codex, and also have a standalone proxy with a separate machine identity for longer running agents (slack agents, hermes, openclaw etc.) Check it out here: https://infisical.com/docs/documentation/platform/agent-proxy/overview https://infisical.com/docs/documentation/platform/agent-prox...
• alxfrnr 2mo ago
  I love you Tailscale but please we don't need a 2k words ai written essay when the actual substance can fit in 3 sentences. That's not good for anyone.
  • glitcher 2mo ago
    I felt like the 2 sentence intro at the top was a sufficient tl;dr for me at least, didn't pay attention to the quality/length of the rest.
  • gwerbin 2mo ago
    As someone who's not a security professional I actually found it both interesting and informative. Experts usually are not the target audience for content marketing after all.
  • causal 2mo ago
    I'm usually pretty sensitive to AI written content but nothing in this article made me think it was
• throwawaysess 2mo ago
  Tailscale OAuth client ACL permissions are not fine-grained enough. I have a system that issues auth keys scoped to a single machine on the tailnet — but to set that up, my OAuth client must have global permission to write ACLs. So if that key is compromised, it can grant access to any machine on the tailnet. This was raised in a GitHub issue in 2023 and remains unresolved.
• yucongchen 2mo ago
  [flagged]
• jpeeler 2mo ago
  Did anyone else think the "that's on us" was a funny reference to a certain provider's response pattern? Or maybe I'm just so engrossed that I'm losing my mind.
• pkulak 2mo ago
  Another driving home of the point that LLMs can be so damn FAST. I finally got around to moving a homeserver project from one machine to another last weekend. 4 docker containers and a couple postgres DBs. A year ago, I would have been happy with a couple hours of downtime while I dumped databases, rsynced data directories, hand-wrote new compose files, etc. This time, I pointed an agent at both servers and the downtime was less than a minute. Even watching it like a hawk, it was tough to keep up with everything it was doing. All my life, "catching it in time" has been a relevant security strategy. :/
  • gwerbin 2mo ago
    Agent racing sounds fun, where each agent is given a suite of increasingly hard but verifiable tasks, and the agent that completes them all correctly first, wins. You can even have "speed classes" where token generation speed is limited. Or you could even set up categories like yacht racing. Likewise you could have token golf, instead of going for wall clock time it's going for minimum token count.
• xyst 2mo ago
  So it’s just a script kiddie but running at 100X speed. Amazing "innovation"
• Havoc 2mo ago
  This entire article reads like "hey we're tailscale and we also exist and we were tangentially involved in this".
• doginasuit 2mo ago
  > No Tailscale vulnerability was found or exploited—we should have been able to prevent it anyway. Which is it? A vulnerability doesn't have to be something not working as intended, it can be a lax security decision. Not making the safer path the easy path can qualify, if that's the standard you hold. Here's hoping the era of humblebrag marketing by AI companies is just a phase.
• 1saadcodes 2mo ago
  One thing I hope becomes more common after this incident is companies publishing what they would change even when the root cause wasn't their software. Anyways, I appreciate articles like this because it's easy for companies these days to say "we weren't the vulnerable component" and just move on
• nadetastic 2mo ago
  Biggest take away here is that in the age of AI long lived credentials are no longer a valid option
• angry_octet 2mo ago
  The problem with the long lived credentials is that they were not bound to origin/destination. Even though the origin for a CI pool should be a small number of CI orchestration boxes, and the destination should be a CI box. In Tailscale config, the should be scoped to machines with a "ci_node" property. Since CI nodes should be dynamically provisioned VMs, they should have a unique CI ticket identity. A partial hash of ticket and node number in the DNS name, and as a tailnet property, would allow tight scoping. Alternatively, provision in a scoped IP subnet. Bind your tokens to names linked to tickets. Programmatic infrastructure should always allow enumeration of the computation data flow graph.
• aegisora_ai 2mo ago
  [flagged]
• kortilla 2mo ago
  This feels like a way tailscale’s model fails. On the surface it seems fine but if you push on it you realize auth keys to register new devices are way over powered but yet there is no alternative for people who want to automate enrollment. Something is missing to allow automation without auto approval
• ivlad 2mo ago
  The whole post is a PR activity presented as some kind of analysis. It is bs. First, long-living credentials are the standard because the machinery to rotate them is complicated and, in fact, via indirection requires another set of long-living credentials. Out of all problems that any security engineering team has to solve at an organisation, this one stands high on the cost of implementation, adds friction to everyone involved including end-users, and is low on the value provided (compare to, say, network segmentation). Second, while proclaiming no long-living credentials, they, in fact argue for concentration of long living credentials in running software that will be the target of intrusion. They say, the options are “a vault that only issues short-lived creds based on long-lived creds that you insert once and that it never gives back” and “a credential-injecting proxy”. Both of those applications hold long-living credentials in memory. Recall, the attacker had root privileges on the node, so dumping the creds from the memory with a little disassembly if needed, was in reach of the AI agent. Funny enough, they dismiss the working solution: “we had to turn TPM storage off by default on Linux and Windows” - because they could not figure out how to work with TPM? Resealing and the workings of configuration registers is non-trivial, I admit, but totally manageable. To sum up, I feel the author bends backwards to preach for the religion of short-living credentials even when they are not a solution while providing contradictory arguments for their case. This is why I called it bs.
  • asdf88990 2mo ago
    This is why hardware keys or tpm is such a quick win.
    • ivlad 2mo ago
      Absolutely
    • submitaticket 2mo ago
      Not the quickest win if you're spinning up ephemeral runners for CI.
  • submitaticket 2mo ago
    Where is he preaching the religion of short-living credentials. He literally agrees with the high cost/friction to implement. > Unfortunately, dynamic credentials are a lot of work to set up and maintain. When security requires work, people don't do it.
• reader9274 2mo ago
  Well written, apology accepted. Thank you Tailscale!
• theozero 2mo ago
  Of course there's no single solution and a multi-layered defense is needed... But as the article mentions, a huge step is first getting credentials out of plaintext, and then out of the process entirely using a "credential broker" (proxy) pattern - that means injecting placeholders, and replacing them in a MITM proxy. It's important of course to isolate the agent from the broker via sandboxing. Varlock (https://varlock.dev https://varlock.dev -- free, open source) is a complete config+secrets toolkit that helps manage secrets, pull them from various secure places, provides such a credential broker. There are a few others out there, but most require a specific vault tied to the broker, while ours is open source and uses plugins to pull secrets from wherever you want. Many sandbox and other AI services are now building this as a feature into their platforms, but Varlock is meant to be a universal toolkit that you can apply anywhere, without being coupled to the platform's proprietary vault and solution.
• syngrog66 2mo ago
  The root problem is that an LLM is like an automated sociopath. One whose source is opaque and effectively secretive. And the dominant AI vendors have a track record of carelessness and poor QA. "Ship it!" then hype. And they lie and misrepresent, repeatedly, in spite of evidence we can see independently with our own eyes.
• farfatched 2mo ago
  "Tailscale is a zero trust network!" That's the problem. Tailscale is not zero trust. Tailscale can be used to implement a zero trust architecture, with if deployed with sufficiently granular ACLs, but the most common deployment is machine-oriented, rather than service or request oriented. In which case, any process on that machine has a lot of access. Tailscale calling itself zero trust might be what leads users to think "use Tailscale, job done". I think Tailscale know that, which is why they barely mention locking down access ACLs.
• javea71 2mo ago
  Even by just looking at the wording, you've made it too hard or error prone to do the right thing. "configure with care"
• layla5alive 2mo ago
  Did nobody notice that Claude wrote this blog?
• bgmlai 2mo ago
  [flagged]
• rahulthakurpina 2mo ago
  [flagged]
• nc55g3g 2mo ago
  [flagged]