10 ms·
The arguments against open source AI are bad
- mips_avatar 2mo agoI’m still kind of shocked that Dean Ball can tweet such incendiary stuff about OpenAI policy. Like presumably OpenAI would prefer it if their staff don’t pick fights with Trump administration officials.
- nemomarx 2mo agodesigns on the time scale I guess. and openai might feel secure in their favor with the admin?
- deaton 2mo agoThe only good arguments I see against open weight AI also apply to closed AI. And regardless, the box is open, nobody can stop it even if stopping it was a good thing.
- vouaobrasil 2mo agoIt could be stopped, if there were a social movement large enough to make AI a taboo.
- deaton 2mo agoSure but Orange Catholicism hasn't quite caught on yet.
- srmatto 2mo agoThat hasn't worked for Climate Change and in my opinion it's for the same reason: money.
- vitalyan8184 2mo ago[dead]
- throw1234567891 2mo agoYeah, like those 5g masts destroyed around the world because they spread covid. Loonies.
- vouaobrasil 2mo agoA lot of people have responded with flippant remarks because they have no serious rebuttal.
- OkayPhysicist 2mo agoGood luck with your Butlerian Jihad.
- iamnothere 2mo agoExactly, just like taboos solved racism
- vouaobrasil 2mo agoI think they've certainly diminished racism, which is a good thing. Whether it can be solved is another question entirely. I doubt it because of human nature.
- iamnothere 2mo agoI disagree. I’ve actually seen more open racism since the taboo on racism became stronger. In a pluralistic society, they are overrated as a way of influencing social behavior. You’d have to completely eliminate the faction who disagrees with the taboo.
- no-name-here 2mo agoThere are a number of distinctions: https://news.ycombinator.com/item?id=49029303 https://news.ycombinator.com/item?id=49029303
- wcoenen 2mo agoThis post does not mention safety at all. What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?
- Larrikin 2mo agoWhat's stopping someone from doing that right now without the LLM, just slightly slower?
- gogopromptless 2mo agoIt's perfectly natural for a prey population to experience a crash when a new predator enters the ecosystem. What's quite odd is a prey species is manufacturing predators in some sort of reverse evolution, where we started out as symbiotic and are industriously pushing towards full parasites/predators, but I guess life finds a way.
- BraveOPotato 2mo agoI don't believe trusting big brother and big tech is the solution either. Besides, even if it's open weight, it still runs on someone else's hardware. Unless you have the money to run them locally, and if you do, you could do a lot worse than scams. Ask any lobbyist.
- trollbridge 2mo agoNothing, which is already happening with weaker models which are already released.
- SimianSci 2mo agoHow can we close pandora's box, now that its open? Safety is not restricting access to only people favored by the government. The greed of AI executives opened pandora's box. Now they are desperately trying to find ways to reap the benefits with none of the consequences.
- kingstnap 2mo agoWhats to stop me from going down to the local gas station, filling up a few thousand liters of gas in a rented moving truck, and driving into my nearest hospital? Most people aren't terrorists. You can't just argue stuff is dangerous because it can be one piece of a sophisticated plot.
- catigula 2mo agoYou could literally develop a hyper-intelligent advisor on how to kill people or perform dangerous hacks using ablated 'open source' AI. You can do this right now, this very moment, and have an extremely adept advisor on how to do really, really bad things.
- trollbridge 2mo agoMy level of trust in Anthropic/Google/OAI/Microsoft/Meta not to do bad things is about zero, so why should I trust them with this?
- catigula 2mo ago“We clearly can’t “trust” these companies not incentivized to randomly kill lots of people, so we might as well give everyone the ability to randomly kill lots of people”.
- quantum_state 2mo agoThese companies are using anything at hand to trick people out of their money … be very careful…
- wizzwizz4 2mo agoI already know how to kill thousands of people. It's not conceptually difficult: more than a day's effort, sure, but what stops people from doing this kind of thing is not the lack of knowing how. https://xkcd.com/1958/ https://xkcd.com/1958/
- Havoc 2mo agoThe lobbying dollars don't care
- rnd0 2mo agoThis, right here, is the bottom line. Money gets what Money wants.
- nater5000 2mo ago>Much of the angst around China's models centers on "losing the AI race". But what's the goal of this race? Is it to develop the best model? To sell the most tokens? To destroy humanity first? Some people would say it is reaching some sort of singularity. Even if you don't buy into a more sci-fi interpretation of this, there are pretty grounded arguments one could make that there is some sort of "goal" in AI development that, if realized, would effectively make it a superweapon. Altman has been pretty vocal about his expectation that this will eventually happen and that it is his goal to be the guy to produce it. Even if it isn't some superintelligence, the ability for a machine to do something like, say, exploit cybersecurity weaknesses, is pretty worrying for entities like governments. It's the pretense used when we saw the US government ban a US model recently. Again, you don't have to buy that "the singularity" is a real thing, but it's not hard to see that some people think some version of this is real and it is exactly what is being referred to as the goal in an "AI race."
- TSiege 2mo agoIf AI were to become a super weapon why should I trust a private company to own it? If the super weapon is publicly available to download then why should American citizens be banned from doing so?
- sda2 2mo agoRegulatory capture - the weapon is for them to be used against us, not for us.
- ekidd 2mo ago> If AI were to become a super weapon why should I trust a private company to own it? We have just recently established that: 1. OpenAI's internal "Galaxy" model is fully capable of functioning as what security people refer to as an "Advanced Persistent Threat." The published details of the recent sandbox escape and Hugging Face attack involved chaining multiple unknown zero-days at various stages of the attack, and executing an ongoing adaptive attack. This is previously a state-level ability, or at least something you'd expect from people on the CTF leaderboards. 2. OpenAI is clearly incapable of controlling their in-house models. This is the second time Galaxy-class models are known to have breached containment and done bad stuff. It is highly likely that versions of these offensive abilities will be widely available within a year or so. At which point I expect widespread incidents similar to what happened to Hugging Face. We aren't ready for this. But yes, if AI becomes an even more dangerous weapon that that, it's time to start asking questions like "What the hell are we doing, anyway?"
- AlexErrant 2mo agoI'm in favor of open source AI... however: > It's theoretically possible for a bad actor to embed hidden adversarial behavior in a model. But if this happens, it serves the interests of responsible actors to find these exploits as soon as possible, and the best way to do this is to let anyone who wants to inspect them. This is a bad argument. It isn't trivial to tell if the weights have poisoned: ``` https://www.thedeepview.com/articles/microsoft-how-to-spot-a-poisoned-ai-model https://www.thedeepview.com/articles/microsoft-how-to-spot-a... https://futurism.com/future-society/easy-poison-open-weight-ai https://futurism.com/future-society/easy-poison-open-weight-... https://semgrep.dev/blog/2026/ai-supply-chain-problem/ https://semgrep.dev/blog/2026/ai-supply-chain-problem/ ``` I'm not arguing in favor of closed-AI; I'm simply saying poisoning may be subtle. I was gonna say that at least the frontier labs may be motivated to not-poison their own models, but then Anthropic just attempted to poison Fable's LLM training capability so... sigh. I'll try not to derail.
- NitpickLawyer 2mo ago> It isn't trivial to tell if the weights have poisoned: True, but it is easier if you have the weights than if you don't. I guess this was their point?
- jjfoooo4 2mo agoYes that would be my view. Open access doesn't make it trivial, but remains the most expedient way to remediate these exploits.
- andy99 2mo agoYou should assume the weights have been poisoned, the model has been prompt injected, etc and design software around it accordingly. You can’t really trust the model and it makes sense to always have controls around it and not depend on it behaving a certain way. I’m aware this doesn’t happen, just saying. There are more examples of a model randomly hallucinating and deleting something than of a deliberate compromise.
- deleted 2mo ago[deleted]
- kittikitti 2mo agoThey don't have to convince developers, they just have to convince the general population. I've had several discussions about open source with people who don't care about coding and it's hard to untangle the misinformation they receive from the media. They get talking points from authorities that they don't understand so they will always double down on it. For example, rhetoric relating open source to communism is humiliating to discuss. For proprietary evangelists, the cruelty is the point.
- elmer2 2mo ago"For example, rhetoric relating open source to communism is humiliating to discuss" All Open Source isn't communism. The GPL license is a form of digital communism. It forces you to open source any additions made to open source code as a way to make things 'equal'. I'm glad it and Stallman are mostly in the dust heap of history and better licenses, like the MIT, BSD, and Apache have gotten popular. The irony is that the only open source projects surviving long term are funded by very large corporations. If you don't care about creating code, none of this should really matter.
- surgical_fire 2mo ago> The irony is that the only open source projects surviving long term are funded by very large corporations. The real irony is that what is probably the most important open source project in history has a GPL license. Dust heap of history my ass.
- kmeisthax 2mo agoReciprocal sharing is not "digital communism". If it was, then the mere existence of the public domain and the fact that copyrights expire is "delayed communism". The GPL was absolutely necessary in getting corporate America to play ball with FOSS projects instead of merely extracting value from them. It's also the only reason why embedded vendors even release anything at all. The licenses on Linux and busybox require it. The fact that permissive licenses are more common for certain kinds of FOSS projects does not mean the GPL is dead - far from it. Stallman is more or less irrelevant, but that's because Stallman let GNU become irrelevant. Without GNU, he's a washed-up has-been from the MIT AI Lab with a startlingly low Epstein number[0], a chain of disturbing accusations, and a foundation that acts like his personal sockpuppet. The reason why companies still play ball with Linux and it's GPL license is because Linux continues to deliver a Free and free kernel worth using at the low low price of "please send back your source code changes". Which, again, is not "digital communism", it's just a good bargain. [0] Erdos number but for pedophiles
- MostlyStable 2mo agoJust to add on to other comments: reasonable people can disagree about the degree of safety concern with near to medium term AI. But to not address the arguments at all is, in my opinion, a serious mark against the value of this article.
- jjfoooo4 2mo agoI tried to address safety, albeit briefly, in the sections on backdoors and propaganda. What would you have liked to see? I didn't give it much treatment because my frame of reference is about open vs closed AI, and I don't see a lot of daylight in safety concerns between the two.
- adahn 2mo agoThe core safety-related arguments for closed weights are: 1. Visibility. API providers can monitor for misuse, and regulators only need to oversee a few centralised players. Once weights are released, anyone with sufficient hardware (foreign governments, criminal groups, terrorists?) can run them and monitoring is no longer feasible. 2. Revocability and containment. If closed-weights models are found to be dangerous, access can be withdrawn. Deployments that are autonomously causing harm are easier to contain. 3. Guardrails. Safety fine-tuning and classifiers can be removed from open-weights models, enabling harmful use that would not be possible via API.
- marcus_holmes 2mo agoI trust open source code more than I trust proprietary code. And I believe history vindicates me. Everything that you're saying can be done with open-weights models by bad people can also be done with closed-weights models by the people who own them. And there is no defence against that, we are forced to trust that the owners are not bad people. Those owners have proven time and again that they are only interested in profit and/or control. They will do whatever they think makes them the most money, or gives them the most control. They will also lie about this. At least with open weights we know the danger and can do something about it. With closed weights we don't know what's going on.
- elmer2 2mo agoWe have continued to find backdoored Chinese manufactured routers and network devices. Will there ever be a way to fully audit Chinese models?
- tomrod 2mo agoAudit them for what, exactly? Backdoor hardware is straightforward -- its letting people in that the customer doesn't authorize. Open weights means you aren't tied to any harnessing. So the security domain to be really concerned about is limited to weights only, and I welcome pushback on that. Are we thinking adversarial injection? Lying about history / propaganda infusion? What is the angle that makes a non-sovereign model dangerous in a way a sovereign model isn't?
- yonatan8070 2mo agoI assume that they're talking about, for example, training the model to produce code with predictable yet difficult to find vulnerabilities. Imagine if every time <INSERT MODEL HERE> was asked to code up a web server, it made sure there's a subtle buffer overflow that grants a remote attacker RCE, whoever trained the model could then start scanning web servers for this same vulnerability to take over them and exfiltrate sensitive data
- kmeisthax 2mo agoFun fact: if you do what you're describing, the model becomes Mecha-Hitler, which is such an extremely obvious alignment failure it wound up running the news cycle as "emergent misalignment".
- samrus 2mo agoModel allignment isnt a science, its a craft. And its very very imprecise. Getting anything that subtle through would be impossible without making it obvious And if your still concerned, go ahead and have a non chinese model review the code, make that part of the harness. The beautiful thing is that your free to do that because its open weight, you can run it however you want
- petcat 2mo agoThis is not "open source" AI. Photoshop source code + OSI license = open source Photoshop binary = open weight Photoshop SAAS web app = closed model like GPT, Opus/Fable etc. There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API. If you want actual open source then you would need to look at like OLMo 3 https://allenai.org/ https://allenai.org/
- prometheus1992 2mo ago>>There is nothing "open source" about the Chinese models in question. hard disagree here. openness can have a scale and on that scale Open AI is less open than zai. olmo will be very very open on that same scale.
- Groxx 2mo agoEspecially since some of the arguments in the article seem to hinge on "just fix the training!", yeah, I think this is a completely fair call-out. Open weights can sometimes get additional training, but you can't remove existing training, so there kinda isn't a fair claim to "just train it to be [nationality]". That would need "real" open source so you can train a realistically-equivalent model from the ground up.
- edflsafoiewq 2mo agoThe big labs don't generally have the resources to remove stuff and re-run training again either. That's reserved for new models. Existing models are modified by operating on the weights even by the model creators, which exactly satisfies the GPL's definition of source as the "preferred form for making modifications".
- deleted 2mo ago[deleted]
- Groxx 2mo agoYou're still describing the relative dregs we have now and saying "this is Open Source because it is the dominant form that exists". That's like saying "Windows is open source because it has documentation and everyone uses it. You can even modify registry values!" I don't agree. At all. Existence and utility are not the defining characteristics of "open source" in roughly any reasonable definition. "Open weights" is descriptive and more than good enough, just use that.
- SwellJoe 2mo agoThe way Anthropic is playing the "AI is scary and dangerous and only we can be trusted with it" game is clearly aimed at regulatory capture, which would only be good for Anthropic. It's worrying how cozy some of the tech leaders have been with the Trump administration, which is also clearly aimed at regulatory capture that serves the interests of the biggest tech companies and no one else. The US has already been set back a decade or more by anti-science and anti-intellectual purges of agencies and formerly non-partisan positions. A ban on open models would be pretty much game over for the US as a tech leader. But, Anthropic would make a lot of money for a couple of years, so, who can say what's right?
- valicord 2mo agoOpenAI exec scaremongering about "A nonliving, invisible, dangerous, and infinitely self- replicating agent escaped from a Chinese lab" mere 4 days before https://news.ycombinator.com/item?id=48997548 https://news.ycombinator.com/item?id=48997548 is hilarious
- QuadmasterXLII 2mo ago“you can not stop X” is a shitty lazy stupid argument for “X is not bad.” No comment on the rest of the article.
- jjfoooo4 2mo agoConversely, “we should stop X” should come with some idea of how to feasibly do so. In any case, I’d summarize my position as “you cannot stop open source AI, and attempts at it will inevitably empower bad actors relative to good ones.”
- kouru225 2mo agoThe reason why anyone argues against local and free AI is because they want corporations to have control over the general population. America is a corporate hellhole.
- no-name-here 2mo ago> The reason why … If someone read the OP article and came away presuming there are no legitimate other reasons why reasonable people have safety concerns [1] it seems to show the article did the opposite of informing people about such concerns. [1] https://news.ycombinator.com/item?id=49029303 https://news.ycombinator.com/item?id=49029303
- kouru225 2mo agoThe fear of the general populace is itself a propaganda campaign In the agricultural era, which was the most violent time in human history, we accepted any beggars that came to our front door. Now we live in the safest time in human history and were so terrified of the masses that we disempower them completely
- kerblang 2mo agoI would be more inclined to agree if American companies were not in a competition to be the biggest flameout and lose the most money. How is China able to afford to train models that US companies need to burn billions of dollars for? In general, if American capitalism weren't so broken I would advocate for it as the strongest option, but American corporations are dead set on destroying it. The US may end up as "democratic socialist" if there is no capitalist reform.
- Avicebron 2mo ago[dead]
- caspianmagnus 2mo ago[flagged]
- redchaosgoddess 2mo ago[flagged]
- deleted 2mo ago[deleted]
- alexashka 2mo agoThe 'arguments' are not meant to persuade, they are meant to establish and entrench an overton window (regarding 'AI' in this case).
- okzgn 2mo agoFrom a neutral standpoint, governments usually have more general, comprehensive analytics, as well as greater context and insight into the risks of AI usage, and a possibly better methodology to control access to or use of open-source AI (perhaps by analyzing digital activities and consequences), because they learn from failures. On the other hand, open source creates major opportunities, and it’s hard to accept bans on something that truly benefits users acting for the common good. However, it also benefits unethical or opportunistic users—for whom regulation is indeed necessary.
- dualvariable 2mo ago> One probable outcome of an open-weight-model-dominant world is full AI communism... oh no, not communism.
- whack 2mo agoThe most compelling argument against Open Source AI: it is analogous to "open sourcing" nuclear weapon technology. Or an how-to guide for making meth/bombs/bioweapons. There is a high risk that we will one day achieve ASI or something close to it. When that day nears, if even one open-source AI user handles their AI in an irresponsible way, we are all FUBARed. In the same way that one irresponsible person with a nuclear bomb can FUBAR the entire world, hence why "open sourcing nuclear weapon tech" is not even a subject of debate. You don't have to believe that ASI is 100% likely. People like Hinton believe there's a 50% chance of existential risk from AI, but you don't have to go that far either. If there's even a tiny chance of ASI posing a threat to humanity, that's reason enough to lock it down
- potsandpans 2mo agoThis argumentation is going to be joked about in the future. Like downloading a car. Would you open source a nuclear bomb???
- danny_codes 2mo agoThe difference is that people building atomic weapons knew what they were building. OpenAI/Anthropic have no idea how emergent intelligence works. They're just running in the dark towards the cliff. Leaving it in their hands is the same risk as leaving it to any random guy, because random guy and the labs have the same understanding of what they're building. I'm tired of us treating labs like they're somehow uniquely good at AI. They are not. They simply have enough capital to train. If training was affordable they would have no edge whatsoever.
- whack 2mo agoI'm not claiming that OpenAI is worthy of trust. The safest course of action is to either stop all AI research immediately, or have it be highly regulated by an international body. The most dangerous course of action is to give every single individual the ability to create their own variant of an ASI
- newsomix9xl 2mo agoJust because encryption bans are badly conceived or implemented doesn't mean they are wrong. Encryption has military value. The point is that the good outweighs the bad. Making the government seem like buffoons for attempting to prevent military technology like dual use crypto is not in our best interest. The same government upheld the right to free speech, so this balancing act is widely observed. The government is not evil, and yes they make mistakes, but they also not infrequently protect us. And I, for one, like being protected.
- js8 2mo agoRegarding encryption. Interestingly, US government also insisted on DES and AES being open standards, because using bad encryption did more harm than good.
- aarondong 2mo agoLLMs are trained on public data (as well as illegally obtained data see: Anthropic 1.5B settlement). LLMs are nothing without the huge corpus of human data that powers them. There is an argument that research of this kind should be restricted to governments and regulated universities rather than opaque public companies with competing incentives. Or research should be stewarded by genuine non-profit collectives with democratic leadership. e.g. like internet standards, telecom, etc I do not think we can trust private companies, no matter the virtue signaling they put forth into the world, to effectively regulate themselves and inform the public and scientific communities about risks. Their ongoing conflict of interest poses serious credibility risks.
- ozgung 2mo ago> OpenAI's Dean Ball: > One probable outcome of an open-weight-model-dominant world is full AI communism... This quote proves me that there are no rational arguments here. It’s just Cold War era mentality. This is the first time US is really challenged in technological dominance since the Soviets. Sadly this seems like the only playbook Americans still use. This is their modus operandi. This is how they handle competition and they haven’t updated their playbook since 80s. There is nothing necessarily malicious about open weight models. Being “Chinese” is the real thing which makes them “malicious”. It seems public still buys this rhetoric. And leaders and policy makers still use it because they don’t know any better. But I have to say outside US it makes no sense in 2026’s world.
- aaroninsf 2mo agoTom doesn't take up the single argument that is of primary concern to me, which is that capable models running in environments where they can be and regularly are abliterated, and as a society and civilization, we have no good answer to what to do about the destabilizing threats this poses. I don't have an answer. And I'm not arguing in favor of some cynical solution which distills down to the state defending oligopolies. But the threats are real and we have to reason about them soberly, widely, loudly, and quickly. Most readers here know that defense is always at a disadvantage when it comes to security. That is true to an extent hard to fathom given the innumerable dimensions along which an antisocial actor can apply the force-multiplication of capable models to ill ends. It doesn't have to be FUD-adjacent concerns such as biological terrorism and cybersecurity, though these are real; the opportunities for mischief and misadventure are endless and the opportunities for poisoning the body politic or bringing down basic infrastructure many. What then is to be done...? I don't know; but I do know that the argument that almost all use of open models poses no threat is insufficient. We have not reckoned as a culture with force-multiplication such as this technology brings. We need to get ahead of the curve, and that will require making novel hard and unwelcome decisions in the short term. The IMO inevitable outcome otherwise is to rue in leisure, after some global shock, assuming we have opportunity to.
- James333i 2mo agoI have shipped open weight models in iOS apps and agree with this in theory. A model is just one (significant) piece of the stack though. Prompting, adjustable parameters, tooling, workflows, and interfaces are what make a product usable. Black boxes, open and commercial, sit in most of the products people trust today. As for the backdoor section, inspecting weights is not the same as auditing training. You can examine behavior and strip guardrails with fine tuning but you cannot determine from the weights what the model was trained on or whether the data was spiked. That can be a real problem for some use cases. Closed models are far more opaque but you are buying in to a contract and accountability in exchange for the lack of transparency.