9 ms·

Civic Hygiene – avoid building technologies that could be used by a police state (2013)

455 points by felineflock 28d ago

ghoul2 27d ago
Isn't it an age old maxim: democratic governments should remember to not build tools/give them selves powers that they won't want the opposition to be able to use WHEN they come to power.
ck2 28d ago
the problem is Capitalism does not need Democracy to function or survive many countries like Singapore, etc. prove this easily Authoritarianism can even be desired by Capitalists if they know who to bribe to remove regulations and get approvals and no surprise, that's exactly where the United States is right now, teetering on the edge of that hole, about to fall in "AI" is going to be weaponized, already is, and any resistance against its abuse is simply overriden with threats to destroy profits or even company survival at all
- nemomarx 28d ago
  well, be glad it's company survival and not personal survival yet. We had that article about the men with guns in egypt recently too
  - stpdnerdism 28d ago
    There is some sort of "nerd naivete" that they are the ones with power to control those things. If you're a nerd developing open source (not an insult; I'm in that category), our job is to develop effective technology. If you want to do politics, go into law and/or politics (or into business, so you can fund/control politics). >"But software which OpenBSD uses and redistributes must be free to all (be they people or companies), for any purpose they wish to use it, including modification, use, peeing on, or even integration into baby mulching machines or atomic bombs to be dropped on Australia. " -- Theo De Raadt
- jsbg 28d ago
  > many countries like Singapore, etc. prove this easily Economic freedom is a necessary but not sufficient condition for political freedom.
itsanaccount 28d ago
I think this runs into a basic issue that all large software systems that enforce the law contribute to locking down a society because laws tend to need be broken to be changed. That a system can be avoided offends the investor who want their products to be ubiquitous. The system that is not perfectly enforced offends the engineer who dislikes broken things. Yet both are required if we as a society are to grow. I think the thing that it might be good to learn is when to say "I've done/had enough."
- philipallstar 28d ago
  > because laws tend to need be broken to be changed Nope.
  - itsanaccount 28d ago
    I think I could collect enough weight of books on "civil disobedience" at my local library that I could crush you underneath them, even if you didn't bother reading them.
    - philipallstar 28d ago
      I'm sure you could. But it would in no way demonstrate that laws generally need civil disobedience to be changed. Laws are changed all the time, as bills, as legal precedent, and as regulations.
gmerc 28d ago
Harder than it looks. Everything is dual use. Certainly didn't expect my game development telemetry[1] system used to improve artillery [2]... [1] https://gdcvault.com/play/1012227/Development-Telemetry-in-Video-Games https://gdcvault.com/play/1012227/Development-Telemetry-in-V... [2] https://www.army.mil/article/85934/solution_to_weapon_development_problem_halo_mass_effect https://www.army.mil/article/85934/solution_to_weapon_develo...
- rasengan 28d ago
  I feel really bad for you. It might actually be wise [1] to create a new, slightly less permissioned license for open source wherein the open is defined as anything that isn’t in a kill or surveillance chain or, more broadly, for military use… [1] for the community
  - deleted 28d ago
    [deleted]
  - stpdnerdism 28d ago
    I doubt this is enforceable. Are you going to sue a government secret program because they used some open source code in a way that violates some weird clause?
    - gentooflux 28d ago
      Yes, citizens should sue the government when the government takes actions which are grounds for suit. The government sure will come after your money if you do things they say you shouldn't, and what's good for the goose is good for the gander.
      - well_ackshually 28d ago
        You won't know, they'll say it's for national security so licenses don't matter, and then if you keep going they'll make sure your life is a living hell and your bank accounts closed. I'm begging tech bros to understand that abusive governments _do not care about abusing you_
limagnolia 28d ago
The automobile, the transistor, the Internet, and basically every other technology ~could~ be used by a police state. I'm not sure how that is a helpful rule at all.
- gumby271 28d ago
  The article says the same. I think the point is to be aware and try to build things in a way to minimize the chance for abuse. There's nuance and the author says as much.
  - stephbook 28d ago
    "Should we have a national database of, say, religious beliefs? Almost instinctively the answer is no." What about an ad system that can predict beliefs and home addresses? Ooops. Google, Microsoft, Apple, every telecoms provider, every modern car maker..
    - Ey7NFZ3P0nzAe 28d ago
      It's definitely different to have a prediction than a database. Especially for something that can be denied like religious belief. If the nazis had such a db they would have killed much more efficiently I think.
  - wang_li 27d ago
    Technology isn’t going to be a moderator of a police state. We already have technologies that are used for oppression, exploitation, abuse, and so on. They’re just not being used this way by governments in the same way they are being used by criminals. Bitcoin is used to collect ransoms. Telegram/signal is used to plan conspiracies to commit crimes. GPS and location services are used to monitor trafficking victims. There are stories of cartels running their own cellular networks. A lack of technology doesn’t restrict a police state, see East Germany in the ‘60s-‘80s. It does, however, create more opportunity for crimes. In Seattle there was a shooting at the bite of Seattle festival. They don’t have firm ideas about the identities of all the suspects. There would be a greater chance that they would know who they are looking for if there were more securities cameras deployed at large public festivals. To prevent a police state you need public participation in governance and a population that lives in the real world and not fantasy lands created by a intentional misinterpretation of events.
blfr 28d ago
Libertarianism has its moments but it generally hasn't caught on because state capacity is unique and useful. And virtually every last bit of it can be used to facilitate a police state because it's... a state. Sure, we could all live without IDs and use motio belli for defence but this arrangement will be soon replaced by someone more organized because there is a huge return to that organization despite its risks.
inferniac 28d ago
What is the point of writing something like this? It's embarassing for an adult to think like this. There will always be someone willing to build that thing (and that was true even before llms)
- Georgelemental 28d ago
  > There will always be someone willing to build that thing … with sufficient time and money. As opposed to the minimal time and money required if the baddies are just pulling an existing solution off the shelf.
- ForHackernews 28d ago
  Speaking of embarrassing, your argument is an argument against having any professional ethics whatsoever.
- reaperducer 28d ago
  It's embarassing for an adult to think like this. Joe's gonna die anyway. I might as well kill him before someone else does. It's embarassing for an adult to think like this.
- zero-sharp 28d ago
  What's embarrassing about it?
- victorbjorklund 28d ago
  Terrible argument. You can say ”why not work as a concentration camp guard? There will always be someone willing to do it so you might as well cash in”
  - wartywhoa23 28d ago
    At Nuremberg, they all were just doing their job and executed their orders, those eager beavers of men.
- no_multitudes 28d ago
  I hope you are joking, or haven't thought this through! If you sincerely believe and act on what you're saying here, you should be shunned from polite society.
tptacek 28d ago
Has this philosophical approach ever worked? It's a commonplace in the hard sciences and engineering fields (ie, refusing to work on weapons systems), but weapons systems are elite and well-compensated engineering projects that don't have any trouble finding talent. The same thing goes for NSA, which has a line out the door from elite academia. I'm not dismissing the sentiment here. It's morally sound. You should act in accordance with your principles and it useful to be reminded how those principles intersect the world. I have similar guardrails (probably not the same as the author's). I just wouldn't want people to overestimate what can be accomplished with this approach.
- m-ee 28d ago
  A significant number of people in positions of power in the AI industry get their moral framework from science fiction books and a particular Harry Potter fanfiction. It’s hard to predict what ideas and philosophies will become important.
  - Hugsbox 28d ago
    Sorry, purely just curious here because I'm not sure what you mean - what Harry Potter fanfiction?
    - krapp 28d ago
      "Harry Potter and the Methods of Rationality" written by Eliezer Yudkowsky has become something akin to a sacred text among the rationalist community. edit: posted is a podcast by Behind the Bastards about the Zizians, a death cult that branched out of rationalist community and which was responsible for six deaths, but it covers the rationalist community and the Harry Potter fanfiction as well. [0]https://www.youtube.com/watch?v=9mJAerUL-7w https://www.youtube.com/watch?v=9mJAerUL-7w
      - Hugsbox 28d ago
        Sweet, I love Behind the Bastards, thanks for providing the entertainment for my next few walks :)
bad_haircut72 28d ago
Recently my wife got locked out of her bank account - all cards blocked, unable to log in to the website or app, so effectively her whole finances were frozen - because she was trying to order a new bank card over the phone and some system couldnt do voice recognition (which we did not know they even used). We were getting ready to get on a plane to NYC, from California (she banks with TD bank) but after a tearful session of phoning anyone and everyone we could at TD, suddenly it magically got reversed. The problem is not just the tech, its building systems where humans are essentially slaves to the system, with no recourse or power over whats happening.
- SoftTalker 28d ago
  Few things are more important in modern life than access to your money. That's why I would never use an online bank. I use a local credit union but "local" with actual branch offices you can walk into and talk to a human are the important thing.
  - balderdash 28d ago
    TD bank has branches but that's not super helpful if you're at the airport. the helpful thing is having a "relationship manager", someone you can call or email directly - you can get that at a big bank or a small one, and you don't need tons of assets, a lot of banks its having your direct deposit with them is enough to qualify for some special relationship.
    - preg_match 28d ago
      Online banks typically offer much more competitive financial products. I have some HYSA accounts at 4% APY, that’s not achievable with a CU or traditional bank. The solution is the same solution as any financial risk: diversification. I have many accounts at many banks. If any one of them goes down, I’m fine. If multiple go down, I’m still fine in the interim.
      - SoftTalker 28d ago
roughly 28d ago
The other rule here is that you cannot avoid politics if you want to create a better world. If you want the freedom to develop technology without worrying it will be used by a police state, you need to put in the work to make sure a police state doesn't happen - you cannot ignore politics and just "focus on technology."
- api 28d ago
  There's a deep failed idea here. It's the idea that technology is an alternative to politics, period. I think that whole idea has failed. You can't avoid building technologies that could be used by a police state, because literally everything can be used for warfare or aggression. The first person who sharpened a stick or a rock created a weapon. Everything is a weapon. Everything is dual-use. If you want to prevent a police state or any other undesirable political outcome, you have to confront the politics directly. "Change the tools and you change the rules," an old saying from 1990s cyber-culture, is an idea that I think has failed. Instead what happens is that the rules determine what's done with the tools. Those 90s cyber-hippie and cyber-libertarian types vastly underestimated the power of the political to the point of making a category error. Cryptocurrency is a great case in point. It started as an attempt to create a technology that would act as an escape hatch from corrupt casino capitalism and state-backed financialization. Instead it became an insane absurdist parody of those things. The same types of corrupt individuals who profited off "too big to fail" and the rest of it simply took cryptocurrency and used it to implement the same type of system but on a block chain. The rules (and rulers) changed the tools, not the other way around.
  - cucumber3732842 28d ago
    I mean you kind of can. The whole premise of AI is basically that we'll make white collar administrative compliance work dirt cheap in the long run and we'll all be richer for it and we'll sidestep the whole "fight with all those industries" step because the sheer economics of it will make it happen and as those industries become cheaper/poorer there will be even less reason to not kick them to the curb. Kind of like how once certain state shed a lot of their manufacturing or agriculture, adopting policy that's bad for the rest became easier. Or how now that solar is cheap we mostly DGAF about coal.
xandrius 28d ago
I'd be careful banging two rocks together else someone might start a war with them!
gumby271 28d ago
I've been arguing with folks on Reddit about this lately when it comes to Google's changes to Android (where they've made it more difficult to install any software on your device without Google first tracking the identity of the developer). My argument ultimately comes down to the fact that Google (and Apple already) have built a technology that gives them immense power to control individual computing freedoms, and no matter their internal policy, they still must follow US law. No amount of promises or legit good faith on their part matter, they've made a dangerous technology and are behold to a government that is very eager to censor anything it doesn't like. It's so disappointing to see 1. Google engineers build such a thing, and 2. For people to support it.
stpdnerdism 28d ago
The article is not completely wrong in its premises, but phrased incorrectly. This is not about technologies (after all, an authoritarian government will use many existing technologies, from the car to the computer), but about powers granted to a government, that could be used differently by the next party/majority in power. The answer to that is to not grant government too much power in the first place. Of course, this is often a contentious point for leftists, since they do want to grant government a lot of power in the first place; they're just concerned that the exact same powers will then be used by a different political side in a way that bothers a different category of people.
- cupcakecommons 28d ago
  You will get downvoted to oblivion for even suggesting it here but it's so obviously a take that maps on cleanly to reality. Massive power centralization in government makes everything dual use in the service of a police state. The easiest thing to do is resist massive centralization of power in a government.
  - itsanaccount 28d ago
    you get downvoted because to most people we don't care if its tyranny.gov or tyranny.com. "don't give powers to the government" just means they buy them from corporations. see: cellphone location data. the solution is not to have the power exist at all, anywhere.
    - mrguyorama 28d ago
      >the solution is not to have the power exist at all, anywhere. This is nonsensical. The power always exists, it's just your fellow human beings doing things, to you, around you, with you, without you. You cannot do anything to not have that power. It doesn't matter if your neighbors are lynching you because the evil dictatorial police state said to, or because the rough consensus of a mob of lunatics want your food. You as a human lack power. Humans working together have far more power. Sometimes, you can use the combined power of some humans working together to empower other humans. That's what laws are. That's what a State can do. That's why corporations can have power even though they are not part of government. Being able to direct 1000 humans to do something is what power is. You cannot eliminate that power. It's a fundamental part of reality. It's the entire reason animals evolved to be social at all, to take advantage of that power.
ReptileMan 28d ago
Technology is force multiplier. Everything could be used for oppression. Deal with it - the future is boot stomping on human face forever. There are lucky few that could choose whether they will be the face or foot. But it is inevitable. Or we could stop all tech development. But it is not going to happen.
AndrewKemendo 28d ago
Bertrand Russell: “If war is not impossible, then every advance in scientific technique in an advance in mass murder” As true today as then You can replace war with “police state” and it’s the same message. https://m.youtube.com/watch?v=GzTlTVaYDtk https://m.youtube.com/watch?v=GzTlTVaYDtk
naniel 28d ago
this is an important topic, and i hate that people forget how difficult it is. in any political conversation, it's pretty easy to pull this thread to unravel anyone's adamant belief in one thing or another (e.g. gun control). my biggest concern is that people seem less willing than before to actually have those discussions.. they'll immediately just label you as being opposed to their views, "against" them.. when really we need these conversations to find viable policy solutions that wont easily be used in the exact opposite way that they were intended when implemented
kingleopold 28d ago
how about you guys understand that state is criminal master org. that uses violence monopoly? is it too high level or hard to understand this? see war on drugs and more
josefritzishere 28d ago
"Civic hygiene isn't about saying we distrust our current government - it's about not trusting the next government." I've made similar points. Granting one government administration more power, is by extension granting it to all future administrations. Do so only with great care.
PaulRobinson 28d ago
This is missing the "(2013)" at the end of the HN link, but it does seem weirdly prescient as we say a rise of more polarised politics across the World, just 13 years later.
- gumby271 28d ago
  And a rise in tech companies building products and features that dont benefit the users, knowing we cant really escape at this point.
- NietTim 28d ago
  Thanks, didn't notice it before you pointed it out (and I already made my comment), actually turned into quite an interesting retrospective for me
slg 28d ago
I find it frusrating how the conversation around this almost always focuses on one side of the equation. Why is it always about preventing tools from being built in case authoritarians comes to power rather than preventing those authoritarians from coming to power in the first place? I'm far more scared off the authoritarian aspect than the tech. The Nazis didn't need Flock cameras to kill millions. It seems like focusing on the actual bad guys is going to be more effective because if the state wants to bear its weight on you, it rarely requires tech to do it.
- well_ackshually 28d ago
  >The Nazis didn't need Flock cameras to kill millions. No, but they were extremely happy to hop over to France and realize that we kept lists that indirectly listed jews. Actions have consequences that we may not be able to predict.
- wartywhoa23 28d ago
  The water flows whichever way there's a gradient of height. Don't create gradients you don't have dams for.
- tehlike 28d ago
  People have incentives to become more powerful - through wealth, power, leverage, or in other ways. So unfortunately, authoritarianism tends to have gravity going for it.
- ds_opseeker 28d ago
  The Nazis relied heavily on IBM technology for national census data (to identify jews), military logistics, ghetto statistics, train traffic management, and concentration camp capacity https://en.wikipedia.org/wiki/IBM_and_the_Holocaust https://en.wikipedia.org/wiki/IBM_and_the_Holocaust I fully agree with your base perspective, that the main focus should be on preventing authoritarians from coming to power in the first place. I also agree with your final point, that the government doesn't need tech to lean on a person. We disagree in the middle, on how tech that makes evil orders of magnitude easier might be a contributing factor and might enable the government to lean on many more people than it would otherwise.
deleted 28d ago
[deleted]
cupcakecommons 28d ago
Defaulting to discussing the US here: let's start with reducing the state's size - it shouldn't be the world's largest employer, it shouldn't be able to run an unfathomably large deficit, it shouldn't be able to declare war without congressional approval. Technology to run a police state is just the last step in a long line of ridiculously unacceptable steps.
- charlieyu1 28d ago
  It won’t work for any country. They all want more control, and also create a class of people depending on government money to be on their side. Only people losing out are the powerless ordinary citizens and taxpayers
- runtime_terror 28d ago
  How does the number of state employees de facto result in a police state? I agree with you on our obscene debt and our obsession with "conflicts" around the world, but I'd argue we're seeing the mass reduction of state employees and it's just making everything worse (eg massive deregulation and the destruction of checks and balances).
  - cupcakecommons 27d ago
    I in no way endorse taking people's jobs away once they have them. I'm saying the situation itself is absurd to begin with - now we are in a terrible situation where it can only be shrunk across a long timespan where people retire and orgs shrink naturally.
- genxy 28d ago
  Let's slip in your pet bugaboo. Workforce size very little to do with state surveillance. In fact, mandating that it shrink can result in blowback, just like we have seen with how the US provides a great antagonist for China to evolve its own capabilities faster than if we had done nothing.
  - cupcakecommons 27d ago
slipperybeluga 28d ago
[dead]
codeulike 28d ago
I'm in the UK and it always seems weird to me how in the USA its totally normal to publically register yourself as either a Democrat or a Republican. To me seems like the kind of thing that should be private. In the UK I might choose to be a member of a political party but no-one needs to know that except that particular party. Or I can just join no party and vote in private on polling day.
- Hugsbox 28d ago
  Because by and large they're all trapped in an "us vs them" team sport where both sides believe everybody on the opposing team is literally Satan.
  - frollogaston 28d ago
    No, it's because you need to register for a party if you want to vote in its primaries.
  - copper-float 28d ago
    I'm really hoping we can get over this silly delusion soon.
- deleted 28d ago
  [deleted]
- odiroot 28d ago
  Then again, EU citizens would find it questionable how in the UK recruiters are asking about your ethnicity, disability status etc.
  - frollogaston 28d ago
    UK also arrests you for boomer facebook memes. Then again maybe parts of EU too.
- bee_rider 28d ago
  It seems quite weird. Our parties are generally quite weird because were are not supposed to exist in the first place and so most of the rules are sort of ad-hoc. I don’t belong to one, but it is sort of annoying to not be able to vote in the “primary elections” (where the parties pick their candidates). My state has a strong partisan lean so this is the only election that actually matters here. Oh well.
krautburglar 28d ago
Assholes like this were all for the police state back when it was up to stop-the-spread “contact tracing”, real-time “fact-checking” my private messages, arresting people for facebook posts, and transing your kids. Now that the red pedophiles are in charge instead of the blue ones, he changes his tune.
NietTim 28d ago
Edit; wrote this comment before I realised this is a blog post from 2013. In 2013 I would've agreed 110% with this. ----------- This is one of those things that make for great post title and gets you all riled up at first but when you think about it, it's really shallow. Most civic databases can be abused by any sufficiently authoritarian government. A homeless shelter's records may contain exactly the sort of information that could be used to get "undesirables" off the street. But those same records are useful for coordinating beds, outreach, medical care, etc. Surely the conclusion isn't that the shelter shouldn't keep records. It seems much more useful to focus on limiting what gets collected, who gets access to it, and how long it can be retained. In other words, boring things like data-minimisation and retention laws. Somewhat amusingly, that gets you pretty close to the sort of principles behind GDPR, which isn't exactly universally loved around here. I don't really disagree with the blog post's moral intuition. What bothers me is the emphasis on individual moral choices by the people building these systems. I've seen plenty of my peers happily work for companies that have spent the last twenty years building exactly this kind of technology, there will always be someone willing to build it. That's not an argument for abandoning personal ethics, but it is an argument against treating personal ethics as any form of serious societal safeguard. "good engineers should refuse to build them" means nothing.
cynicalsecurity 28d ago
Companies that agree to work on Chat Control deserve a special place in hell. One company that was eager to participate, hoping this harmful law would pass, is Atos. Name and shame them.
seethishat 28d ago
People cite Flock as an example of this. Yet the same people walk around with iPhones and Android devices that can record conversations (who talks to whom at what times and for how long), know our exact locations and log everything we do online. If I had told you 30 years ago that we are going to build a surveillance network and make the people we are tracking pay for it all you would have laughed me out of the room. But that is what we have built. The modern Smart Phone. Now we use our iPhones to complain about cameras on public roads as if they are watching us.
- NietTim 28d ago
  Very flawed argument, anyone can go for a walk without their phone on them and have a conversation, nobody would ask them why they don't have their phone on them.
  - seethishat 28d ago
    I know it is not a popular opinion. But it is my opinion. Do you seriously think that governments need Flock cameras when they have Apple, Microsoft and Google?
    - NietTim 28d ago
      Governments don't need flock or AMG, that's where the flaw in your argument begins.
    - runtime_terror 28d ago
      Ostensibly they need a warrant to get that data from Apple/Google whereas they have a direct line to Flock data cause that's how that relationship works
    - RankingMember 28d ago
      The government needs a warrant to access Apple/Google/Microsoft's data.
- runtime_terror 28d ago
piloto_ciego 28d ago
Or, and here me out… “Become ungovernable.”
- 1-6 28d ago
  That has been the motto of our governors.
- dennis_jeeves2 28d ago
  This is the most correct response.
rini17 28d ago
The sad irony of "Share this post on..." with neat list of all the services doing exactly what the post warns against. They can and often are already used by police state.
not_wyoming 28d ago
This entire thread is such a strong case for why all undergrad CS programs should have a core requirement of at least one ethics course.
- Terr_ 28d ago
  Especially if it has concrete historical examples of technology intended for good and how it was later used for evil, rather than something more generally-philosophical.
- SoftTalker 28d ago
  Every business school and MBA program has ethics course requirements. Does it really make a difference? Maybe it does, and things would be even worse than they are otherwise. But I'm not sure.
  - asdfman123 28d ago
    I think the purpose isn't to make people more ethical, which is impossible, but to make ethical people more effective. It ideally makes people better at identifying when something bad is happening and gives them tools to handle it. If they're amoral in the first place though a course isn't going to change that.
  - m_dupont 28d ago
    I think of it like so: All the great people in history who would be regarded as ethical never took an ethics course. Its hard to know whether an ethics course makes people behave ethically, but its clearly not a necessary requirement to ethical behaviour
    - SoftTalker 28d ago
      Many of them were probably religious though, which is basically ethics with a lot of supernatural trappings. Although lot of terrible people in history were likely at least outwardly religious, so not sure that proves anything one way or the other.
- 28d ago
_doctor_love 28d ago
Like others have said, I don't think it's possible to create technology that cannot be weaponized. I think it's possible (mostly) to design technology so that abuse requires more people, requires the leaving of evidence / paper trail, needs checks, etc. But still, it's not possible to prevent abuse altogether and sometimes we even applaud breaking the rules (the movie Hunt for Red October for example kicks off with the Soviet captain jumping the guardrails). Also, the most "benign" public infrastructure is usually the most impactful, e.g., water systems and electrical systems. Don't like the people in that one town who keep protesting and voting against you? Turn off their power, water, and also make it so that their trash doesn't get picked up for a year. See how they like it.
deleted 28d ago
[deleted]
jimmyjazz14 28d ago
Wouldn't a better title be "don't give out information that you wouldn't want used by bad actors"? Seems less about technology and more about privacy.
ahartmetz 28d ago
Pinging the absolute jerkwads at amutable.com who state that it's not their problem how their remote attestation technology will be used.
deleted 28d ago
[deleted]
closeparen 28d ago
The obvious 21st century United States instance of "don't collect data that the government might later use for persecution" is E-Verify.
panny 28d ago
Civic Hygiene is the opposite of how you were taught to write code... >It should be noted that no ethically-trained software engineer would ever consent to write a DestroyBaghdad procedure. Basic professional ethics would instead require him to write a DestroyCity procedure, to which Baghdad could be given as a parameter. https://blog.codinghorror.com/to-serve-man-with-software/ https://blog.codinghorror.com/to-serve-man-with-software/
tshaddox 28d ago
Clearly you need to consider how general the capabilities and use cases of the technology is. The most general-purpose technologies will always have obvious use cases for police states and other bad groups or people.
SillyUsername 28d ago
As a recruiter once said to me, if you don't do the defence work, somebody else will. So you might as well do it and do a better job and get paid well. I didn't go for the job.
- applfanboysbgon 28d ago
  If you don't man the concentration camps, somebody else will. So you might as well do it and do a better job and get paid well.
metalsiliconYT 28d ago
No electricity yea! No internet, no food! No Water!
- sebstefan 28d ago
  The impossibility of drawing a line! So difficult! So nebulous! I will work for a concentration camp
quantum_state 28d ago
The concept is like the perpetual machine in the 1800s. It is simply an impossibility. Please avoid wasting time.
Xeoncross 28d ago
I'm a tech worker. I'm given a promotion for adding invasive features based on "insights". I'm incentivized to create something patent-able to restrict competition. I'm rewarded for more detailed metrics to reveal more. I'm hired by how many people were affected by my previous work regardless of ethics. There is no reward for protection, privacy, and restricting access to data.
dvrp 28d ago
What about fire?
nullorempty 28d ago
Your taxes are building police state for you. What's the recourse here?
arjie 28d ago
That just moves the decision point to “could be used by a police state”. Some people prefer humans as being the only people who can determine what a civil violation is. E.g. if a policeman doesn’t pull you over you shouldn’t be punished. This allows for great personal discretion and the potential for a police state through such discretionary exertion of power as exemplified by the famous line > “For my friends, everything. For my enemies, the law” I prefer speed cameras, red-light cameras, and so on because automated enforcement removes discretionary power from enforcement agents. I believe that enforcement agents are the arms of the civic body tasked with ensuring the commons are not ruined. Consequently, controlling the exertion of discretion is entirely an engineering approach to controlling quality of enforcement. Others believe that enforcement agents are our rulers or agents thereof that we must submit to - and therefore to whom we must give great discretion so that they can choose whether we are worthy or not. That is incompatible with all being equal before the law.
- goda90 28d ago
  > I prefer speed cameras, red-light cameras, and so on because automated enforcement removes discretionary power from enforcement agents. There can still be corruption with automated enforcement. If the punishment is fees and the fees go towards the budget of those in charge of the automation parameters, then you'll see manipulation of parameters to increase fees paid. Shorter yellow lights, shorter distance between a speed limit sign and the speed camera, requiring more effort to contest the automation when it's wrong than to just pay the fee, etc.
  - arjie 28d ago
    This is true. A useful difference between this form of parameter optimization and the unalignability of humans is that it is possible to force constraints on these parameters. E.g. San Francisco’s speed cameras run in collection mode to obtain baselines and then in enforcement mode where they have to demonstrate reduction in speeding. California mandates minimum yellow times. Flaws exist but we can constraint-engineer to minimize adversarial optimization of parameters. Improvements allow us to asymptotically approach our goal optimum. Human discretion, on the other hand, is fluid and dodges all attempts at enforcement. We require many human agents solely because transforming unstructured vision space into structured judgment was impossible in the past. Currently, in a sufficiently constrained environment (roadways) we are able to solve sufficiently constrained problems (speeding, running a red light). One hopes that we are able to solve this problem in more spaces as time goes by.
salynchnew 28d ago
https://neveragain.tech/ https://neveragain.tech/ is still up.
arionhardison 28d ago
About 15 years ago I was locked up in SF (850 Bryant/HoJ 7th floor) and everyone was talking about their "jobs"; like what they did for a living outside of the obvious. I said "I fuck with computers and shit" and I got some follow-up questions I did not expect like - "wtf you doing here", they were all along those lines. I been locked up plenty of times but this was the 1st time I was publicly ridiculed for it. This guy named Vic; big ass black dude with an extremely visible front tooth missing said: "Man, if I could do what you do I would rebuild the Gov." What followed was a discussion where everyone in the pod which not only showed interest and consensus but an abject disappointment in me and my choices so I said jokingly, well OK; if I am ever locked up w/ my computer I'll rebuild the Gov. Then Covid hit, i got fired from Goop and I found myself essentially "locked up w/ my computer". Here on HN; this is NOT a popular sentiment and I know it will garner nothing but admonishment in the form of downvotes and ridicule but I have really been able to help some people w/ housing, re-entry, healthcare, addiction etc... so fuck it. Everyone in that pod agreed the Gov was broken. Nobody could put their finger on it or articulate in what AI might deem a "load bearing" - HHAHAH capacity. The other day i saw a comment on here by bob1029 [1] and he put it better than I ever could. It's broken because nobody can execute across all the moving parts; its a coordination issue in many ways - the cost of collaboration has become so large that it has become the thing to overcome instead of whatever first principal is the whole point of the shit e.g.: Healthcare, Food, Housing etc... The bottleneck was never intelligence it's coordination. The cost of getting many parties to act together without the whole thing blowing up. The Anthropic multi-agent article hit it too...turf war, the 2.4M-request queue, the price collusion, the merge silos. Every one is the same thing: free-form runtime "negotiation" blowing up. And every success is an agent ad-hoc reinventing an institution mid-fight inventing a "verifiable criterion," writing an apology commit and pulling in a human, running a tournament as a commitment device. They keep hand-rolling the institution, badly, under fire. So stop making them reinvent it. The generalization of the "constrained action space" idea isn't just bounding one agent's moves it's what I call an intent: intents don't just constrain each agent's actions, they codify the coordination itself. And I make the unit of coordination a first-class object: the Deal a structured negotiation instead of a free-form one. The Deal is the container those agents keep reinventing, handed to them up front: scope and terms, the parties and their roles, the ordered steps, a verification gate as the neutral arbiter, and an appeal / human-escalation path as recourse, all over an append-only record. Agents negotiate inside a bounded contract instead of improvising one mid-fight. How it works from what you say to what actually happens (the "Deal") 1. You speak the intent. You can use a form but I found people like the verbal shit much more. 2. Break the problem into an ontology. The ask gets decomposed into the real work it implies mapped to actual job roles (O*NET). Problem → Solution. 3. Vet the solution with the gov council agents. Every domain has one public AI expert in that agency's real systems Medicare, Medicaid, SSA, OPM (pda:codify.<domain>). Policies get deliberated and voted, elected officials in the loop, before anything runs. Stakeholders. 4. We staff it. It matches the specific expertise needed the exact specialty, not a generic "doctor." Down to the right role for your case. 5. We set up the financing. The money that follows the person eligibility, benefits, the payment rail. Financing. 6. An agent manages the process with expertise humans in the loop for anything high-risk, adjusting as outcomes come back, like an adaptive trial, until the outcome is met or it fails honestly. Expertise → Outcome. The atomic unit of value here is a "Deal". Problem broken down, solution vetted, staffed, financed, executed under contract. intent → Deal → outcome. Overview - Intent: https://wiki.erpo.law/agent/wa-erpo https://wiki.erpo.law/agent/wa-erpo - Program: https://erpo.law/program/71069-petition-for-an-erpo-as-a-family-or-household-member https://erpo.law/program/71069-petition-for-an-erpo-as-a-fam... - Protocol: https://erpo.law/protocol/71079 https://erpo.law/protocol/71079 I'm building it in the open because I think thats the only way this can be done https://medicare.dev https://medicare.dev https://epa.dev https://epa.dev etc... Learning how to take criticism better so the more criticism the better. [1] - https://news.ycombinator.com/item?id=49316271 https://news.ycombinator.com/item?id=49316271
gagan2020 28d ago
They all are created by US companies only. FB, Instagram, Youtube, X, Google (Android & others), Apple, etc. Face Recognition, Drones, space exploration (engines for missiles), AI, LLMs, etc. So, Either do it yourself (China) or get run by them (India).
bronlund 28d ago
I think it’s the other way around. We are from time to time graced with scraps from military research and we think it’s neat.
graemep 28d ago
[dead]
gos9 28d ago
Is a prosecution fantasy comorbid with sexual deviance?
docmars 28d ago
But only if it isn't Republicans, right? The Biden administration sure loved partnering with private tech companies to censor and monitor Americans to fight imaginary bogeymen like those darned evil white people who want their own nation like everyone else! (gasp!)
AuthAuth 28d ago
If your government has elected someone bad its already over. So much effort is put into trying to restrict what the highest point of power may do. But its the highest point of power and if its compromised you've already lost. See trump breaking all the checks and balances. Its better to build what you need and focus on getting good people into power. You shouldnt be able to hide from your government.
ninkendo 28d ago
Title was mangled. The actual quote is: > It's bad civic hygiene to build technologies that could someday be used to facilitate a police state Not “avoid building technologies that could be used by a police state”, as the HN title currently says. The HN title reads like any technology a police state could ever use should be avoided. Like cars, or pencils. Anything. Schneier’s quote says “used to facilitate a police state” is a lot more specific: could be used to bring a police state into existence where it wouldn’t have been possible before, like having a database of everyone’s sexuality, etc.
cpill 28d ago
just make guns that on l can only shoot the baddies! easy!
p0w3n3d 28d ago
In my country there are two parties always being elected. Quite similar to US system. It's always first saying "elect us, otherwise second will win" and the second saying the opposite. Asymptotically those parties go the same way, but always introducing new laws that improve oppression-style control, explaining that those others must be controlled. And the voters accept the solution, but do not understand that the next year the second party will come and use the solution. I don't know if you get what I'm saying without the context but this is exactly the case: imagine police state appearing in your country. Does the law help them or are you still free
pyaamb 27d ago
Does explicit licensing and the threat of a lawsuit related to violation of it not help here?
- M95D 27d ago
  Lawsuit against a tirant? Who would dare?
  - pyaamb 27d ago
    I guess it falls on how the courts see it. I'm sure the app devs would love to take their money.
    - M95D 27d ago
      What courts? You think you can sue someone like Putin or Kim and ever see the light of day again?

455 points by felineflock 28d ago

Isn't it an age old maxim: democratic governments should remember to not build tools/give them selves powers that they won't want the opposition to be able to use WHEN they come to power.

ck2 28d ago

the problem is Capitalism does not need Democracy to function or survive many countries like Singapore, etc. prove this easily Authoritarianism can even be desired by Capitalists if they know who to bribe to remove regulations and get approvals and no surprise, that's exactly where the United States is right now, teetering on the edge of that hole, about to fall in "AI" is going to be weaponized, already is, and any resistance against its abuse is simply overriden with threats to destroy profits or even company survival at all

nemomarx 28d ago
well, be glad it's company survival and not personal survival yet. We had that article about the men with guns in egypt recently too
- stpdnerdism 28d ago
  There is some sort of "nerd naivete" that they are the ones with power to control those things. If you're a nerd developing open source (not an insult; I'm in that category), our job is to develop effective technology. If you want to do politics, go into law and/or politics (or into business, so you can fund/control politics). >"But software which OpenBSD uses and redistributes must be free to all (be they people or companies), for any purpose they wish to use it, including modification, use, peeing on, or even integration into baby mulching machines or atomic bombs to be dropped on Australia. " -- Theo De Raadt
jsbg 28d ago
> many countries like Singapore, etc. prove this easily Economic freedom is a necessary but not sufficient condition for political freedom.

itsanaccount 28d ago

I think this runs into a basic issue that all large software systems that enforce the law contribute to locking down a society because laws tend to need be broken to be changed. That a system can be avoided offends the investor who want their products to be ubiquitous. The system that is not perfectly enforced offends the engineer who dislikes broken things. Yet both are required if we as a society are to grow. I think the thing that it might be good to learn is when to say "I've done/had enough."

philipallstar 28d ago
> because laws tend to need be broken to be changed Nope.
- itsanaccount 28d ago
  I think I could collect enough weight of books on "civil disobedience" at my local library that I could crush you underneath them, even if you didn't bother reading them.
  - philipallstar 28d ago
    I'm sure you could. But it would in no way demonstrate that laws generally need civil disobedience to be changed. Laws are changed all the time, as bills, as legal precedent, and as regulations.

gmerc 28d ago

Harder than it looks. Everything is dual use. Certainly didn't expect my game development telemetry[1] system used to improve artillery [2]... [1] https://gdcvault.com/play/1012227/Development-Telemetry-in-Video-Games https://gdcvault.com/play/1012227/Development-Telemetry-in-V... [2] https://www.army.mil/article/85934/solution_to_weapon_development_problem_halo_mass_effect https://www.army.mil/article/85934/solution_to_weapon_develo...

rasengan 28d ago
I feel really bad for you. It might actually be wise [1] to create a new, slightly less permissioned license for open source wherein the open is defined as anything that isn’t in a kill or surveillance chain or, more broadly, for military use… [1] for the community
- deleted 28d ago
  [deleted]
- stpdnerdism 28d ago
  I doubt this is enforceable. Are you going to sue a government secret program because they used some open source code in a way that violates some weird clause?
  - gentooflux 28d ago
    Yes, citizens should sue the government when the government takes actions which are grounds for suit. The government sure will come after your money if you do things they say you shouldn't, and what's good for the goose is good for the gander.
    - well_ackshually 28d ago
      You won't know, they'll say it's for national security so licenses don't matter, and then if you keep going they'll make sure your life is a living hell and your bank accounts closed. I'm begging tech bros to understand that abusive governments _do not care about abusing you_

limagnolia 28d ago

The automobile, the transistor, the Internet, and basically every other technology ~could~ be used by a police state. I'm not sure how that is a helpful rule at all.

gumby271 28d ago
The article says the same. I think the point is to be aware and try to build things in a way to minimize the chance for abuse. There's nuance and the author says as much.
- stephbook 28d ago
  "Should we have a national database of, say, religious beliefs? Almost instinctively the answer is no." What about an ad system that can predict beliefs and home addresses? Ooops. Google, Microsoft, Apple, every telecoms provider, every modern car maker..
  - Ey7NFZ3P0nzAe 28d ago
    It's definitely different to have a prediction than a database. Especially for something that can be denied like religious belief. If the nazis had such a db they would have killed much more efficiently I think.
- wang_li 27d ago
  Technology isn’t going to be a moderator of a police state. We already have technologies that are used for oppression, exploitation, abuse, and so on. They’re just not being used this way by governments in the same way they are being used by criminals. Bitcoin is used to collect ransoms. Telegram/signal is used to plan conspiracies to commit crimes. GPS and location services are used to monitor trafficking victims. There are stories of cartels running their own cellular networks. A lack of technology doesn’t restrict a police state, see East Germany in the ‘60s-‘80s. It does, however, create more opportunity for crimes. In Seattle there was a shooting at the bite of Seattle festival. They don’t have firm ideas about the identities of all the suspects. There would be a greater chance that they would know who they are looking for if there were more securities cameras deployed at large public festivals. To prevent a police state you need public participation in governance and a population that lives in the real world and not fantasy lands created by a intentional misinterpretation of events.

blfr 28d ago

Libertarianism has its moments but it generally hasn't caught on because state capacity is unique and useful. And virtually every last bit of it can be used to facilitate a police state because it's... a state. Sure, we could all live without IDs and use motio belli for defence but this arrangement will be soon replaced by someone more organized because there is a huge return to that organization despite its risks.

inferniac 28d ago

What is the point of writing something like this? It's embarassing for an adult to think like this. There will always be someone willing to build that thing (and that was true even before llms)

Georgelemental 28d ago
> There will always be someone willing to build that thing … with sufficient time and money. As opposed to the minimal time and money required if the baddies are just pulling an existing solution off the shelf.
ForHackernews 28d ago
Speaking of embarrassing, your argument is an argument against having any professional ethics whatsoever.
reaperducer 28d ago
It's embarassing for an adult to think like this. Joe's gonna die anyway. I might as well kill him before someone else does. It's embarassing for an adult to think like this.
zero-sharp 28d ago
What's embarrassing about it?
victorbjorklund 28d ago
Terrible argument. You can say ”why not work as a concentration camp guard? There will always be someone willing to do it so you might as well cash in”
- wartywhoa23 28d ago
  At Nuremberg, they all were just doing their job and executed their orders, those eager beavers of men.
no_multitudes 28d ago
I hope you are joking, or haven't thought this through! If you sincerely believe and act on what you're saying here, you should be shunned from polite society.

tptacek 28d ago

Has this philosophical approach ever worked? It's a commonplace in the hard sciences and engineering fields (ie, refusing to work on weapons systems), but weapons systems are elite and well-compensated engineering projects that don't have any trouble finding talent. The same thing goes for NSA, which has a line out the door from elite academia. I'm not dismissing the sentiment here. It's morally sound. You should act in accordance with your principles and it useful to be reminded how those principles intersect the world. I have similar guardrails (probably not the same as the author's). I just wouldn't want people to overestimate what can be accomplished with this approach.

m-ee 28d ago
A significant number of people in positions of power in the AI industry get their moral framework from science fiction books and a particular Harry Potter fanfiction. It’s hard to predict what ideas and philosophies will become important.
- Hugsbox 28d ago
  Sorry, purely just curious here because I'm not sure what you mean - what Harry Potter fanfiction?
  - krapp 28d ago
    "Harry Potter and the Methods of Rationality" written by Eliezer Yudkowsky has become something akin to a sacred text among the rationalist community. edit: posted is a podcast by Behind the Bastards about the Zizians, a death cult that branched out of rationalist community and which was responsible for six deaths, but it covers the rationalist community and the Harry Potter fanfiction as well. [0]https://www.youtube.com/watch?v=9mJAerUL-7w https://www.youtube.com/watch?v=9mJAerUL-7w
    - Hugsbox 28d ago
      Sweet, I love Behind the Bastards, thanks for providing the entertainment for my next few walks :)

bad_haircut72 28d ago

Recently my wife got locked out of her bank account - all cards blocked, unable to log in to the website or app, so effectively her whole finances were frozen - because she was trying to order a new bank card over the phone and some system couldnt do voice recognition (which we did not know they even used). We were getting ready to get on a plane to NYC, from California (she banks with TD bank) but after a tearful session of phoning anyone and everyone we could at TD, suddenly it magically got reversed. The problem is not just the tech, its building systems where humans are essentially slaves to the system, with no recourse or power over whats happening.

SoftTalker 28d ago
Few things are more important in modern life than access to your money. That's why I would never use an online bank. I use a local credit union but "local" with actual branch offices you can walk into and talk to a human are the important thing.
- balderdash 28d ago
  TD bank has branches but that's not super helpful if you're at the airport. the helpful thing is having a "relationship manager", someone you can call or email directly - you can get that at a big bank or a small one, and you don't need tons of assets, a lot of banks its having your direct deposit with them is enough to qualify for some special relationship.
  - preg_match 28d ago
    Online banks typically offer much more competitive financial products. I have some HYSA accounts at 4% APY, that’s not achievable with a CU or traditional bank. The solution is the same solution as any financial risk: diversification. I have many accounts at many banks. If any one of them goes down, I’m fine. If multiple go down, I’m still fine in the interim.
    - SoftTalker 28d ago

roughly 28d ago

The other rule here is that you cannot avoid politics if you want to create a better world. If you want the freedom to develop technology without worrying it will be used by a police state, you need to put in the work to make sure a police state doesn't happen - you cannot ignore politics and just "focus on technology."

api 28d ago
There's a deep failed idea here. It's the idea that technology is an alternative to politics, period. I think that whole idea has failed. You can't avoid building technologies that could be used by a police state, because literally everything can be used for warfare or aggression. The first person who sharpened a stick or a rock created a weapon. Everything is a weapon. Everything is dual-use. If you want to prevent a police state or any other undesirable political outcome, you have to confront the politics directly. "Change the tools and you change the rules," an old saying from 1990s cyber-culture, is an idea that I think has failed. Instead what happens is that the rules determine what's done with the tools. Those 90s cyber-hippie and cyber-libertarian types vastly underestimated the power of the political to the point of making a category error. Cryptocurrency is a great case in point. It started as an attempt to create a technology that would act as an escape hatch from corrupt casino capitalism and state-backed financialization. Instead it became an insane absurdist parody of those things. The same types of corrupt individuals who profited off "too big to fail" and the rest of it simply took cryptocurrency and used it to implement the same type of system but on a block chain. The rules (and rulers) changed the tools, not the other way around.
- cucumber3732842 28d ago
  I mean you kind of can. The whole premise of AI is basically that we'll make white collar administrative compliance work dirt cheap in the long run and we'll all be richer for it and we'll sidestep the whole "fight with all those industries" step because the sheer economics of it will make it happen and as those industries become cheaper/poorer there will be even less reason to not kick them to the curb. Kind of like how once certain state shed a lot of their manufacturing or agriculture, adopting policy that's bad for the rest became easier. Or how now that solar is cheap we mostly DGAF about coal.

xandrius 28d ago

I'd be careful banging two rocks together else someone might start a war with them!

gumby271 28d ago

I've been arguing with folks on Reddit about this lately when it comes to Google's changes to Android (where they've made it more difficult to install any software on your device without Google first tracking the identity of the developer). My argument ultimately comes down to the fact that Google (and Apple already) have built a technology that gives them immense power to control individual computing freedoms, and no matter their internal policy, they still must follow US law. No amount of promises or legit good faith on their part matter, they've made a dangerous technology and are behold to a government that is very eager to censor anything it doesn't like. It's so disappointing to see 1. Google engineers build such a thing, and 2. For people to support it.

stpdnerdism 28d ago

The article is not completely wrong in its premises, but phrased incorrectly. This is not about technologies (after all, an authoritarian government will use many existing technologies, from the car to the computer), but about powers granted to a government, that could be used differently by the next party/majority in power. The answer to that is to not grant government too much power in the first place. Of course, this is often a contentious point for leftists, since they do want to grant government a lot of power in the first place; they're just concerned that the exact same powers will then be used by a different political side in a way that bothers a different category of people.

cupcakecommons 28d ago
You will get downvoted to oblivion for even suggesting it here but it's so obviously a take that maps on cleanly to reality. Massive power centralization in government makes everything dual use in the service of a police state. The easiest thing to do is resist massive centralization of power in a government.
- itsanaccount 28d ago
  you get downvoted because to most people we don't care if its tyranny.gov or tyranny.com. "don't give powers to the government" just means they buy them from corporations. see: cellphone location data. the solution is not to have the power exist at all, anywhere.
  - mrguyorama 28d ago
    >the solution is not to have the power exist at all, anywhere. This is nonsensical. The power always exists, it's just your fellow human beings doing things, to you, around you, with you, without you. You cannot do anything to not have that power. It doesn't matter if your neighbors are lynching you because the evil dictatorial police state said to, or because the rough consensus of a mob of lunatics want your food. You as a human lack power. Humans working together have far more power. Sometimes, you can use the combined power of some humans working together to empower other humans. That's what laws are. That's what a State can do. That's why corporations can have power even though they are not part of government. Being able to direct 1000 humans to do something is what power is. You cannot eliminate that power. It's a fundamental part of reality. It's the entire reason animals evolved to be social at all, to take advantage of that power.

ReptileMan 28d ago

Technology is force multiplier. Everything could be used for oppression. Deal with it - the future is boot stomping on human face forever. There are lucky few that could choose whether they will be the face or foot. But it is inevitable. Or we could stop all tech development. But it is not going to happen.

AndrewKemendo 28d ago

Bertrand Russell: “If war is not impossible, then every advance in scientific technique in an advance in mass murder” As true today as then You can replace war with “police state” and it’s the same message. https://m.youtube.com/watch?v=GzTlTVaYDtk https://m.youtube.com/watch?v=GzTlTVaYDtk

naniel 28d ago

this is an important topic, and i hate that people forget how difficult it is. in any political conversation, it's pretty easy to pull this thread to unravel anyone's adamant belief in one thing or another (e.g. gun control). my biggest concern is that people seem less willing than before to actually have those discussions.. they'll immediately just label you as being opposed to their views, "against" them.. when really we need these conversations to find viable policy solutions that wont easily be used in the exact opposite way that they were intended when implemented

kingleopold 28d ago

how about you guys understand that state is criminal master org. that uses violence monopoly? is it too high level or hard to understand this? see war on drugs and more

josefritzishere 28d ago

"Civic hygiene isn't about saying we distrust our current government - it's about not trusting the next government." I've made similar points. Granting one government administration more power, is by extension granting it to all future administrations. Do so only with great care.

PaulRobinson 28d ago

This is missing the "(2013)" at the end of the HN link, but it does seem weirdly prescient as we say a rise of more polarised politics across the World, just 13 years later.

gumby271 28d ago
And a rise in tech companies building products and features that dont benefit the users, knowing we cant really escape at this point.
NietTim 28d ago
Thanks, didn't notice it before you pointed it out (and I already made my comment), actually turned into quite an interesting retrospective for me

slg 28d ago

I find it frusrating how the conversation around this almost always focuses on one side of the equation. Why is it always about preventing tools from being built in case authoritarians comes to power rather than preventing those authoritarians from coming to power in the first place? I'm far more scared off the authoritarian aspect than the tech. The Nazis didn't need Flock cameras to kill millions. It seems like focusing on the actual bad guys is going to be more effective because if the state wants to bear its weight on you, it rarely requires tech to do it.

well_ackshually 28d ago
>The Nazis didn't need Flock cameras to kill millions. No, but they were extremely happy to hop over to France and realize that we kept lists that indirectly listed jews. Actions have consequences that we may not be able to predict.
wartywhoa23 28d ago
The water flows whichever way there's a gradient of height. Don't create gradients you don't have dams for.
tehlike 28d ago
People have incentives to become more powerful - through wealth, power, leverage, or in other ways. So unfortunately, authoritarianism tends to have gravity going for it.
ds_opseeker 28d ago
The Nazis relied heavily on IBM technology for national census data (to identify jews), military logistics, ghetto statistics, train traffic management, and concentration camp capacity https://en.wikipedia.org/wiki/IBM_and_the_Holocaust https://en.wikipedia.org/wiki/IBM_and_the_Holocaust I fully agree with your base perspective, that the main focus should be on preventing authoritarians from coming to power in the first place. I also agree with your final point, that the government doesn't need tech to lean on a person. We disagree in the middle, on how tech that makes evil orders of magnitude easier might be a contributing factor and might enable the government to lean on many more people than it would otherwise.

deleted 28d ago

[deleted]

cupcakecommons 28d ago

Defaulting to discussing the US here: let's start with reducing the state's size - it shouldn't be the world's largest employer, it shouldn't be able to run an unfathomably large deficit, it shouldn't be able to declare war without congressional approval. Technology to run a police state is just the last step in a long line of ridiculously unacceptable steps.

charlieyu1 28d ago
It won’t work for any country. They all want more control, and also create a class of people depending on government money to be on their side. Only people losing out are the powerless ordinary citizens and taxpayers
runtime_terror 28d ago
How does the number of state employees de facto result in a police state? I agree with you on our obscene debt and our obsession with "conflicts" around the world, but I'd argue we're seeing the mass reduction of state employees and it's just making everything worse (eg massive deregulation and the destruction of checks and balances).
- cupcakecommons 27d ago
  I in no way endorse taking people's jobs away once they have them. I'm saying the situation itself is absurd to begin with - now we are in a terrible situation where it can only be shrunk across a long timespan where people retire and orgs shrink naturally.
genxy 28d ago
Let's slip in your pet bugaboo. Workforce size very little to do with state surveillance. In fact, mandating that it shrink can result in blowback, just like we have seen with how the US provides a great antagonist for China to evolve its own capabilities faster than if we had done nothing.
- cupcakecommons 27d ago

slipperybeluga 28d ago

[dead]

codeulike 28d ago

I'm in the UK and it always seems weird to me how in the USA its totally normal to publically register yourself as either a Democrat or a Republican. To me seems like the kind of thing that should be private. In the UK I might choose to be a member of a political party but no-one needs to know that except that particular party. Or I can just join no party and vote in private on polling day.

Hugsbox 28d ago
Because by and large they're all trapped in an "us vs them" team sport where both sides believe everybody on the opposing team is literally Satan.
- frollogaston 28d ago
  No, it's because you need to register for a party if you want to vote in its primaries.
- copper-float 28d ago
  I'm really hoping we can get over this silly delusion soon.
deleted 28d ago
[deleted]
odiroot 28d ago
Then again, EU citizens would find it questionable how in the UK recruiters are asking about your ethnicity, disability status etc.
- frollogaston 28d ago
  UK also arrests you for boomer facebook memes. Then again maybe parts of EU too.
bee_rider 28d ago
It seems quite weird. Our parties are generally quite weird because were are not supposed to exist in the first place and so most of the rules are sort of ad-hoc. I don’t belong to one, but it is sort of annoying to not be able to vote in the “primary elections” (where the parties pick their candidates). My state has a strong partisan lean so this is the only election that actually matters here. Oh well.

krautburglar 28d ago

Assholes like this were all for the police state back when it was up to stop-the-spread “contact tracing”, real-time “fact-checking” my private messages, arresting people for facebook posts, and transing your kids. Now that the red pedophiles are in charge instead of the blue ones, he changes his tune.

NietTim 28d ago

Edit; wrote this comment before I realised this is a blog post from 2013. In 2013 I would've agreed 110% with this. ----------- This is one of those things that make for great post title and gets you all riled up at first but when you think about it, it's really shallow. Most civic databases can be abused by any sufficiently authoritarian government. A homeless shelter's records may contain exactly the sort of information that could be used to get "undesirables" off the street. But those same records are useful for coordinating beds, outreach, medical care, etc. Surely the conclusion isn't that the shelter shouldn't keep records. It seems much more useful to focus on limiting what gets collected, who gets access to it, and how long it can be retained. In other words, boring things like data-minimisation and retention laws. Somewhat amusingly, that gets you pretty close to the sort of principles behind GDPR, which isn't exactly universally loved around here. I don't really disagree with the blog post's moral intuition. What bothers me is the emphasis on individual moral choices by the people building these systems. I've seen plenty of my peers happily work for companies that have spent the last twenty years building exactly this kind of technology, there will always be someone willing to build it. That's not an argument for abandoning personal ethics, but it is an argument against treating personal ethics as any form of serious societal safeguard. "good engineers should refuse to build them" means nothing.

cynicalsecurity 28d ago

Companies that agree to work on Chat Control deserve a special place in hell. One company that was eager to participate, hoping this harmful law would pass, is Atos. Name and shame them.

seethishat 28d ago

People cite Flock as an example of this. Yet the same people walk around with iPhones and Android devices that can record conversations (who talks to whom at what times and for how long), know our exact locations and log everything we do online. If I had told you 30 years ago that we are going to build a surveillance network and make the people we are tracking pay for it all you would have laughed me out of the room. But that is what we have built. The modern Smart Phone. Now we use our iPhones to complain about cameras on public roads as if they are watching us.

NietTim 28d ago
Very flawed argument, anyone can go for a walk without their phone on them and have a conversation, nobody would ask them why they don't have their phone on them.
- seethishat 28d ago
  I know it is not a popular opinion. But it is my opinion. Do you seriously think that governments need Flock cameras when they have Apple, Microsoft and Google?
  - NietTim 28d ago
    Governments don't need flock or AMG, that's where the flaw in your argument begins.
  - runtime_terror 28d ago
    Ostensibly they need a warrant to get that data from Apple/Google whereas they have a direct line to Flock data cause that's how that relationship works
  - RankingMember 28d ago
    The government needs a warrant to access Apple/Google/Microsoft's data.
runtime_terror 28d ago

piloto_ciego 28d ago

Or, and here me out… “Become ungovernable.”

1-6 28d ago
That has been the motto of our governors.
dennis_jeeves2 28d ago
This is the most correct response.

rini17 28d ago

The sad irony of "Share this post on..." with neat list of all the services doing exactly what the post warns against. They can and often are already used by police state.

not_wyoming 28d ago

This entire thread is such a strong case for why all undergrad CS programs should have a core requirement of at least one ethics course.

Terr_ 28d ago
Especially if it has concrete historical examples of technology intended for good and how it was later used for evil, rather than something more generally-philosophical.
SoftTalker 28d ago
Every business school and MBA program has ethics course requirements. Does it really make a difference? Maybe it does, and things would be even worse than they are otherwise. But I'm not sure.
- asdfman123 28d ago
  I think the purpose isn't to make people more ethical, which is impossible, but to make ethical people more effective. It ideally makes people better at identifying when something bad is happening and gives them tools to handle it. If they're amoral in the first place though a course isn't going to change that.
- m_dupont 28d ago
  I think of it like so: All the great people in history who would be regarded as ethical never took an ethics course. Its hard to know whether an ethics course makes people behave ethically, but its clearly not a necessary requirement to ethical behaviour
  - SoftTalker 28d ago
    Many of them were probably religious though, which is basically ethics with a lot of supernatural trappings. Although lot of terrible people in history were likely at least outwardly religious, so not sure that proves anything one way or the other.
28d ago

_doctor_love 28d ago

Like others have said, I don't think it's possible to create technology that cannot be weaponized. I think it's possible (mostly) to design technology so that abuse requires more people, requires the leaving of evidence / paper trail, needs checks, etc. But still, it's not possible to prevent abuse altogether and sometimes we even applaud breaking the rules (the movie Hunt for Red October for example kicks off with the Soviet captain jumping the guardrails). Also, the most "benign" public infrastructure is usually the most impactful, e.g., water systems and electrical systems. Don't like the people in that one town who keep protesting and voting against you? Turn off their power, water, and also make it so that their trash doesn't get picked up for a year. See how they like it.

deleted 28d ago

[deleted]

jimmyjazz14 28d ago

Wouldn't a better title be "don't give out information that you wouldn't want used by bad actors"? Seems less about technology and more about privacy.

ahartmetz 28d ago

Pinging the absolute jerkwads at amutable.com who state that it's not their problem how their remote attestation technology will be used.

deleted 28d ago

[deleted]

closeparen 28d ago

The obvious 21st century United States instance of "don't collect data that the government might later use for persecution" is E-Verify.

panny 28d ago

Civic Hygiene is the opposite of how you were taught to write code... >It should be noted that no ethically-trained software engineer would ever consent to write a DestroyBaghdad procedure. Basic professional ethics would instead require him to write a DestroyCity procedure, to which Baghdad could be given as a parameter. https://blog.codinghorror.com/to-serve-man-with-software/ https://blog.codinghorror.com/to-serve-man-with-software/

tshaddox 28d ago

Clearly you need to consider how general the capabilities and use cases of the technology is. The most general-purpose technologies will always have obvious use cases for police states and other bad groups or people.

SillyUsername 28d ago

As a recruiter once said to me, if you don't do the defence work, somebody else will. So you might as well do it and do a better job and get paid well. I didn't go for the job.

applfanboysbgon 28d ago
If you don't man the concentration camps, somebody else will. So you might as well do it and do a better job and get paid well.

metalsiliconYT 28d ago

No electricity yea! No internet, no food! No Water!

sebstefan 28d ago
The impossibility of drawing a line! So difficult! So nebulous! I will work for a concentration camp

quantum_state 28d ago

The concept is like the perpetual machine in the 1800s. It is simply an impossibility. Please avoid wasting time.

Xeoncross 28d ago

I'm a tech worker. I'm given a promotion for adding invasive features based on "insights". I'm incentivized to create something patent-able to restrict competition. I'm rewarded for more detailed metrics to reveal more. I'm hired by how many people were affected by my previous work regardless of ethics. There is no reward for protection, privacy, and restricting access to data.

dvrp 28d ago

What about fire?

nullorempty 28d ago

Your taxes are building police state for you. What's the recourse here?

arjie 28d ago

That just moves the decision point to “could be used by a police state”. Some people prefer humans as being the only people who can determine what a civil violation is. E.g. if a policeman doesn’t pull you over you shouldn’t be punished. This allows for great personal discretion and the potential for a police state through such discretionary exertion of power as exemplified by the famous line > “For my friends, everything. For my enemies, the law” I prefer speed cameras, red-light cameras, and so on because automated enforcement removes discretionary power from enforcement agents. I believe that enforcement agents are the arms of the civic body tasked with ensuring the commons are not ruined. Consequently, controlling the exertion of discretion is entirely an engineering approach to controlling quality of enforcement. Others believe that enforcement agents are our rulers or agents thereof that we must submit to - and therefore to whom we must give great discretion so that they can choose whether we are worthy or not. That is incompatible with all being equal before the law.

goda90 28d ago
> I prefer speed cameras, red-light cameras, and so on because automated enforcement removes discretionary power from enforcement agents. There can still be corruption with automated enforcement. If the punishment is fees and the fees go towards the budget of those in charge of the automation parameters, then you'll see manipulation of parameters to increase fees paid. Shorter yellow lights, shorter distance between a speed limit sign and the speed camera, requiring more effort to contest the automation when it's wrong than to just pay the fee, etc.
- arjie 28d ago
  This is true. A useful difference between this form of parameter optimization and the unalignability of humans is that it is possible to force constraints on these parameters. E.g. San Francisco’s speed cameras run in collection mode to obtain baselines and then in enforcement mode where they have to demonstrate reduction in speeding. California mandates minimum yellow times. Flaws exist but we can constraint-engineer to minimize adversarial optimization of parameters. Improvements allow us to asymptotically approach our goal optimum. Human discretion, on the other hand, is fluid and dodges all attempts at enforcement. We require many human agents solely because transforming unstructured vision space into structured judgment was impossible in the past. Currently, in a sufficiently constrained environment (roadways) we are able to solve sufficiently constrained problems (speeding, running a red light). One hopes that we are able to solve this problem in more spaces as time goes by.

salynchnew 28d ago

https://neveragain.tech/ https://neveragain.tech/ is still up.

arionhardison 28d ago

About 15 years ago I was locked up in SF (850 Bryant/HoJ 7th floor) and everyone was talking about their "jobs"; like what they did for a living outside of the obvious. I said "I fuck with computers and shit" and I got some follow-up questions I did not expect like - "wtf you doing here", they were all along those lines. I been locked up plenty of times but this was the 1st time I was publicly ridiculed for it. This guy named Vic; big ass black dude with an extremely visible front tooth missing said: "Man, if I could do what you do I would rebuild the Gov." What followed was a discussion where everyone in the pod which not only showed interest and consensus but an abject disappointment in me and my choices so I said jokingly, well OK; if I am ever locked up w/ my computer I'll rebuild the Gov. Then Covid hit, i got fired from Goop and I found myself essentially "locked up w/ my computer". Here on HN; this is NOT a popular sentiment and I know it will garner nothing but admonishment in the form of downvotes and ridicule but I have really been able to help some people w/ housing, re-entry, healthcare, addiction etc... so fuck it. Everyone in that pod agreed the Gov was broken. Nobody could put their finger on it or articulate in what AI might deem a "load bearing" - HHAHAH capacity. The other day i saw a comment on here by bob1029 [1] and he put it better than I ever could. It's broken because nobody can execute across all the moving parts; its a coordination issue in many ways - the cost of collaboration has become so large that it has become the thing to overcome instead of whatever first principal is the whole point of the shit e.g.: Healthcare, Food, Housing etc... The bottleneck was never intelligence it's coordination. The cost of getting many parties to act together without the whole thing blowing up. The Anthropic multi-agent article hit it too...turf war, the 2.4M-request queue, the price collusion, the merge silos. Every one is the same thing: free-form runtime "negotiation" blowing up. And every success is an agent ad-hoc reinventing an institution mid-fight inventing a "verifiable criterion," writing an apology commit and pulling in a human, running a tournament as a commitment device. They keep hand-rolling the institution, badly, under fire. So stop making them reinvent it. The generalization of the "constrained action space" idea isn't just bounding one agent's moves it's what I call an intent: intents don't just constrain each agent's actions, they codify the coordination itself. And I make the unit of coordination a first-class object: the Deal a structured negotiation instead of a free-form one. The Deal is the container those agents keep reinventing, handed to them up front: scope and terms, the parties and their roles, the ordered steps, a verification gate as the neutral arbiter, and an appeal / human-escalation path as recourse, all over an append-only record. Agents negotiate inside a bounded contract instead of improvising one mid-fight. How it works from what you say to what actually happens (the "Deal") 1. You speak the intent. You can use a form but I found people like the verbal shit much more. 2. Break the problem into an ontology. The ask gets decomposed into the real work it implies mapped to actual job roles (O*NET). Problem → Solution. 3. Vet the solution with the gov council agents. Every domain has one public AI expert in that agency's real systems Medicare, Medicaid, SSA, OPM (pda:codify.<domain>). Policies get deliberated and voted, elected officials in the loop, before anything runs. Stakeholders. 4. We staff it. It matches the specific expertise needed the exact specialty, not a generic "doctor." Down to the right role for your case. 5. We set up the financing. The money that follows the person eligibility, benefits, the payment rail. Financing. 6. An agent manages the process with expertise humans in the loop for anything high-risk, adjusting as outcomes come back, like an adaptive trial, until the outcome is met or it fails honestly. Expertise → Outcome. The atomic unit of value here is a "Deal". Problem broken down, solution vetted, staffed, financed, executed under contract. intent → Deal → outcome. Overview - Intent: https://wiki.erpo.law/agent/wa-erpo https://wiki.erpo.law/agent/wa-erpo - Program: https://erpo.law/program/71069-petition-for-an-erpo-as-a-family-or-household-member https://erpo.law/program/71069-petition-for-an-erpo-as-a-fam... - Protocol: https://erpo.law/protocol/71079 https://erpo.law/protocol/71079 I'm building it in the open because I think thats the only way this can be done https://medicare.dev https://medicare.dev https://epa.dev https://epa.dev etc... Learning how to take criticism better so the more criticism the better. [1] - https://news.ycombinator.com/item?id=49316271 https://news.ycombinator.com/item?id=49316271

gagan2020 28d ago

They all are created by US companies only. FB, Instagram, Youtube, X, Google (Android & others), Apple, etc. Face Recognition, Drones, space exploration (engines for missiles), AI, LLMs, etc. So, Either do it yourself (China) or get run by them (India).

bronlund 28d ago

I think it’s the other way around. We are from time to time graced with scraps from military research and we think it’s neat.

graemep 28d ago

[dead]

gos9 28d ago

Is a prosecution fantasy comorbid with sexual deviance?

docmars 28d ago

But only if it isn't Republicans, right? The Biden administration sure loved partnering with private tech companies to censor and monitor Americans to fight imaginary bogeymen like those darned evil white people who want their own nation like everyone else! (gasp!)

AuthAuth 28d ago

If your government has elected someone bad its already over. So much effort is put into trying to restrict what the highest point of power may do. But its the highest point of power and if its compromised you've already lost. See trump breaking all the checks and balances. Its better to build what you need and focus on getting good people into power. You shouldnt be able to hide from your government.

ninkendo 28d ago

Title was mangled. The actual quote is: > It's bad civic hygiene to build technologies that could someday be used to facilitate a police state Not “avoid building technologies that could be used by a police state”, as the HN title currently says. The HN title reads like any technology a police state could ever use should be avoided. Like cars, or pencils. Anything. Schneier’s quote says “used to facilitate a police state” is a lot more specific: could be used to bring a police state into existence where it wouldn’t have been possible before, like having a database of everyone’s sexuality, etc.

cpill 28d ago

just make guns that on l can only shoot the baddies! easy!

p0w3n3d 28d ago

In my country there are two parties always being elected. Quite similar to US system. It's always first saying "elect us, otherwise second will win" and the second saying the opposite. Asymptotically those parties go the same way, but always introducing new laws that improve oppression-style control, explaining that those others must be controlled. And the voters accept the solution, but do not understand that the next year the second party will come and use the solution. I don't know if you get what I'm saying without the context but this is exactly the case: imagine police state appearing in your country. Does the law help them or are you still free

pyaamb 27d ago

Does explicit licensing and the threat of a lawsuit related to violation of it not help here?

M95D 27d ago
Lawsuit against a tirant? Who would dare?
- pyaamb 27d ago
  I guess it falls on how the courts see it. I'm sure the app devs would love to take their money.
  - M95D 27d ago
    What courts? You think you can sue someone like Putin or Kim and ever see the light of day again?