10 ms·

Stop developing this technology

• baal80spam 4y ago
  Technology isn't inherently good or evil, it's neutral. If Company A / Country A / Person A won't do it, then Company B / Country B / Person B will do it and use it to bankrupt you / attack and possibly kill you / take advantage of you. It's that simple.
  • Havoc 4y ago
    >Technology isn't inherently good or evil, it's neutral. Broadly yes, but this severe simplification glosses over the some technologies having more potential for abuse than others. A machine gun has more potential for evil than a dessert spoon.
  • gizmo 4y ago
    No, technology falls into different categories. Aspirin is unambiguously a technological good. As is the bicycle. There are many technological advances which are exclusively (or almost exclusively) used for the benefit of mankind. Then there is morally neutral technology. Think of a hammer or a knife. Can be used just as easily for good as for evil. It's up to the person who wields it. The third category is evil technology. Technology that just makes the world a worse place. Think of landmines, nerve gas, or biologically engineered viruses. If we could uninvent things, these are the things we would uninvent in a heart beat.
  • davidivadavid 4y ago
    So technology is good for the Company / Country / Person who builds it first and evil for the one who doesn't. Something being good for some people and evil for others doesn't make it "neutral."
• proto-n 4y ago
  Here's what I would have answered to the OP in the link: This technology is going to be developed regardless of what we do here. Please realize that you are not advocating for it not to be developed: rather, you are advocating for it not to be developed in the open.
  • josephg 4y ago
    Scammers don't have an unlimited amount of money and skill. Frankly, if the people working at scammer call centers had the skill to make something like this, they probably wouldn't want to work in a scamming call center. Let this technology be made by someone else. Let it be made by governments and whoever in secret. We know its possible; but just because its possible doesn't mean you have to be the person who does it. If we can delay scammers using this stuff for a few years, we'll keep millions of people from being scammed in the intervening time. Thats a win in my book.
• arein3 4y ago
  This doesn't produce any physical harm, so I see no problem in developing it. It will not spiral out of control. There will be a break in period, but the conclusion will be check the source of the information. Making this easily available will make the break in period easier.
• blondin 4y ago
  haven't paid much attention to the deepfake community. but this one is debatable. one of their linked forums has a section for flagging uncredited videos or work. so, deepfake authors want credits for their work. that's perplexing. what's more, this is happening while they seem to be ignoring the ethical concerns raised in the issue. citing that people can do whatever they want with the tech.
  • Rimintil 4y ago
    Technology like this has blown up on Twitch/Twitch circles within the past couple of days (I won't link to it due to the sexual nature, but you can find references on /r/livestreamfail). Men taking women's [Twitch streamers] faces and use your imagination from there. Completely inappropriate and unethical. What value does this technology add? Bringing Princess Leia back to another Star Wars movie? Anything more than that?
    • cwillu 4y ago
      You don't have to use your imagination, they proudly list “mrdeepfakes - the biggest NSFW English deepfake community” in the repo's readme.
• jakelazaroff 4y ago
  Regardless of what I think about this technology in particular, I want to respond to this line from the second comment: [1] > on the contrary, it must be developed. No, it mustn’t. There’s not a gun to your head, forcing you to do this. You want to develop this technology. That’s why it’s happening. Technology isn’t inevitable. It’s a choice we make. And we can go in circles about whether it’s good or bad, but at least cop to the fact that you have agency and you’re making that choice. [1] https://github.com/iperov/DeepFaceLive/issues/41#issuecomment-1047017440 https://github.com/iperov/DeepFaceLive/issues/41#issuecommen...
  • jolmg 4y ago
    "must" is probably the wrong word, but if they don't develop it, someone else will. At most, it would be delayed, and I'm not sure there's much difference in that.
    • jakelazaroff 4y ago
      Can we agree that “someone would have done this anyway” is a poor justification for one’s actions? That’s what you say when you 1) recognize that you did a bad thing, but 2) don’t want to be blamed for it.
      • jolmg 4y ago
        Can we agree that it's fundamentally not a bad thing to develop a technology? If you were a legislator, how would you like to criminalize the development of technology? You know, the most difficult part of this tech is probably the facial tracking, which is probably also used for the animation of 3d avatars. The jump is probably not huge.
        • MattRix 4y ago
          Many things can be unethical/immoral without being illegal, and of course there are some technologies that are fundamentally bad.
• Lacerda69 4y ago
  an age old discussion. ultimately i prefer for this technology to be developed in the open on github and be aware of it (and able to combat nefarious use of it). the alternative is that its being developed hidden and used by the most vile and evil without many being aware of it (which it most definitely will) As with nuclear, the cats out of the bag or the babys bathwater(?) already spilled, no way to turn back the clock on technological innovation.
  • tgsovlerkhgsel 4y ago
    I disagree. Most scammers are unable to build this. Providing attackers with ready to use tech will increase attacks.
  • ano88888 4y ago
    now that it is in the open , any high school boy can create some porn deepfakes of their classmates and suicide rate will probably rise. Let's see if you still hold this view when one of your family members is a victim
    • notfed 4y ago
      Hard disagree. Quite the opposite: the existence of deepfakes provides a safety net of plausible deniability. Suicide rates should drop as the technology becomes more prevalent.
    • charcircuit 4y ago
      Rule 34 of the internet. It's inevitable.
    • number6 4y ago
      We are lifing just fine with car related deaths. The rate is much higher than suicide rates. Bullying is alive and well even without this tech. I guess scissoring out a photo from the yearbook and glue it in a porn magazine would work as good. It is not about believability but about the psychological damage when it comes to bullying
• dusted 4y ago
  I think it's a good thing. Not that it's being used for evil things, but because it should help make it obvious that you can't trust anything you see on a screen. Using fake media to trick people into believing anything used to be a privileged reserved for nation states and the ultra rich. Now that _ANYONE_ and their cat can do it, it should follow that nobody can believe anything that's on a screen anymore (this comment included).
  • ano88888 4y ago
    Right. When that chaos happen, just remember that you will not be safe from it
    • saurik 4y ago
      I am already not safe from it, but it is less obvious because people like to think the technology doesn't exist merely because it is kind of expensive.
      • lovemenot 4y ago
        Thank you saurik for what you have done ... and if appropriate for what you have chosen not to do.
    • dusted 4y ago
      Whats your alternative? Attempt to hide the fact that everything can be easily faked, and that you don't need to be a hollywood studio to do it? What good do you think that does? Instill a false sense of security and trust in a medium which cannot bear it ? How's that not worse? Even outright banning this technology won't make a dent in the bad uses, since those individuals are highly motivated, highly competent and don't care the least about the ban.. It's not like the development won't happen, it will just be hidden, making it less obvious to the potential victims (increasing the size of this group, since you then need to be "in the know"). So no, the only way forward is to have this out in the open as much as possible so that as many people as possible become aware of how trivial it already is to fake stuff. So, what are you saying ?
• yipbub 4y ago
  I'm convinced that this idea that technology is completely neutral is wrong. It is not neutral in the face of human psychology. The human species is a different animal that the human individual, and it is powerful, but does not make truly conscious decisions. Once you let the genie out of the bottle, a wish will be made. A technology might not be inherently bad, but neither are knives, and we don't leave those lying around. That said, it is the human species that develops technology, rarely is one human individual capable of holding back a technology.
  • bigDinosaur 4y ago
    'Technology is neutral' is simply a banal observation that technology is only ever used by humans who can decide what to do with it. It's not saying that a new technology doesn't enable new and terrible human choices, simply that those choices rest with the people, not the technology itself. Knives seem like the perfect example, really. We do leave them lying around in our drawers, usually under zero real security against any malicious guest, but we recognise it is about the choices of the people we invite into our houses, not the existence of the knives themselves, that is the real danger. You can certainly argue though as you have that humans simply shouldn't have the choice to do certain things - what comes to mind immediately is nuclear weapons.
    • yipbub 4y ago
      Yeah, my argument with the knives was pretty weak and I was trying to make the point you make in your last paragraph. I think I meant to type "lying around children". I should've said bombs.
    • unhammer 4y ago
      Knives always seem like a perfect example because they really do have so many good uses, along with some bad ones. But just because two pieces of technology both have good uses and bad uses doesn't mean they're morally equivalent. As to your example, even if I trust people who enter my house I wouldn't want to store a nuke in my kitchen drawers :) And there are many ways to discuss the ethics of tech beyond "good and bad uses", such as to what extent a user of the technology controls it or is controlled by it, what are the negative externalities, does it increase power asymmetry, etc.
• qthrowayq0909 4y ago
  Tangent, but I think things like that will spell the end of remote working and remote interviewing.
  • ideamotor 4y ago
    Most people will follow the social norm of not doing this. Until that norm is broken. Perhaps the first successful presidential candidate that is revealed to be virtual will be the event that changes things.
    • ShowalkKama 4y ago
      I vote waldo
  • atemerev 4y ago
    Why do you even need to look at someone's face for remote working? Most of communications are text-based and asynchronous.
    • qthrowayq0909 4y ago
      because HR-wise a company needs to be sure it is working with whom it thinks it is working. See that under the prism of rising interview scams.
• winrid 4y ago
  On a side note, is this really all in Python? I imagine it's offloading some stuff to the GPU right? Maybe the GPU instructions are also stored in python??
  • wnkrshm 4y ago
    Among others it's using pytorch and numpy, two python packages written in compiled languages, torch can use the GPU.
    • winrid 4y ago
      Ah, thanks!
• kefka_p 4y ago
  Can anybody demonstrate a legitimate use of deepfake software? Has it ever been used to facilitate a socially positive or desirable outcome? While I recognize my experiences are far from definitive, I hazard most would be hard pressed to name anything positive that came out of deepfake technology. edit: I’ll take your knee-jerk DV, and any others, as an admission of an inability to speak to positive utility of this technology.
  • thejarren 4y ago
    Edit: this comment is referring to deepfakes more broadly, and is not a commentary on the validity of the source linked here. I can't speak to the reputability of the community developing this, or how it has been used so far. -- I'm a fairly visual and imaginative person, and it's pretty easy for me to come up with some very useful applications. No hostility intended, genuinely sharing my thoughts: 1. CGI for video editing - lower the bar of entry to de-age actors, or use a stand-in. Actor can't make it to a shoot that day? No worries, replace their face in post easily. 2. Identity protection - Cold call with someone that reached out to you, you're not sure if they're safe or dangerous, could be a good way to protect yourself. 3. Social media content for clients - become a fake avatar for hire essentially, customize your narrator for any video or brand. Video call centers with fake video (they already have voice modifiers and fake names), Enhanced VTuber sort of things (virtual avatars for streaming). 4. Unexpected outcomes: for example Holly Herndon created (and sold) access to an AI replica of her singing voice (n1), and I could see artists selling or renting access to their faces. Obviously this can and will be used maliciously, but I personally could see myself using it for more positive reasons. n1. https://holly.mirror.xyz/54ds2IiOnvthjGFkokFCoaI4EabytH9xjAYy1irHy94 https://holly.mirror.xyz/54ds2IiOnvthjGFkokFCoaI4EabytH9xjAY...
    • kefka_p 4y ago
      First let me thank you for a thoughtful riposte! I do appreciate that. My question was an honest one and I imagine, not the easiest to conceive an answer to. I genuinely appreciate your taking the time to share your thoughts. With that said almost every use-case cited was financial or monetary gain whereas I enquired about social utility and value. That dishonesty ie creation of a fake avatar is cited as being of social utility strikes me as a reach. I don’t see how adding more dishonesty and facades to the world adds social value, but then I may just be of limited imagination.
• ttsalami 4y ago
  Quite often in science fiction media, despite the advancement of technology, I see text and graphics(but not pictures) only as User Interfaces. I wonder if this is the path we will be going down. Zero trust towards images and video
• nathias 4y ago
  There are a lot of completely benign usecases. 1. I want to deepfake myself to have an avatar for online interaction. 2. I want to generate videos instead of filming by pasting people into existing videos. 3. Prevent a Face/Off of scenario.
• ge96 4y ago
  Hmm can't close this issue
• mihaic 4y ago
  HN seems to actively work on a cognitive dissonance: on the one hand producing inspirational stories of entrepreneurs changing the world and on the other abandoning all hope that technology/market forces can be controlled in any way. I'm thinking now this is to justify away the collective guilt of bringing into the mainstream harmful products. It seems to come from the same origins as "crypto can't be regulated", "government can't to anything", "it's ok because it's legal" and it's always worrying me to not really see any sort of moral stance being taken anymore.
  • atemerev 4y ago
    Well for me, crypto is highly moral and should be developed to prevent artificial trading restrictions and eliminate borders. There are many people who consider individual freedom to be of highest importance, and who are willing to develop any technology promoting it.
    • mihaic 4y ago
      You seem to be thinking ideologically-driven, while I'm more goal-driven. Is absolute freedom the end-all purpose? Shouldn't I be allowed to murder my enemies, if I'm willing to risk the vengeance of their group? I know it's an absurd question, but my point is that the social negotiation for where regulation and limits should come into effect has been abandoned by many.
      • JohnFen 4y ago
        Absolute freedom is impossible in a world where people interact with and affect each other. Me having absolute freedom means that the freedom of others must be limited. The goal should be to maximize freedom for everyone. To accomplish this goal means that nobody can have absolute freedom.
    • jevgeni 4y ago
      Most crypto is controlled by a few key players with even less oversight and accountability than the governments.
• rvieira 4y ago
  Particularly ironic using the defence of "it’s not the technology that’s to blame, but the person, not the machine gun, but the person". Machine guns, an advanced piece of engineering widely known to being developed purely as an academic exercise. No one could expect other uses.
  • wnkrshm 4y ago
    The argument is slightly different here though: The OP is blaming the person who invented the machine gun in the analogy.
    • rvense 4y ago
      Which is obviously reasonable! If you know how to build weapons, don't. Let somebody else do it.
      • Buffout 4y ago
        Scientist have moral responsibility not to build weapons of mass destruction for example. Too bad even Einstein couldn't keep with your advice. Scientists are sometimes not so smart. They can be fooled like anybody else.
        • wnkrshm 4y ago
          We're not talking about accidentally inventing a machine gun, we're talking about being cautioned about inventing a machine gun and not caring: "Hey you're inventing a machine gun!" "No I'm not, and even if I was, machine guns can be used for many things other than violence!" Edit: Also, Einstein in this analogy is the one cautioning about someone being able to invent a machine gun.
• caporaltito 4y ago
  This reminds of the whole crowd of artists calling for a ban of AI generated art because "it's stealing". The change will happen, whether you want it or not. So those guys'd better raise the prices for their unique, by-hand manufactured, hard worked pieces of work and leave the low-quality, generic and industrially generated ones to AI. Embrace the change, as they say.
  • YurgenJurgensen 4y ago
    Do you carry a gas mask with you wherever you go? Mustard gas is over a hundred years old, and change will happen, whether you want it or not, so you should be able to just go down to the supermarket and buy it. I'm personally very annoyed by random mustard gas attacks, but there's just no mechanism in society for preventing them.
    • caporaltito 4y ago
      Your analogy is 100% on point, I guess?
• epups 4y ago
  The big confusion here is that this particular development effort is the one creating this technology, rather than diffusing it. If somehow we could get the United Nations to agree to ban deepfake development worldwide, then surely we should enter an ethical discussion about whether we should do it or not. In a world where sophisticated actors already have access to these (and much better) tools, having an open-source GitHub repo is a good thing in my view.
• dmingod666 4y ago
  How much are you going to regulate and stop? Game engines currently give photorealistic realtime content.. stop unreal and metahumans too? What about v6 of unreal? What if this same repo was owned by Nvidia and it had some commercial interest on that product and were ready to litigate.. would everyone still pileup on it? Is it not on some level disdain that it's just run by a bunch of guys that can be pushed around without much consequence. Would we have a thread saying, screw it shut down ChatGPT, doesn't fit my moral world-view.. why is that absurd but this is fair discussion?
• tgv 4y ago
  Remember: if you helped develop it, you're responsible for it. If it kills people, you share the blame. So, what's the possible scenario for that outcome? Well, look at the upcoming elections in Nigeria. The BBC writes: "With an estimated 80 million Nigerians online, social media plays a huge role in national debates about politics. Our investigation uncovered different tactics used to reach more people on Twitter. Many play on divisive issues such as religious, ethnic and regional differences." ABC News writes: "At least 800 people died in post-election violence after the 2011 polls." Adding deepfakes into this mix can trigger violent reactions. Should that happen, the creators of deepfakes are obviously to blame, but also those who enabled them, and that includes the original researchers, are responsible. Ignoring that is just putting your head in the sand.
  • tlongo 4y ago
    I think "blame" is the wrong term here. No one can be blamed for unintended consequences (most of the time, I guess). However, the fact remains that one is a crucial part of the chain of events that led to the very existence of the consequence.
    • tgv 4y ago
      Knowingly. It's not as if no-one warned, and as if there aren't any precedents. But you wouldn't blame the man who puts a poison pill near a playground, and claims "I didn't make them eat it"?
  • kyleyeats 4y ago
    This reply to your comment could only exist because you posted your comment. Are you responsible for anything I say in this box? You enabled it!
    • tgv 4y ago
      Exaggerating one aspect of argument into ridiculousness and ignoring the rest does not show "good faith".
  • ronsor 4y ago
• azubinski 4y ago
  Official discord channel: English / Russian. 中文交流论坛，免费软件教程、模型、人脸数据 LOL BTW. It's not a "technology" in any classical sense of this word. This is a funny and technologically useless rattle that can be used as a Chinese-made Kalashnikov assault rifle.
• wruza 4y ago
  if someone blame this technology, why not to blame guns, warships, tanks, airplanes, shotguns, machineguns before blaming this technology? We actually blame them, except for airplanes. Most of these were invented at the times when lives had much less value and are of no use unless some half-minded pig attacks you or tries to undermine your defenses. I’d like to see how this line of reasoning changes when someone releases a virus for your DNA in your backyard, made with funnyjokes/easy-create-virus-for-a-drone-app.
  • thedorkknight 4y ago
    I'll give them props for using an example that is at least challenging to think of positive uses for. The comments snarkily advocating banning electricity or penises just make me wonder what the users' reading comprehension skills are, given that the initial comment is pretty clear on the problem being that they don't think deepfakes have ANY positive use
• kstenerud 4y ago
  This kind of technology is far too useful to repressive regimes and those who wish to do nasty things with it. This means that the incentive to develop this technology is already there, and so it WILL be developed no matter how much people wish it wouldn't. The only difference at this point is whether some of the implementations are developed in public view or not. If none are public, then all of them will be done in secret, and our opportunities to develop countermeasures will be severely hampered by having fewer eyes on it, and a smaller entry funnel for potential white hats.
  • The_Colonel 4y ago
    > it WILL be developed no matter how much people wish it wouldn't. Sure, but there's a value in delaying that development. Delaying harm is a valid tactic.
    • ilyt 4y ago
      The entities with interest in it have bigger pockets than some random open source project.
      • The_Colonel 4y ago
        There are many small entities with interest in such a technology which don't have huge budgets. Small terrorist / extremist organizations pushing a specific agenda. Countries like China can no doubt develop this independently, but they are unlikely to share the technology with random extremists.
    • x-complexity 4y ago
      > > it WILL be developed no matter how much people wish it wouldn't. > Sure, but there's a value in delaying that development. Delaying harm is a valid tactic. It is a horrible tactic when your adversary has ultra-deep pockets & mountains of bodies to throw at the problem. Whatever's being developed here has 1/10th the capability of the tech being developed in black site facilities at the behest of national militaries.
• dtx1 4y ago
  This is akin to developing Bioweapons. Can it be done? Yes. Should it be done? Absolutely not. > “Your scientists were so preoccupied with whether they could, they didn’t stop to think if they should.”
  • CatWChainsaw 4y ago
    Only in this case it's software engineers. They don't get a free pass.
• lucumo 4y ago
  There's a book in Dutch literature, The Assault by Harry Mulisch. It's a great read if you're interested in multi-faceted morality. It deals with guilt and responsibility when there's a chain of things that need to happen for a bad outcome. During WWII the protagonist's parents are shot by Germans after a body of a collaborator was found in front of their house. The parents were arguing with the soldiers about something when arrested and ended up shot during the arrest. The collaborator was shot by the resistance in front of their neighbours house, and the neighbours moved the body in front of the protagonist's house. Over the years, he encounters many people involved in this event, and starts seeing things from many sides. One of the themes that's explored is who bears moral responsibility for his parents' death? The Germans for shooting them? His mother for arguing? The neighbours for moving the body? The resistance for shooting the collaborator? The collaborator for collaborating? All of their actions were a necessary link in the chain that lead to their death. One of the characters utters a simple and powerful way of dealing with that morality: "He who did it, did it, and not somebody else. The only useful truth is that everybody is killed, by who he is killed, and not by anyone else." It's a self-serving morality, because the character was part of the resistance group that shot the collaborator in a time where reprisals where very common. But it's also very appealing in its simplicity and clarity. I find myself referring back to it, in cases like this. In the imagined future, where this tech is used for Bad Things, who is responsible for those Bad Things? The person that did the Bad Thing? The person that developed this tech? The people that developed the tech that lead up to this development? I'm much inclined to only lay blame on the person that did the Bad Thing.
  • xg15 4y ago
    That's a very convenient kind of morals for the people who invested billions of dollars into developing this tech, fully aware what it could be used for. (And OpenAI was aware, it's their entire justification why they don't publish the models)
  • leobg 4y ago
    I want to thank you for the recommendation. I am reading it right now. It is excellent!
  • DoreenMichele 4y ago
• renewiltord 4y ago
  All the communities that are against this are also against everything else. They've cried wolf enough times. Everything is dangerous and everything is a crisis. Consequently, I will ignore their warnings about this as well. It'll be okay. Tomorrow the community will forget about it. Tomorrow the crisis will be that some one-person blog is not GDPR compliant.
• givemeethekeys 4y ago
  I can see a few good uses for this tech: Face swapping + voice swapping + auto translate = your customer support can be anyone on the planet but look and sound familiar to you. Maybe you're getting over a facial injury. Face swapping = you no longer have to put on make up. Just swap your made up face for meetings. Face swapping + voice recordings + AI that learns = that scene in Contact where Jody Foster talks to the alien - but he takes the form of her father to make her feel more comfortable.
• college_physics 4y ago
  Its hard to take seriously the argument that "tech is neutral" when it concerns software. One could maybe make this argument for the hardware underneath (the chips and the cables). They are called after all "general purpose computing" devices and the packets moving around are general purpose streams of bits as well [0]. But software is not "tech". It is the explicit expression and projection of cultural objectives and values onto a particular type of tech. You can take the exact precise hardware we have today and reprogram a million different worlds on it, some better, some worse. Developers are simply the willing executioners of prevalent power structures. Deal with it. If you have a moral backbone (i.e., you don't agree with the prevalent morality as expressed in what the software industry currently does) do something about it. [0] Ofcourse upon deeper examination overall system design (e.g. how client or server heavy the configuration, what kind of UI is promoted etc) is not neutral either. Cultural/political/economic choices creep * everywhere*
  • davidivadavid 4y ago
    > But software is not "tech". It is the explicit expression and projection of cultural objectives and values onto a particular type of tech. You can take the exact precise hardware we have today and reprogram a million different worlds on it, some better, some worse. But all of them with computing as a cultural artifact of some kind — and certainly not a "neutral" one. I've basically come to think of the idea of "general purpose computing" as basically a trojan horse for the naive "tech is neutral" POV. "Making things computable faster" is not a "neutral" change of the universe, it's a step in a specific direction, it defines the boundaries of a particular future cone at the exception of many others.
• qwerty456127 4y ago
  The deepfake thechnology is awesome and should be available to everybody. Because this is the only way everybody can be finally taught to think critically about everything thay hear/see. Can you believe a politician saying something on TV? Hell no! You should exercise logic about the whole political play he is a part of. Should you think bad about a person you find on a porn site? Absolutely no, what good could result out of this in any case? This has always been like this but now there is a thing which can push this into the common sense.
  • trashtester 4y ago
    This rabbit hole goes waaay deep. Why should people believe anything you say on this topic? How can they know you're not a bot or a psy-ops troll working for Xi or Putin with the mission to undermine the very fabric of Democratic societies? What truths/facts should people NOT doubt when they're using "common sense" and thinking "critically". Are we not all affected by propaganda that directly targets what we consider "facts" and "common sense"? So, to complete the paradox, I will claim one "fact": As humans there are VERY FEW "facts" we can know from first principles, and there are even fewer pinciples we can now for a fact to be universal, or even useful. We DEPEND on at least some authorities, whether those are people, institutions, ideas or beliefs. When someone says "think critically!", they mean that you should put some higher burden of evidence on SOME of your beliefs. But not all, and definitely not those that this someone take as axiomatical truths. An my main worry is not to find some celebrity's face on PornHub. My main concern is the day when almost everything we see that claims to be news has been tampered with in such a way. If we have now way to tell a deepfake from a true video, we can be made to believe absolutely anything. This can be used to ruin lives, trigger wars, including civil wars, cause nuclear holocaust. And it's already happening. Twitter is full of lies of every kind and from all parties and countries. We may need to find a way back to a world where there is one main shared narrative that we can all more or less trust. Where the custodians of the institutions that provide the narrative understand the need to maintain the trust and the risks involved in undermining the trust for personal gain, and where there are checks and balances that removes bad actors from such positions. Without this, I believe we're f'ed, but HOW to obtain it, I don't know.
• _8j50 4y ago
  All the negative responses are slippery slope fallacies.
• jablala 4y ago
  What utterly horrible minds in that comment thread. The justification that other bad things happen means we can reasonably create more evil is disgusting. Those posses a complete lack of a moral compass.
  • noAnswer 4y ago
    Joseph Weizenbaum once said to the question (paraphrasing) "If I don't develop evil/unethical thing Z, someone else will." with: "In city X every day there is rape and if I don't do it, someone else will." Showing how stupid this line of reasoning is.
    • bondarchuk 4y ago
      But that's stupid, if he does it there is MORE rape.
      • noAnswer 4y ago
        So not only will someone else do it, if you join, there will be even more harm. So don't do it!
        • bondarchuk 4y ago
          Which is a different situation than the one with developing some technology. If someone develops faceswap, and then someone else develops it too, we won't have twice the amount of faceswap-badness. edit: which is not to say that I think "if I don't do it, someone else will" is a good argument for doing something you disagree with..
  • thedorkknight 4y ago
    It's not just that, they also lack reading comprehension skills. The initial comment could not be more clear that this tech has practically no positive use whatsoever, yet all the snarky "guess we should ban electricity" comments bring up things that have pretty obvious positive uses
  • 4y ago
• spaceman_2020 4y ago
  Clear that we're moving to a post-truth society. Visuals can be deepfaked, voices AI-generated. Could crypto unironically be the way out of this mess? If a document isn't signed by a wallet associated with you, it should not be considered authentic?
  • kozziollek 4y ago
    And why would you need crypto for that? You can sign stuff without crypto.
    • spaceman_2020 4y ago
      Sure. I just mentioned crypto because that's what I'm familiar with, and there's already some infrastructure in place to facilitate it. But you will likely need to sign messages posted online - crypto or with your encryption key.
    • msm_ 4y ago
      You can't sign stuff without crypto (unless you mean by hand, on paper). Cryptographic signatures by design involve cryptography. I know what you both meant, just playing with words, sorry. I just hope now that the dust has settled, we can go back to the world where "crypto" meant "cryptography", not anything else[1] [1] https://www.cryptoisnotcryptocurrency.com/ https://www.cryptoisnotcryptocurrency.com/
  • petesergeant 4y ago
    > we're moving to a post-truth society This is far too dramatic. What's actually happening is that we simply no longer give _unattributed_ audio and video the benefit of the doubt. Data pedigree and attribution will become more important, but we're already in a place where real media is routinely misrepresented by bad actors (as a lot of purported Ukraine war footage has shown), and this will simply make us more skeptical still. > Could crypto unironically be the way out of this mess I did pitch a PhD study area on this[0] before deciding I didn't want to do a PhD or get into crypto. But I do think signing media and signed attribution chains will become more important ... but I don't see much need for it to be distributed. I (along with a huge number of other people) already sign documents digitally using a 3rd party service 0: https://github.com/pjlsergeant/multimedia-trust-and-certification https://github.com/pjlsergeant/multimedia-trust-and-certific...
• bondarchuk 4y ago
  At the end of the day, it's just pixels on a screen. It's not fair to compare it to machine guns or atomic bombs which cause real physical harm. You might argue that the technology to make pixels on a screen resembling real humans is bad, but then you have to actually make that argument (and "some people got scammed" is indeed such an argument, albeit a pretty weak one), not just shift it to "this is technology, machine guns are technology, machine guns are bad".
• AlexAltea 4y ago
  Instead of blocking technology, what about addressing the root problem: People need to understand concepts such as "chain of trust". Do you trust videocall participants because you recognize their faces and voices? ...Or because a server certified by a root CA has authenticated the other participants? The age of deepfakes has started, nobody can stop it. Improving our mental security models will become as essential as literacy.
• amarant 4y ago
  Lots of comments here and in the GitHub thread claiming there are no legitimate uses for this, so I thought I'd drop a legitimate use just to have an example: I saw an unrelated article today where someone had used some deepfake technology to change the spoken language of an actor. Imagine what that would mean for dubbed movies/TV if it gets good enough. There are legit usecases, and that justifies the technologies existence. The bad actors don't make it immoral to develop a technology IMO.
  • noAnswer 4y ago
    > Imagine what that would mean for dubbed movies/TV if it gets good enough. In Germany everything is dubbed. Never have I cared about the mouth movement. It's all about the voice and the voice acting. But ok, it will be used at everything, and everyone will get used to the fake. Still, it is (not will, already is) mostly used for revenge porn, synthetic CP and CEO fraud.
• CyborgCabbage 4y ago
  A lot of people in the comments here are saying that this is beneficial because it will teach people that they can't trust video or audio. But I don't see how that makes sense because this isn't some neutered or weakened form of the technology. That's like saying shooting people makes them more aware of gun violence.
  • notfed 4y ago
    > That's like saying ... It's like releasing some software that cracks all encryption (or more accurately, authentication), before of which only elite members of society had access to. It's not a physical weapon, it's an information-security weapon; I don't think the gun metaphor is appropriate.
• sva_ 4y ago
  It's crazy how we experienced this (in the future) very small time window where you could more or less trust digital media.
• jmnicolas 4y ago
  You can't stop these kinds of double edge swords from being developed but what I'd like is for people to get together and dev a counter to that, something like DetectDeepFace. For me everything (image, text, sound etc) that comes from a computer is suspect nowadays.
• halicarnassus 4y ago
  Regarding the downvotes and the second comment on the linked page: I guess the non-sensical argument "it's not the <insert technology> that <insert and thing>, but the person using the <technology>" will never die out. If you don't have <the technology>, it's much harder to <do the bad thing>, has to be done hands-on from a very close distance with much higher risk for the perpetrator.
• deniscepko2 4y ago
  I wonder if some form of regulation is coming to tech. We do not allow people to spread heroin freely or slavery or some other sort of horrible stuff.
  • wincy 4y ago
    How are you going to effectively regulate a white paper? Or Stable diffusion? These things can be trained on readily available, easily purchased by the average citizen commodity GPU hardware. Heck, there’s rumors NVIDIA has an RTX titan in the pipe for under $3000 with 48GB of VRAM! Even if the government regulates that I can’t physically have one, I could rent a rig of A100X8 servers in Russia, VPN tunnel into it, and download whatever artifacts the server generates. How in the world could any government short of North Korea effectively regulate such a thing, even if they went full authoritarian, the blowback from such draconian measures would be intense?
    • deniscepko2 4y ago
      I don't really know it does sound super hard. But maybe if we consider that we go down the path where you do not own your tech (we are close to that now) and the companies have some monitoring tools to see what you are doing. And then you have this kind of network of trusted machines that you communicate with. Edit: i really don't like the idea that i have to be paranoid about every interaction with tech.
• 55555 4y ago
  I don’t think any specific technology is good or bad, but I do think you could attempt to quantify its impact better by measuring how often it’s used for “good” versus “bad”. It’s not going to end the argument though, not least because you’ll then have to assign a relative value to abstract things like “personal freedom” or “artistic expression”. Even if you could quantify the losses to crypto scams, you can’t put an objective value on some of its more ideological benefits.
• WFHRenaissance 4y ago
  You will never stop the march of technology, especially when it requires so few developers to create it. It will emerge.
• EVa5I7bHFq9mnYK 4y ago
  Has any deep technology benefited ordinary people so far? It's mostly used by totalitarian governments, by big tech to fine tune ads, by seo spammers etc. Can't wait for web to fill up with deep nonsense and "art".
• LeanderK 4y ago
  Again and again I am astonished that people without ethics exist, that they are confident in what they are doing and that they appear to be completely unable to reflect upon their actions. They just don't care and appear to be proud of it. If this is used to scam old people out of their belongings then you really have to question your actions and imho bear some responsibility. Was it worth it? Do the positive uses outweigh their negatives? They use examples of misuse of technology as if that would free them of any guilt. As if previous errors would allow them to do anything because greater mistakes were made. You are not, of course, completely responsible for the actions others take but if you create something you have to keep in mind bad actors exist. You can't just close your eyes if your actions lead to a strictly worse world. Old people scammed out of their savings are real people and it is real pain. I can't imagine the desperation and the helplessness following that. It really makes me angry how someone can ignore so much pain and not even engage in an argument whether it's the right thing to do.
  • mansoon 4y ago
    These people are successful.
  • junofan 4y ago
    I can already fire off emails and text messages claiming to be whomever I want. I can even hire impersonators and lookalikes. I don’t get the big push against these newer technologies.
    • lm28469 4y ago
      "I'll die at some point so why don't you stab me now"
    • fps_doug 4y ago
      You're exactly proving the point GP makes, good job.
      • pc86 4y ago
        No they're not and this flippant dismissal is so intellectually lazy. The ability to scam people exists and isn't going away. People went town to town in covered wagons selling bullshit. People scammed others out of money centuries before that. The answer isn't pearl-clutching nonsense about how this technology is so different and so morally reprehensible compared to everything else. The answer is education. If someone shows up at your door and sells you a bottle of water that will cure all your ills for $100 then skips town, most people would see it as your fault for being so gullible. Someday there will be some technological or social way to easily differentiate deep fakes from real video the same way you can with photoshopped images today.
• gloosx 4y ago
  Why is it such a big concern? As far as I know deep fake can be recognised with solid confidence using other neural net model? It's only a matter of time this will be detected in every kind of real-time communication app if it poses some kind of a threat and it is rising. Sure, it must be developed, and it's a good thing it is done in public, so the defenders can prepare to defend on open-source material, no?
• JasonFruit 4y ago
  I think it's inevitable that sub-societies will form that accept technology only in ways that preserve the humanity of our interactions. We use the term "disruptive" all the time around here, but what's being disrupted is increasingly close to the heart of human life: the ability to engage with the ideas, emotions, and opinions of another human being, and to know them as a person. If a computer can convincingly impersonate anybody, putting any words you choose in their mouth, and at the same time can generate convincing words on any given subject, it can flood electronic communication with noise that's impossible to remove. It can destroy the trust we have in any interactions not had in person, and we've allowed our society to develop in such a way that interacting only in person is no longer practical. I know that it's expected that a man of a certain age will begin to say these things, but I think it's true now in a way that it was not true about "those young whippersnappers with their motorcars": we've taken a wrong turn, and I don't like where we're headed.
  • dorchadas 4y ago
    I've been having this argument over the past few days with some friends/acquaintances in a discord server. I just keep getting told that it's all just 'sensationalist' reactions to new tech. Some of them are quite gung-ho on the 'progress' and never stop to question what it's doing to our humanity and what it means to be human. I personally think the internet inherently devalues the humanity of interactions - even on voice chat on Discord or some other service. There's something missing from what we get in person that we're losing out on, and I think that's part of why the internet is, generally, a cesspool (doubly so with anonymity thrown in). But we can't escape -- go to a concert, for instance, and how many people are actually paying attention versus just capturing the moment to prove they were there and share it on Snap/Insta? Like you, I fear we're going a bad way...and they're all just "I can't wait to see how this changes things!" without ever seeing to stop and seriously question things I feel. Their arguments seem to come down to "Well, it's perfect connection for me and I think it's neat"...but then they complain about being depressed and not getting out and doing anything. Discord and the internet isn't a substitution for IRL, and we're fixing to take people assuming it is to the extreme with this new tech. A reckoning is coming, I fear.
• Sevii 4y ago
  The unfortunate part is that the alternative is for only government entities to have this technology. If open source can make a credible attempt at creating live deep fake technology, the government already has a team working on it.
• kderbyma 4y ago
  There is no good for this technology. Nothing good that truly outweighs the bad. I agree with the Sentiment. These deepfakes are not good...they cheapen everything and lower the standard for all....it's literally scammers who want this stuff and people who want to take shortcuts.... essentially you can morally judge a person by this technology and their approach
• Hard_Space 4y ago
  I think it's pretty obvious that autoencoder deepfake tech and similar technologies are going to be useful, maybe even essential in visual effects. The perceived problem seems to be that the 'irresponsible rabble' also have access to it. But as the barrier to entry for really convincing output goes up (768px /1024px training pipelines, and beyond), and it suddenly becomes something that one person alone can't really do well any more, the 'amateur' stuff is going to look far worse to people than it does now. You just have to wait for that barrier to rise, and I can tell you as a VFX insider that that is happening right now. Deepfakes are the reverse of CGI, which began as an inaccessible technology and gradually became accessible, before the scale of its use in VFX reversed that again. Now, assuming you can either afford or will pirate the right software, you could probably match any CGI VFX shot in a major blockbuster if you gave up your job and worked on it non-stop for a year of 18-hour days (assuming you'd already been through the steep learning curve of the pipeline). So it's out of reach, really, and so will the best deepfakes be. This stuff everyone is so scared of will end up gate-kept, if only for logistical reasons (never mind any new laws that would address it) - at least at the quality that's so feared in these comments.
• DennisP 4y ago
  Everything people fear about deepfakes has been true of text for the entire history of writing. We've a brief period in human history during which you could mostly believe data you received from afar, without having to trust the source, because telling a lie with video was much harder than telling a lie with text. Now it's almost as easy to tell a lie with video so we'll have to check sources again. Somehow I think we'll survive.
• ncr100 4y ago
  The sci-fi horror fantasy of engineers being assaulted for working on future-dangerous technology seems a predictable outcome of this kind of rhetoric, soon.
• lonelyasacloud 4y ago
  The problem here is how our societies handle the arrival of new technology, not the technology itself. Here it's obvious what's going to happen without robust legislation protecting the likeness of all individuals (and not just special-case celebs) from the non-consensual generation of new material. The fact that such legislation is unlikely to happen before an awful lot of suffering has occurred is a testament both to naive belief that everything new is good. And to legislative processes with a bandwidth from the age of sail that is riddled with vested interests to handle the downsides when scaling breaks the happy path assumptions. Focusing efforts on the legislative process seems likely to be more productive than point solutions that rely on techies and scientists not to develop tech that can be used for nefarious purposes.
• return_to_monke 4y ago
  if this was a discussion on generative AI, I'd agree that the cat is out of the bag and there's no stopping this now. BUT. what are people using deepfakes for, in good faith? can someone provide one example that isn't malicious? the best I could imagine is maybe amateur filmmakers deepfaking their faces onto existing footage to cut costs - but it doesn't seem that this outweighs the drawbacks
  • salt4034 4y ago
    Film studios can use deepfakes to bring back actors who passed away. Whistle-blowers could use it to have video calls with journalists without being identified. Privacy-minded regular folks could use it to have a YouTube/Twitch channel without revealing their identity. These are just a few good-faith applications, but I think the list goes on.
    • CatWChainsaw 4y ago
      >Film studios can use deepfakes to bring back actors who passed away. That just reminds me of the short story about a society where nobody died. It wasn't meant to be a happy story.
• SergeAx 4y ago
  Let me see how crypto is banned first. Same case, even worse: not just used to scam people out of their belongings, but to buy and sell illegal things too.
• zzo38computer 4y ago
  Technology can have good and bad uses. If they do not make them FOSS in public, then the Conspiracy will invent their own and use it for bad uses only. Furthermore, even if a program is written, you can decide not to use it; that it is written (as FOSS) means that you can read its working, now that someone else wrote about it. You can also execute it on a computer, if that is what is desired. Also, if it is well known enough, then hopefully if someone does use it deceptively against you, then you might be able to guess, or to figure it out (although it might be difficult, at least it might be possible if it is known enough). I have no intention using such a thing, but someone else might figure out uses of it. (For example, maybe there are some uses that can be used with movies, for example, if the original actor has been injured for an extended period of time (including if they are dead) or if they want to make up a picture of someone who does not exist. (Although, they should avoid being deceptive. For example, include in the credits, the mention of using such a thing.) Even if it is considered acceptable though, some people will prefer to make movies without it, and such a thing should be acceptable too anyways.) (I think even in Star Trek, in story, in some episodes they made deepfake movies of someone. And even in Star Trek, both good and bad uses are possible. Or, am I mistaken?) Nevertheless, there may be some dangers involved, but there are potential danger with anything; if you are careful, then you can try to avoid it, hopefully.
  • zzo38computer 4y ago
    Perhaps another thing that I should mention, which should be considered if an evaluation is attempting to be made, is that such a technology already exists. (I forgot to mention that earlier.) Also, as mentioned in some other comments: "Alternatively, having tools like this easily available makes it easier to raise awareness and build teams to combat them." Another thing mentioned in another comment: "The entities with interest in it have bigger pockets than some random open source project. ... There are many small entities with interest in such a technology which don't have huge budgets. Small terrorist / extremist organizations pushing a specific agenda." However, in the small case, it is perhaps not as impactful, now that it is FOSS and that it allows others to raise awareness and combat them as described by the other comment. In the big case, where others could make it independently, having this FOSS implementation helps even more against it, since they would otherwise just make it up by themself.