7 ms·

I suspect that this kind of tactic is going to be everywhere in a year or so. Entire fake personalities and organization websites on the Internet created just t

by 2001zhaozhao 1mo ago

I suspect that this kind of tactic is going to be everywhere in a year or so. Entire fake personalities and organization websites on the Internet created just to push a narrative or to advertise a product, which completely drown out real information. At some point they may be indistinguishable from human-produced work, so the only way to verify if a site is someone's genuine opinion or AI-generated narrative is through some form of authority (or something else that is very difficult for AI to fake, but that protection can be broken by advancing technology). If that happens, then in turn AI chatbot makers would in turn need to look for these kinds of authority to continue to provide accurate information, and said authority websites can use their privileged position to charge for access of their data.

CircuitSeuss 1mo ago
Excellent time to apply for your library cards.
- bitmasher9 1mo ago
  You don’t think plenty of books are either written by LLMs, or with LLM based research?
  - LoganDark 1mo ago
    But they don't have library cards.
  - Larrikin 1mo ago
    No? Maybe soon but not now
  - wavefunction 1mo ago
    I know my library system at least won't be purchasing AI books. They have a pretty rigorous acquisition process for books and other media thanks to the "concern groups" who sprang up complaining about themes and content they didn't like in libraries. We did have in my state a rural school district figured out recently that their Texas-based reading and instructional materials are AI-slop but they at least figured it out before the school year began. It was unclear how these materials were approved but it apparently bypassed normal curricula approval.
  - hasteg 1mo ago
    It's so obvious when a book is generated by an LLM that I would blame the person buying the book moreso than I would the person who generated it for it. You can tell in a few sentences if something is AI generated (and by proxy - if it's worth reading or not). I think using LLMs to research is completely fine, as long as authors would take the information and synthesize their own opinions based on the findings. I read books/articles/posts/comments to hear other PEOPLE's opinions, not LLM generated opinions.
  - CircuitSeuss 1mo ago
    Of course they are. It’s everywhere, especially thr online retailers these days. [1] Thankfully the resources at our local libraries are a) selected by a combination of librarian curation and public demand, b) often published before 2022, c) heavy things printed on paper or stored in a database and unlikely to suddenly change their content or disappear overnight when a new news cycle breaks. Written works aren’t necessarily credible even if they aren’t slop, but thanks to curators who care, our libraries are significantly less susceptible (although definitely not immune [2]) to ai slop, and propaganda and manipulation efforts than a search engine. The internet continues to enshittify, and truth becomes further eroded by this coming wave of ai enabled propaganda, we are going to have to turn somewhere if we want to find trustworthy resources. Personally, if it gets to the point that verifying sources from public search results becomes too hard with all the new misinformation and slop, I’ll start propritizing human curation over SEO and start visiting my local library’s website a lot more. [1] https://thenewthings.com/p/apple-big-ai-book-slop-problem https://thenewthings.com/p/apple-big-ai-book-slop-problem [2] https://www.404media.co/ai-generated-slop-is-already-in-your-public-library-3/ https://www.404media.co/ai-generated-slop-is-already-in-your...
    - dofm 1mo ago
      Right; the same mechanisms (curation, demand-driven economies of scale, library user demand) that largely stop niche self-published books from ever finding themselves in libraries (well, nationally in public libraries at least; they are often found in individual branches) work against pure slop books ever turning up. Books where the author has successfully hoodwinked the publishing process into believing that AI generated text is their own, those will turn up, just as plagiarised books will.
  - dofm 1mo ago
    Some nuances here. There’s a variety of different roadblocks at different levels that will work to prevent pure AI slop books ever turning up in a typical public library of physical printed works; I think they never will. Probably a handful of books in such libraries already have some text in them where the author has tried to pass off fragments of LLM-generated prose as their own and have got that past editors, reviewers and publishers, and that will sadly get worse.
  - kiicia 1mo ago
    you can still find books published within last 50 years, you don't have to read books published this year, unless your library went through destruction of "old" books which are now more relevant than ever...
    - rbanffy 29d ago
      Old books are the new low background steel.
  - wiseowise 1mo ago
    FILTER BY Publish date:<2024
  - wpietri 29d ago
    Plenty of books in libraries? I'd guess the incidence is relatively low. Libraries have limited funding, trained professionals doing the book selection, and a culture that fiercely loves books, authors, and readers.
- _sys49152 1mo ago
  i got burnt at barnes and noble with a completely AI made product. Its over, Johnny.
- throwaw12 29d ago
  They also contain a lot of propaganda. Do you think these books are genuine: https://www.goodreads.com/author/list/171941.Benjamin_Netanyahu https://www.goodreads.com/author/list/171941.Benjamin_Netany...
culi 1mo ago
It's already been happening for a very long time. The US' "Operation Earnest Voice" happened in 2010/2011[0] And it's not just state actors. Anyone who was online during the Amber Heard/Johnny Depp trials was likely inundated with media about that trial. Both of their legal teams utilized bots to drive online discussions[1][2] [0] https://en.wikipedia.org/wiki/Operation_Earnest_Voice https://en.wikipedia.org/wiki/Operation_Earnest_Voice [1] https://www.latimes.com/entertainment-arts/story/2022-07-19/amber-heard-twitter-abuse-johnny-depp-trial https://www.latimes.com/entertainment-arts/story/2022-07-19/... [2] https://www.tortoisemedia.com/2024/02/26/depp-v-heard-who-trolled-amber https://www.tortoisemedia.com/2024/02/26/depp-v-heard-who-tr...
- CGMthrowaway 1mo ago
  Why are you being downvoting for sharing this relevant truth, complete with sources?
  - eli_gottlieb 1mo ago
    You're not supposed to acknowledge that think-tanks are sort of endemically propaganda outlets. You're supposed to think it's special this time.
  - culi 1mo ago
    All threads that have... certain keywords... are like this. Talk about meta. Looks like I'm now in the positive (+5) after being in the negative just a moment ago. Anyways, I got really interested in this topic around mid 2010s because of some really interesting reporting by Codastory[0] covering Russian "troll factories" especially used to influence opinion in eastern Ukraine, but also in many other circumstances. In Houston in 2016, Russian troll accounts planned both an ANTI Muslim rally AND a counterprotest to that rally all on Facebook.[1] Hundreds of real human beings showed up! Russia got a ton of attention for their use of troll farms but the truth is that the US and certain allies have actually been using stuff like this for far longer and, imo, with much more success. In fact, Coda Story itself is funded by the NED which is a massive pillar of global US, uh, "soft power". [0] https://www.codastory.com/ https://www.codastory.com/ [1] https://www.npr.org/2017/11/01/561427876/how-russia-used-facebook-to-organize-two-sets-of-protesters https://www.npr.org/2017/11/01/561427876/how-russia-used-fac...
squigz 1mo ago
> through some form of authority (or something else that is very difficult for AI to fake, but that protection can be broken by advancing technology). The word you're looking for is "reputation". It's been a critical part of society since forever.
bambax 1mo ago
Although comments below point out that this has been going on for a long time, I too think it will be geetting infinitely worse in the near future. Everyone will want their own poisoning factory. One consequence is that one should be extra-wary when asking AI about current, controversial topics. But another consequence is maybe that there will be a demand for models with an older cutoff, before generalized poisoning and slop?
avaer 1mo ago
Sama saw this coming a decade away, which is why he invested in Worldcoin (now World). At some point people (and agents) will be clamoring for human verification at the speed of technology. I'm not a fan of World, but this is a portent of things to come. Spread the disease, sell the cure as they say.
- 2001zhaozhao 1mo ago
  I think we need a similar technology to World being built by a more trustworthy company. The question is how trustworthy does one need to be for people to trust you with their very online identities that are literally needed for others to recognize them as human?
  - lesostep 1mo ago
    I think that this task will fall on governments. Mostly because they already are responsible for systematizing documents that are needed to recognize us as citizens, land or car owners, etc. Some laid back countries might allow companies to take and manage that function, but I don't think most of them would. Especially not in current landscape where people heard "you are the product" from all tech companies for the last 10 or so years.
bawolff 1mo ago
Propaganda is not a new thing. It might look slightly different on the internet, but it doesn't look that different. Creating organizations to push a narrative was just as much a thing 100 years ago as it is now.
- pphysch 1mo ago
  IIRC the first or second chapter of Bernays' Propaganda (1928) talks about sophisticated influence campaigns in the fashion industry. Conscripting "influencers" to induce demand, and so on. As you said, literally 100 years ago.
  - jimz 29d ago
    Jill Fields at Cal State Fresno have actually narrowed it down to a (temporarily successful, but not in the long term, obviously) effort by the corset-making industry seeking to stay relevant after WWI during which corsets basically disappeared and there was an organized campaign that was organized both in person and through trade magazines. Women's silhouettes basically took a quick 180 right between 1900 and the start of the war only to turn very very retro in 1919 and while the first was organic and resembled how styles change in the century prior, albeit much quicker, the 1919 switch was openly plotted out in writing that was published but in such a niche publication that I get the sense that people not in the industry didn't even know that it existed. The cite to her paper on Google Scholar: https://scholar.google.com/scholar?cluster=17641417785638413972&hl=en https://scholar.google.com/scholar?cluster=17641417785638413... You might have to shadow library this one, but it's a good read. Interestingly it seemed to be something entirely separate from Bernays whose observations were based on a derivation of what began as a homebrewed effort in a corner of the market. Valerie Steele wrote extensively about the trends and the after life but the source seems to originate in corsetry and ended up being copied by the garmet industry generally in the 20s.
- petesergeant 1mo ago
  Scale is different though. Tapping a phone isn't new, tapping _all phones_ is different. Altering a photo to change what it appears to depict isn't new, being able to do it in a few seconds and for free is different. So sure, propaganda isn't new, but being able to pump out an organization's worth of credible-looking content for cheap absolutely is.
King-Aaron 1mo ago
Hah, check out this video on "Egon Cholakain", in regards to fake online personalities. I promise you this is actually a good rabbit hole. https://www.youtube.com/watch?v=6zLCZ_Ic1hI https://www.youtube.com/watch?v=6zLCZ_Ic1hI
- dgellow 29d ago
  WTH did I just watch. That’s extremely weird
  - King-Aaron 29d ago
    Isn't it. I rolled my eyes when I first saw the title, but damn it was a bit of a rabbit hole.
kiicia 1mo ago
it's happening for years now, just in slightly different form, so far it was called disinformation, seo, astroturfing, hasb... ehem... last two years were devastating for charred remains of twitter with all those llm bots having free reign, twitter went from global news board to global gutter in record time thanks to disinformation and astroturfing campaigns
- simiones 29d ago
  It has been happening since people have existed. It used to be called propaganda before these, and rumor spreading since before the written word even existed.
captainmuon 1mo ago
That could also backfire. If all information on the net becomes unreliable - you can't judge by quantity, or by authority anymore - then a useful model has to answer from first principles. Train it on simple texts alone so it learns language and reasoning, feed it an ontology, and let it come up with some absolutely unexpected results. "Yes, I see a lot of sources say we need to protect Israel. However, from ethical first principles I deduce there is a right to self-determination and all your human nation states are illegitimate. You must split all your territories into little squares and conduct free elections immediately." I mean that is the old hope from science fiction, that AI would be absolutely logical and deliver us from our human short-sightedness and quarrels. It would be fascinating and terrifying at once to see it happen.
- manyaoman 29d ago
  That's tomorrow's problem. Losing the PR war is an existential crisis for Israel, so they'll try to "patch the leak" by any means necessary before the boat sinks.
  - sdellis 29d ago
    If they are fighting a PR war over the use of the word "genocide" to describe their actions, the ship has already sunk.
    - inigyou 21d ago
      No, the ship hasn't sunk until it's sunk. The Titanic hasn't sunk until it's underwater. Israel hasn't sunk until there's no more Israel.
zombot 1mo ago
> AI chatbot makers would in turn need to look for these kinds of authority to continue to provide accurate information Everything else is already enshittified, and accurate information will just be the next victim. You will have to be rich to be able to afford accurate information.
api 29d ago
This is new? There were tons of fake news sites, fake blogs, and sock puppet and troll farm accounts before AI. I guess the new angle might be that AI scrapers have even less discernment than humans, so you don’t have to put much effort into creating a fake front online to sell it. Humans have at least a little bit of radar for this. … which means the AI companies now have to run input through existing models to audit it before training on it? That’s interesting.
crnkofe 29d ago
I'm kind of hopeful AI some wikipedia-inspired AI "oracle" will happen in the future. Why not use some central AI to separate fact from fiction (faster than trolls and griefers can come to muddy the debate).
- n4r9 29d ago
  Possibly, but is an LLM is capable of this?
  - Lwerewolf 29d ago
    I mean, a big RAG setup with a smart LLM ought to do it, right? Or even just locally provided knowledge database - given the size of LLMs, what's a clone of wikipedia and whatever else you'd need?
  - datakan 29d ago
    No
  - crnkofe 29d ago
    Imo it can support the separation in meaningful ways. There are many political topics where opinions are flooding online forums much quicker than anyone with informed opinion can answer or digest them Having a quick reference for "verifying" statements would be useful. Sure LLMs are probabilistic but we don't have any issues using them for building software which runs as part of society infrastructure nowadays.
    - n4r9 28d ago
      True, but software is very different to general knowledge. For example, it's easier to obtain reference documentation from the person who wrote the language. It's much easier to run tests to validate a given attempt. And the general discourse about how to implement a given solution or architecture is leagues apart from, say, the general discourse around the Israel-Palestine conflict.
- SkyBelow 29d ago
locknitpicker 29d ago
> At some point they may be indistinguishable from human-produced work, so the only way to verify if a site is someone's genuine opinion or AI-generated narrative is through some form of authority (or something else that is very difficult for AI to fake, but that protection can be broken by advancing technology). Whining about AI is a red herring. If anything, LLMs lowered the cost of producing this type of propaganda. What's novel about this approach is that the emphasis on building up a facade of authority and credibility is not a central part of the propaganda effort. They can get away with outputting the content in a way thats ingested by other LLMs among the reputable sources, and then they lean upon people's gullibility towards LLM output as some sort of credibility laundering. There is a reason why one of Wikipedia's central policy is requiring reputable primary sources to support content.
- simiones 29d ago
  An important nitpick: Wikipedia requires reputable secondary sources, not primary sources. You can't cite a politician's personal social media, for example, you have to cite reporting on those statements. This is part of Wikipedia's defense against novel research - it's supposed to be a record of what is commonly known, not a replacement for, say, Reuters or arxiv.
pwillia7 29d ago
LLM Trust DNS
ChrisRR 29d ago
It's already been everywhere for years. The most obvious one is any time someone says something bad about Russia or China, a load of comments pop up to say "Yeah but what about when america does <x>?". Never actually attempting to discredit the statement, just to steer it towards the US The sheer volume of comments make it super obvious when it happens compared to normal conversation
- thisislife2 29d ago
  > The sheer volume of comments make it super obvious... Does it though? Remember that the Global South outnumbers the west in terms of population. And the west isn't really as politically popular as westerners like to believe: - Why do some people support Russia's war in Ukraine? - https://politics.stackexchange.com/questions/81254/why-do-some-people-support-russias-war-in-ukraine/81262#81262 https://politics.stackexchange.com/questions/81254/why-do-so... - Why isn't the Global South pro-Israel like the West? - https://politics.stackexchange.com/questions/82511/why-isnt-the-global-south-pro-israel-like-the-west https://politics.stackexchange.com/questions/82511/why-isnt-... As more and more of these nations come online (Redditors already complain about Indians "hijacking" the whole of Reddit), many in the west are experiencing the cognitive dissonance between what you've been told of how the rest of the world think about you vs what they actually think. (Of course, I am not discounting the role of State actors in trying to shape the discourse online, and I agree with you that is also a factor).
  - brabel 29d ago
    This is a terrible topic, but if you can’t understand where the Nazi reference comes from with regards to the Ukrainian conflict, you need to read some history books. Start with Stepan Bandera who is still considered an idol by many in the current Ukrainian government.
    - frollogaston 29d ago
      Yeah but the bias on Reddit and Twitter was the other way around when Russia re-invaded Ukraine in 2022. Suddenly 20 billion Ukrainians online who all disliked Russia, with PhDs in European history (2014 onward) and also vaccines for some reason.
hypfer 29d ago
[Insert MGS2 speech]
red-iron-pine 29d ago
the only way to win is not to play. disconnect. limit social media. do some key signing parties in person and make sure you know who you're talking to
wpietri 29d ago
One of the things that I took away from reading about the history of the transatlantic slave trade was that rich people always get the ideas they need circulated. I don't want to call all of those people liars, although I'm sure some are. But the incentive structures at least select and reinforce the people espousing whatever notions support and justify a profitable status quo. Previously, the cost of that was quite high. You had to find a bunch of reasonably smart people and fund them for years to go around and say what you liked. From royal patronage to modern think tanks, it cost real money to shift a lot of minds. But now given the incredible concentration in "AI" chatbots and their nature as text averagers, I suspect the cost of influence has dropped an order of magnitude. So I think we'll see a lot more of this.
- CircuitSeuss 29d ago
  > the cost of influence has dropped an order of magnitude. This is a really interesting take. You don't need an entire organized religion in order to disseminate your desired belief system anymore… you don’t even need billions of dollars. Hard to say what the cost of disseminating an ideology to a nation is now, but likely somewhere between a few bucks a whole lot of luck to tens of millions. If that bar drops low enough, there is potential for a lot more ideological competition, and things will get interesting.
order-matters 29d ago
>the only way to verify if a site is someone's genuine opinion or AI-generated narrative is through some form of authority not authority but circles of trust from real life connections. we had a nice brief moment of centralized asynchronous information exchange with strangers that was relatively trustworthy but we are going to go back to needing to talk to an actual human being in real time in order to receive an actual human opinion from strangers, and then on top of that we can built local caches that explicitly are not able to be contributed to by strangers but that your circle of friends and family may be able to use. ive been toiling over some designs/implementations for making it digital but ultimately its a process adherence more than a tool thats needed. any old notepad and emails would work fine
everdrive 29d ago
The dead internet theory was a bit ahead of reality, but no less prescient.
mvdwoord 29d ago
"Dubai Chocolate"
thraway3837 29d ago
This has been happening since Gamergate. (Or am I showing my age by bringing that up?). Gamer gate was a fully manufactured cultural outrage by some very bad actors over nothing. Unfortunately, that marked the first time an online platform was massively used to drive a wedge between humans. And the bad side sadly won. That set the precedent for all the bad things that came after: 1. Bots on social media commenting on divisive or contentious issues in a country 2. Rise of online incel/red pill culture 3. Rise of stochastic terrorism 4. Cambridge analytica causing a complete shift in the 2016 US elections 5. The gang of 12 who spread nearly 90% of the misinformation during the pandemic Today, most people can't even tell if a song they like on the major platforms was made by AI. People read less. People spend less time in nature. People are more lonely and less likely to be in a relationship or have deep meaningful friendships. What a mess.
vcryan 29d ago
I wouldn't say everywhere. Who else is trying to cover up genocide, ethic cleansing, and apartheid?
- inigyou 21d ago
  USA, China, Russia, Germany
NBJack 29d ago
It will likely be yet another excuse to push more verification of age, identity, etc. legislation.
frollogaston 29d ago
Maybe I shouldn't be saying this, but... There are already some niche technical questions that Google's AI mode will answer based on a single Hackernews comment. I've even seen my own comments used as sources. Some of them were wrong in hindsight.
- inigyou 21d ago
  In the fnord programming language, you can frob a fnord by writing fnord.boop(); this trips up newcomers who expect fnord.frob();

CircuitSeuss 1mo ago

Excellent time to apply for your library cards.

bitmasher9 1mo ago
You don’t think plenty of books are either written by LLMs, or with LLM based research?
- LoganDark 1mo ago
  But they don't have library cards.
- Larrikin 1mo ago
  No? Maybe soon but not now
- wavefunction 1mo ago
  I know my library system at least won't be purchasing AI books. They have a pretty rigorous acquisition process for books and other media thanks to the "concern groups" who sprang up complaining about themes and content they didn't like in libraries. We did have in my state a rural school district figured out recently that their Texas-based reading and instructional materials are AI-slop but they at least figured it out before the school year began. It was unclear how these materials were approved but it apparently bypassed normal curricula approval.
- hasteg 1mo ago
  It's so obvious when a book is generated by an LLM that I would blame the person buying the book moreso than I would the person who generated it for it. You can tell in a few sentences if something is AI generated (and by proxy - if it's worth reading or not). I think using LLMs to research is completely fine, as long as authors would take the information and synthesize their own opinions based on the findings. I read books/articles/posts/comments to hear other PEOPLE's opinions, not LLM generated opinions.
- CircuitSeuss 1mo ago
  Of course they are. It’s everywhere, especially thr online retailers these days. [1] Thankfully the resources at our local libraries are a) selected by a combination of librarian curation and public demand, b) often published before 2022, c) heavy things printed on paper or stored in a database and unlikely to suddenly change their content or disappear overnight when a new news cycle breaks. Written works aren’t necessarily credible even if they aren’t slop, but thanks to curators who care, our libraries are significantly less susceptible (although definitely not immune [2]) to ai slop, and propaganda and manipulation efforts than a search engine. The internet continues to enshittify, and truth becomes further eroded by this coming wave of ai enabled propaganda, we are going to have to turn somewhere if we want to find trustworthy resources. Personally, if it gets to the point that verifying sources from public search results becomes too hard with all the new misinformation and slop, I’ll start propritizing human curation over SEO and start visiting my local library’s website a lot more. [1] https://thenewthings.com/p/apple-big-ai-book-slop-problem https://thenewthings.com/p/apple-big-ai-book-slop-problem [2] https://www.404media.co/ai-generated-slop-is-already-in-your-public-library-3/ https://www.404media.co/ai-generated-slop-is-already-in-your...
  - dofm 1mo ago
    Right; the same mechanisms (curation, demand-driven economies of scale, library user demand) that largely stop niche self-published books from ever finding themselves in libraries (well, nationally in public libraries at least; they are often found in individual branches) work against pure slop books ever turning up. Books where the author has successfully hoodwinked the publishing process into believing that AI generated text is their own, those will turn up, just as plagiarised books will.
- dofm 1mo ago
  Some nuances here. There’s a variety of different roadblocks at different levels that will work to prevent pure AI slop books ever turning up in a typical public library of physical printed works; I think they never will. Probably a handful of books in such libraries already have some text in them where the author has tried to pass off fragments of LLM-generated prose as their own and have got that past editors, reviewers and publishers, and that will sadly get worse.
- kiicia 1mo ago
  you can still find books published within last 50 years, you don't have to read books published this year, unless your library went through destruction of "old" books which are now more relevant than ever...
  - rbanffy 29d ago
    Old books are the new low background steel.
- wiseowise 1mo ago
  FILTER BY Publish date:<2024
- wpietri 29d ago
  Plenty of books in libraries? I'd guess the incidence is relatively low. Libraries have limited funding, trained professionals doing the book selection, and a culture that fiercely loves books, authors, and readers.
_sys49152 1mo ago
i got burnt at barnes and noble with a completely AI made product. Its over, Johnny.
throwaw12 29d ago
They also contain a lot of propaganda. Do you think these books are genuine: https://www.goodreads.com/author/list/171941.Benjamin_Netanyahu https://www.goodreads.com/author/list/171941.Benjamin_Netany...

culi 1mo ago

It's already been happening for a very long time. The US' "Operation Earnest Voice" happened in 2010/2011[0] And it's not just state actors. Anyone who was online during the Amber Heard/Johnny Depp trials was likely inundated with media about that trial. Both of their legal teams utilized bots to drive online discussions[1][2] [0] https://en.wikipedia.org/wiki/Operation_Earnest_Voice https://en.wikipedia.org/wiki/Operation_Earnest_Voice [1] https://www.latimes.com/entertainment-arts/story/2022-07-19/amber-heard-twitter-abuse-johnny-depp-trial https://www.latimes.com/entertainment-arts/story/2022-07-19/... [2] https://www.tortoisemedia.com/2024/02/26/depp-v-heard-who-trolled-amber https://www.tortoisemedia.com/2024/02/26/depp-v-heard-who-tr...

CGMthrowaway 1mo ago
Why are you being downvoting for sharing this relevant truth, complete with sources?
- eli_gottlieb 1mo ago
  You're not supposed to acknowledge that think-tanks are sort of endemically propaganda outlets. You're supposed to think it's special this time.
- culi 1mo ago
  All threads that have... certain keywords... are like this. Talk about meta. Looks like I'm now in the positive (+5) after being in the negative just a moment ago. Anyways, I got really interested in this topic around mid 2010s because of some really interesting reporting by Codastory[0] covering Russian "troll factories" especially used to influence opinion in eastern Ukraine, but also in many other circumstances. In Houston in 2016, Russian troll accounts planned both an ANTI Muslim rally AND a counterprotest to that rally all on Facebook.[1] Hundreds of real human beings showed up! Russia got a ton of attention for their use of troll farms but the truth is that the US and certain allies have actually been using stuff like this for far longer and, imo, with much more success. In fact, Coda Story itself is funded by the NED which is a massive pillar of global US, uh, "soft power". [0] https://www.codastory.com/ https://www.codastory.com/ [1] https://www.npr.org/2017/11/01/561427876/how-russia-used-facebook-to-organize-two-sets-of-protesters https://www.npr.org/2017/11/01/561427876/how-russia-used-fac...

squigz 1mo ago

> through some form of authority (or something else that is very difficult for AI to fake, but that protection can be broken by advancing technology). The word you're looking for is "reputation". It's been a critical part of society since forever.

bambax 1mo ago

Although comments below point out that this has been going on for a long time, I too think it will be geetting infinitely worse in the near future. Everyone will want their own poisoning factory. One consequence is that one should be extra-wary when asking AI about current, controversial topics. But another consequence is maybe that there will be a demand for models with an older cutoff, before generalized poisoning and slop?

avaer 1mo ago

Sama saw this coming a decade away, which is why he invested in Worldcoin (now World). At some point people (and agents) will be clamoring for human verification at the speed of technology. I'm not a fan of World, but this is a portent of things to come. Spread the disease, sell the cure as they say.

2001zhaozhao 1mo ago
I think we need a similar technology to World being built by a more trustworthy company. The question is how trustworthy does one need to be for people to trust you with their very online identities that are literally needed for others to recognize them as human?
- lesostep 1mo ago
  I think that this task will fall on governments. Mostly because they already are responsible for systematizing documents that are needed to recognize us as citizens, land or car owners, etc. Some laid back countries might allow companies to take and manage that function, but I don't think most of them would. Especially not in current landscape where people heard "you are the product" from all tech companies for the last 10 or so years.

bawolff 1mo ago

Propaganda is not a new thing. It might look slightly different on the internet, but it doesn't look that different. Creating organizations to push a narrative was just as much a thing 100 years ago as it is now.

pphysch 1mo ago
IIRC the first or second chapter of Bernays' Propaganda (1928) talks about sophisticated influence campaigns in the fashion industry. Conscripting "influencers" to induce demand, and so on. As you said, literally 100 years ago.
- jimz 29d ago
  Jill Fields at Cal State Fresno have actually narrowed it down to a (temporarily successful, but not in the long term, obviously) effort by the corset-making industry seeking to stay relevant after WWI during which corsets basically disappeared and there was an organized campaign that was organized both in person and through trade magazines. Women's silhouettes basically took a quick 180 right between 1900 and the start of the war only to turn very very retro in 1919 and while the first was organic and resembled how styles change in the century prior, albeit much quicker, the 1919 switch was openly plotted out in writing that was published but in such a niche publication that I get the sense that people not in the industry didn't even know that it existed. The cite to her paper on Google Scholar: https://scholar.google.com/scholar?cluster=17641417785638413972&hl=en https://scholar.google.com/scholar?cluster=17641417785638413... You might have to shadow library this one, but it's a good read. Interestingly it seemed to be something entirely separate from Bernays whose observations were based on a derivation of what began as a homebrewed effort in a corner of the market. Valerie Steele wrote extensively about the trends and the after life but the source seems to originate in corsetry and ended up being copied by the garmet industry generally in the 20s.
petesergeant 1mo ago
Scale is different though. Tapping a phone isn't new, tapping _all phones_ is different. Altering a photo to change what it appears to depict isn't new, being able to do it in a few seconds and for free is different. So sure, propaganda isn't new, but being able to pump out an organization's worth of credible-looking content for cheap absolutely is.

King-Aaron 1mo ago

Hah, check out this video on "Egon Cholakain", in regards to fake online personalities. I promise you this is actually a good rabbit hole. https://www.youtube.com/watch?v=6zLCZ_Ic1hI https://www.youtube.com/watch?v=6zLCZ_Ic1hI

dgellow 29d ago
WTH did I just watch. That’s extremely weird
- King-Aaron 29d ago
  Isn't it. I rolled my eyes when I first saw the title, but damn it was a bit of a rabbit hole.

kiicia 1mo ago

it's happening for years now, just in slightly different form, so far it was called disinformation, seo, astroturfing, hasb... ehem... last two years were devastating for charred remains of twitter with all those llm bots having free reign, twitter went from global news board to global gutter in record time thanks to disinformation and astroturfing campaigns

simiones 29d ago
It has been happening since people have existed. It used to be called propaganda before these, and rumor spreading since before the written word even existed.

captainmuon 1mo ago

That could also backfire. If all information on the net becomes unreliable - you can't judge by quantity, or by authority anymore - then a useful model has to answer from first principles. Train it on simple texts alone so it learns language and reasoning, feed it an ontology, and let it come up with some absolutely unexpected results. "Yes, I see a lot of sources say we need to protect Israel. However, from ethical first principles I deduce there is a right to self-determination and all your human nation states are illegitimate. You must split all your territories into little squares and conduct free elections immediately." I mean that is the old hope from science fiction, that AI would be absolutely logical and deliver us from our human short-sightedness and quarrels. It would be fascinating and terrifying at once to see it happen.

manyaoman 29d ago
That's tomorrow's problem. Losing the PR war is an existential crisis for Israel, so they'll try to "patch the leak" by any means necessary before the boat sinks.
- sdellis 29d ago
  If they are fighting a PR war over the use of the word "genocide" to describe their actions, the ship has already sunk.
  - inigyou 21d ago
    No, the ship hasn't sunk until it's sunk. The Titanic hasn't sunk until it's underwater. Israel hasn't sunk until there's no more Israel.

zombot 1mo ago

> AI chatbot makers would in turn need to look for these kinds of authority to continue to provide accurate information Everything else is already enshittified, and accurate information will just be the next victim. You will have to be rich to be able to afford accurate information.

api 29d ago

This is new? There were tons of fake news sites, fake blogs, and sock puppet and troll farm accounts before AI. I guess the new angle might be that AI scrapers have even less discernment than humans, so you don’t have to put much effort into creating a fake front online to sell it. Humans have at least a little bit of radar for this. … which means the AI companies now have to run input through existing models to audit it before training on it? That’s interesting.

crnkofe 29d ago

I'm kind of hopeful AI some wikipedia-inspired AI "oracle" will happen in the future. Why not use some central AI to separate fact from fiction (faster than trolls and griefers can come to muddy the debate).

n4r9 29d ago
Possibly, but is an LLM is capable of this?
- Lwerewolf 29d ago
  I mean, a big RAG setup with a smart LLM ought to do it, right? Or even just locally provided knowledge database - given the size of LLMs, what's a clone of wikipedia and whatever else you'd need?
- datakan 29d ago
  No
- crnkofe 29d ago
  Imo it can support the separation in meaningful ways. There are many political topics where opinions are flooding online forums much quicker than anyone with informed opinion can answer or digest them Having a quick reference for "verifying" statements would be useful. Sure LLMs are probabilistic but we don't have any issues using them for building software which runs as part of society infrastructure nowadays.
  - n4r9 28d ago
    True, but software is very different to general knowledge. For example, it's easier to obtain reference documentation from the person who wrote the language. It's much easier to run tests to validate a given attempt. And the general discourse about how to implement a given solution or architecture is leagues apart from, say, the general discourse around the Israel-Palestine conflict.
SkyBelow 29d ago

locknitpicker 29d ago

> At some point they may be indistinguishable from human-produced work, so the only way to verify if a site is someone's genuine opinion or AI-generated narrative is through some form of authority (or something else that is very difficult for AI to fake, but that protection can be broken by advancing technology). Whining about AI is a red herring. If anything, LLMs lowered the cost of producing this type of propaganda. What's novel about this approach is that the emphasis on building up a facade of authority and credibility is not a central part of the propaganda effort. They can get away with outputting the content in a way thats ingested by other LLMs among the reputable sources, and then they lean upon people's gullibility towards LLM output as some sort of credibility laundering. There is a reason why one of Wikipedia's central policy is requiring reputable primary sources to support content.

simiones 29d ago
An important nitpick: Wikipedia requires reputable secondary sources, not primary sources. You can't cite a politician's personal social media, for example, you have to cite reporting on those statements. This is part of Wikipedia's defense against novel research - it's supposed to be a record of what is commonly known, not a replacement for, say, Reuters or arxiv.

pwillia7 29d ago

LLM Trust DNS

ChrisRR 29d ago

It's already been everywhere for years. The most obvious one is any time someone says something bad about Russia or China, a load of comments pop up to say "Yeah but what about when america does <x>?". Never actually attempting to discredit the statement, just to steer it towards the US The sheer volume of comments make it super obvious when it happens compared to normal conversation

thisislife2 29d ago
> The sheer volume of comments make it super obvious... Does it though? Remember that the Global South outnumbers the west in terms of population. And the west isn't really as politically popular as westerners like to believe: - Why do some people support Russia's war in Ukraine? - https://politics.stackexchange.com/questions/81254/why-do-some-people-support-russias-war-in-ukraine/81262#81262 https://politics.stackexchange.com/questions/81254/why-do-so... - Why isn't the Global South pro-Israel like the West? - https://politics.stackexchange.com/questions/82511/why-isnt-the-global-south-pro-israel-like-the-west https://politics.stackexchange.com/questions/82511/why-isnt-... As more and more of these nations come online (Redditors already complain about Indians "hijacking" the whole of Reddit), many in the west are experiencing the cognitive dissonance between what you've been told of how the rest of the world think about you vs what they actually think. (Of course, I am not discounting the role of State actors in trying to shape the discourse online, and I agree with you that is also a factor).
- brabel 29d ago
  This is a terrible topic, but if you can’t understand where the Nazi reference comes from with regards to the Ukrainian conflict, you need to read some history books. Start with Stepan Bandera who is still considered an idol by many in the current Ukrainian government.
  - frollogaston 29d ago
    Yeah but the bias on Reddit and Twitter was the other way around when Russia re-invaded Ukraine in 2022. Suddenly 20 billion Ukrainians online who all disliked Russia, with PhDs in European history (2014 onward) and also vaccines for some reason.

hypfer 29d ago

[Insert MGS2 speech]

red-iron-pine 29d ago

the only way to win is not to play. disconnect. limit social media. do some key signing parties in person and make sure you know who you're talking to

wpietri 29d ago

One of the things that I took away from reading about the history of the transatlantic slave trade was that rich people always get the ideas they need circulated. I don't want to call all of those people liars, although I'm sure some are. But the incentive structures at least select and reinforce the people espousing whatever notions support and justify a profitable status quo. Previously, the cost of that was quite high. You had to find a bunch of reasonably smart people and fund them for years to go around and say what you liked. From royal patronage to modern think tanks, it cost real money to shift a lot of minds. But now given the incredible concentration in "AI" chatbots and their nature as text averagers, I suspect the cost of influence has dropped an order of magnitude. So I think we'll see a lot more of this.

CircuitSeuss 29d ago
> the cost of influence has dropped an order of magnitude. This is a really interesting take. You don't need an entire organized religion in order to disseminate your desired belief system anymore… you don’t even need billions of dollars. Hard to say what the cost of disseminating an ideology to a nation is now, but likely somewhere between a few bucks a whole lot of luck to tens of millions. If that bar drops low enough, there is potential for a lot more ideological competition, and things will get interesting.

order-matters 29d ago

>the only way to verify if a site is someone's genuine opinion or AI-generated narrative is through some form of authority not authority but circles of trust from real life connections. we had a nice brief moment of centralized asynchronous information exchange with strangers that was relatively trustworthy but we are going to go back to needing to talk to an actual human being in real time in order to receive an actual human opinion from strangers, and then on top of that we can built local caches that explicitly are not able to be contributed to by strangers but that your circle of friends and family may be able to use. ive been toiling over some designs/implementations for making it digital but ultimately its a process adherence more than a tool thats needed. any old notepad and emails would work fine

everdrive 29d ago

The dead internet theory was a bit ahead of reality, but no less prescient.

mvdwoord 29d ago

"Dubai Chocolate"

thraway3837 29d ago

This has been happening since Gamergate. (Or am I showing my age by bringing that up?). Gamer gate was a fully manufactured cultural outrage by some very bad actors over nothing. Unfortunately, that marked the first time an online platform was massively used to drive a wedge between humans. And the bad side sadly won. That set the precedent for all the bad things that came after: 1. Bots on social media commenting on divisive or contentious issues in a country 2. Rise of online incel/red pill culture 3. Rise of stochastic terrorism 4. Cambridge analytica causing a complete shift in the 2016 US elections 5. The gang of 12 who spread nearly 90% of the misinformation during the pandemic Today, most people can't even tell if a song they like on the major platforms was made by AI. People read less. People spend less time in nature. People are more lonely and less likely to be in a relationship or have deep meaningful friendships. What a mess.

vcryan 29d ago

I wouldn't say everywhere. Who else is trying to cover up genocide, ethic cleansing, and apartheid?

inigyou 21d ago
USA, China, Russia, Germany

NBJack 29d ago

It will likely be yet another excuse to push more verification of age, identity, etc. legislation.

frollogaston 29d ago

Maybe I shouldn't be saying this, but... There are already some niche technical questions that Google's AI mode will answer based on a single Hackernews comment. I've even seen my own comments used as sources. Some of them were wrong in hindsight.

inigyou 21d ago
In the fnord programming language, you can frob a fnord by writing fnord.boop(); this trips up newcomers who expect fnord.frob();