9 ms·

People who comment on these moderator issues always overlook the huge huge mental red flag that these moderators always discuss: images of children being molest

People who comment on these moderator issues always overlook the huge huge mental red flag that these moderators always discuss: images of children being molested and tortured. Endless images of young children. Being sexually assaulted, penetrated, made to do degrading things. Think about how that statement makes you feel, and then imagine you’ve been tricked into looking at this stuff for money as a “content moderator.” It’s a form of cruel economic punishment. Facebook wouldn’t have these problems if they weren’t obsessed with acquiring as many users as possible. They are the ones who should be forced to pay for the long term mental health of these workers. California ballot initiative, anyone?

commandlinefan 5y ago
I'm shocked by the implication that there's so much of it - that there's enough to damage somebody's mental health. I'm sure numbers are hard to come by, but I'd have guessed that there wouldn't be more than a couple of these cases per month - the problem can't be that widespread, or you'd assume it would spill over into other areas as well.
- mbg721 5y ago
  There doesn't have to be that much of it created, if it can be copied for free, and someone is willing to automate posting it. Cynical-me totally believes there are enough people doing that to traumatize a human content-sorter.
  - carlhjerpe 5y ago
    YouTubes contentID system is quite good at detecting just a couple seconds of sound to determine if something is copyrighted or not. It should be quite easy to match copies of content and just get rid of it straight away, meaning they don't have to see duplicates. I would assume they already do this. Which leads me to believe there's actually that much content being produced, I imagine most of it is not produced in the West though.
    - larrywright 5y ago
      There is a national database of child porn content that most big companies that host video and photos use (Facebook, Twitter, YouTube, et al). It allows them to filter the known content. I don’t know how fast that gets updated though. The larger issue is the new content. Facebook makes it very easy to take and share video, and makes producing and sharing this content easier.
- diamondhandle 5y ago
  It’s way, way more common than you’d ever want to believe. Here’s a starting point, from one very repressed culture where the act of going to the police puts great shame on your family and turns you into an outcast: > There seems to be no sign of decline in the number of sexual abuse cases against children. According to the Ministry of Health, Labor and Welfare, child consultation centers nationwide responded to 2,077 cases of child sexual abuse in fiscal 2019, up 2.75 times from fiscal 2000, when the Act on the Prevention, etc. of Child Abuse went into force. In 1,056 of these cases, the perpetrators were biological fathers. https://mainichi.jp/english/articles/20210322/p2a/00m/0na/014000c https://mainichi.jp/english/articles/20210322/p2a/00m/0na/01... Please, listen to children and believe their stories.
- wiedelphine 5y ago
  It feels like you are suggesting that under a certain amount looking at images of children being sexually abused or tortured shouldn't have an impact on mental health? For me personally I think even a single image like that would haunt me.
  - pdpi 5y ago
    Seeing one image once will have an impact but probably allow you to still lead a normal life after that. Spending all day looking at that sort of thing over an extended period of time is much more likely to have serious long-lasting effects. I don’t think there’s a magical line where it suddenly has an impact on mental health, and the details will vary from person to person, but there is most certainly a point where you go from functioning adult to justifying a formal diagnosis of PTSD or something similar.
- samstave 5y ago
  Are you kidding me??? When I was at FB ~2012+ -- I commuted in with one of the content mods on the CP/abuse -- and shit I heard from him was HORRIFIC. and there is TONS of it. And with all the pedo-network shit coming to light these days.. child abuse is WAY WAY WAY worse than anyone can imagine on this planet. Ill take your downvotes without comment as a tacit acquiescence to child abuse.
  - diamondhandle 5y ago
    I am really glad you posted this, as I’ve heard similar from people within the org. It becomes hard to justify working on such products when you know you’re perpetuating so much harm, with zero interest from leadership in resolving the issues. You might be getting downvoted by your former colleagues. Would you be surprised?
  - commandlinefan 5y ago
    That's surprising to me. I believe you, but I'm surprised. I would have assumed that this information would have been turned over to the police and the perpetrators whittled out of the general population in short order.
    - samstave 5y ago
      They do turn stuff over, I have no idea of what % is though... They should be transparent about it, IMO - as we really need a growing outcry on child safety rights. We constantly talk about the rights and protections for all these various groups, yet, in really really dire situations like this, sex trafficking etc, by comparison is way more important than kvetching over identity pronouns and such. We need a global anti-pedo-abuse force.
- sethammons 5y ago
  I don’t know about you, but I’ve accidentally came across a few pics on an old hard drive I was given second hand. I couldn’t even use my computer for a couple of weeks and I can still feel the revulsion decades later.
  - sethammons 5y ago
    I'll burn some karma here: I know one is not supposed to question down votes, but here it goes. Why would someone down vote my comment? If someone doesn't think it adds to the narrative, skip the vote. If you feel it actively detracts or presents a false narrative, down vote. > I'm shocked by the implication that there's so much of it - that there's enough to damage somebody's mental health And I reply that just a few images had an outsized impact on me. And down voted. Normally, anecdotal evidence is replied to with the general response that "the plural of anecdote is not data." Maybe the down note is that I'm not referencing mental health? Am I supposed to connect all the dots in my reply and go from "if a few images can have a visceral impact on me, that it is within reason that much more exposure especially over time could lead to mental health issues"?
    - Anthony-G 5y ago
      It seems like some of your sibling comments have similarly been down-voted. God knows why. I’ve upvoted yours and theirs because I consider personal anecdotes to be useful contributions to discussion. If discussions were to be based solely on peer-reviewed meta-analyses of empirical data, they would be poorer for missing the colour and detail – and far, far drier.
- pdpi 5y ago
  Low frequency outcomes over large number of events can produce surprising numbers. 0.01% of 2B people is still 200k people. If each of them posts a single image once a year you’re still going to get around 600 pictures of child abuse per day.
- timthorn 5y ago
  Here's a currently open job role for a content assessor at the Internet Watch Foundation: https://www.iwf.org.uk/what-we-do/who-we-are/work-us https://www.iwf.org.uk/what-we-do/who-we-are/work-us The stats there say they've dealt with 1.2m reports over 23 years, with 560,000 takedowns. That's certainly more than a couple a month.
carlhjerpe 5y ago
I don't understand why on earth anyone would use Facebook as a platform for sharing this kind of content, they know It'll be removed and that Facebook is quite good at tracking you, so unless you're using a VM connected to a VPN with a browser soley for this content you'll have the police called on you? It just doesn't make sense to me. Also, why is it that facebooks content moderators have to look at everything while Google just removed content and says nothing more about it? Does Google have more confidence in their AI while Facebook doesn't and therefore need people to manually check it? Or is it that there might be more content on Facebook uploaded that an AI would misinterpret as child abuse (baby photos and the likes that people seem so desperate to share on Facebook these days).
- rscoots 5y ago
  This podcast from last year answers all these questions. Although the answers are not encouraging... Very eye-opening stuff: https://youtu.be/qv_hokG2oSo https://youtu.be/qv_hokG2oSo
- fighterpilot 5y ago
  They use a number of platforms for this but FB is quite aggressive about cracking down on it, unlike AWS among others who mostly turn a blind eye.
chrisjc 5y ago
> They are the ones who should be forced to pay for the long term mental health of these workers. I don't disagree... However, this doesn't do anything to address the issue that someone still has to sift through this disturbing content, and society as a whole probably still pays the price. I don't think any sort of compensation or mental help would prevent the kinds of side effects this sort of role is exposed to.
- mc32 5y ago
  I don’t know why FB believes it should aim to be the repository for all of everyone’s media. Most photos and videos are crappy. They could throttle it to 10 per person per day and increase quality. I’m sure people would complain and FB would think they lose out to a competitor with unlimited uploads. However, a reduced load means easier job to address the bad stuff. It’s probably unworkable, but making people more discerning rather than less isn’t a bad thing.
stickfigure 5y ago
You know who else has to deal with these images - and much worse, real-world living cases? Law enforcement. Lawyers. Juries. Do we just sit back and say "let's stop subjecting those poor officers from having to see evil things"? Of course not. We hire people with strong stomachs and a strong sense of duty. Not everyone who sees this stuff is scarred for life. See also: Soldiers. Firefighters. Emergency medical workers. Personally, I'd rather moderate gross things than, say, work in a coal mine. But that's me. I don't appreciate the insinuation that this makes me some sort of monster.
- ceejayoz 5y ago
  "We hire people with strong stomachs and a strong sense of duty. Not everyone who sees this stuff is scarred for life." https://www.theatlantic.com/national/archive/2014/09/police-officers-who-hit-their-wives-or-girlfriends/380329/ https://www.theatlantic.com/national/archive/2014/09/police-... > Two studies have found that at least 40 percent of police officer families experience domestic violence, in contrast to 10 percent of families in the general population. A third study of older and more experienced officers found a rate of 24 percent, indicating that domestic violence is two to four times more common among police families than American families in general.
  - baumy 5y ago
    Those studies have been thoroughly discredited. Not only do they qualify things like "shouting" and "loss of temper" as domestic violence (which of course do not meet the legal threshold for that term), they have never been able to be reproduced. When the same exact researchers attempted to reproduce, they got a wildly different rate of 24% using the exact same methodology. Other researchers using better methodology and appropriate definitions of "domestic violence" have found numbers generally between 7% and 11%. The 40% study itself even says that the victims reported a 10% rate of physical domestic violence from their partner, but specifically does not indicate who the aggressor is - the officer or their spouse. The study is also 30+ years old at this point. [0] More recent research from 2009, which imo has a far superior methodology and a larger sample size, says that "over 87 percent of officers reported never having engaged in physical domestic violence in their lifetime" [1] Other studies [2] [3] report rates between about 2% and 11% for mixtures of one time and recurring domestic violence, which are in line with population averages. [0] https://babel.hathitrust.org/cgi/pt?id=umn.31951003089863c https://babel.hathitrust.org/cgi/pt?id=umn.31951003089863c [1] http://scholarcommons.usf.edu/etd/1862 http://scholarcommons.usf.edu/etd/1862 [2] http://webapp1.dlib.indiana.edu/virtual_disk_library/index.cgi/4951188/FID707/Root/New/030PG297.PDF http://webapp1.dlib.indiana.edu/virtual_disk_library/index.c... [3] https://digitalcommons.law.umaryland.edu/cgi/viewcontent.cgi?article=2519&context=fac_pubs https://digitalcommons.law.umaryland.edu/cgi/viewcontent.cgi...
coryrc 5y ago
Why not hire the sickos who like these things to do the filtering?
nonbirithm 5y ago
> Facebook wouldn’t have these problems if they weren’t obsessed with acquiring as many users as possible. Is there any venture capitalist that would generally believe "there is value in not trying to connect everyone on Earth together?" It feels like when you market your hot startup idea, connecting the world or enacting change is merely a cool new thing that carries no long-term downside. Find a market, connect with a target audience, grow. Maybe that's how Facebook started (or not). I feel that, because of human nature, if it weren't Facebook then some other company would have inevitably taken its place. Being able to connect to someone faster and easier than ever sounds like a tangible gain for average people. The idea, with no strings attached, was too tempting to pass up for someone that truly wants to change the world and be a hero of some kind. It worked for a single university, after all. It's the sense of starting a technology company when the future is completely unknown and the potential for change is exciting. I'm not sure there was anyone that would have understood the consequences of taking such an idea to its logical conclusion, and now Facebook has become irreversibly baked into human culture. The more people that want to connect, the more moderation staff is needed to match. Unfortunately, the desire to connect with others is wired into pretty much everyone on Earth at the deepest level (with exceptions, of course), and so your potential market is... pretty much everyone on Earth.
nitwit005 5y ago
> Facebook wouldn’t have these problems if they weren’t obsessed with acquiring as many users as possible. Even small sites have people upload child porn and other stuff into them, and some IT guy has to go handle it. The only thing that can really reduce the scale of this stuff is for the people doing it to suffer a consequence like jail time. That's clearly not happening in most cases, despite Facebook and others sharing information with the authorities. The same is true for other issues like people being hounded by death/rape threats and the like online. The police mostly ignore it as an issue, so moderators have to read a bunch of it, and the people doing it just get a suspension or ban from the one platform.

commandlinefan 5y ago

I'm shocked by the implication that there's so much of it - that there's enough to damage somebody's mental health. I'm sure numbers are hard to come by, but I'd have guessed that there wouldn't be more than a couple of these cases per month - the problem can't be that widespread, or you'd assume it would spill over into other areas as well.

mbg721 5y ago
There doesn't have to be that much of it created, if it can be copied for free, and someone is willing to automate posting it. Cynical-me totally believes there are enough people doing that to traumatize a human content-sorter.
- carlhjerpe 5y ago
  YouTubes contentID system is quite good at detecting just a couple seconds of sound to determine if something is copyrighted or not. It should be quite easy to match copies of content and just get rid of it straight away, meaning they don't have to see duplicates. I would assume they already do this. Which leads me to believe there's actually that much content being produced, I imagine most of it is not produced in the West though.
  - larrywright 5y ago
    There is a national database of child porn content that most big companies that host video and photos use (Facebook, Twitter, YouTube, et al). It allows them to filter the known content. I don’t know how fast that gets updated though. The larger issue is the new content. Facebook makes it very easy to take and share video, and makes producing and sharing this content easier.
diamondhandle 5y ago
It’s way, way more common than you’d ever want to believe. Here’s a starting point, from one very repressed culture where the act of going to the police puts great shame on your family and turns you into an outcast: > There seems to be no sign of decline in the number of sexual abuse cases against children. According to the Ministry of Health, Labor and Welfare, child consultation centers nationwide responded to 2,077 cases of child sexual abuse in fiscal 2019, up 2.75 times from fiscal 2000, when the Act on the Prevention, etc. of Child Abuse went into force. In 1,056 of these cases, the perpetrators were biological fathers. https://mainichi.jp/english/articles/20210322/p2a/00m/0na/014000c https://mainichi.jp/english/articles/20210322/p2a/00m/0na/01... Please, listen to children and believe their stories.
wiedelphine 5y ago
It feels like you are suggesting that under a certain amount looking at images of children being sexually abused or tortured shouldn't have an impact on mental health? For me personally I think even a single image like that would haunt me.
- pdpi 5y ago
  Seeing one image once will have an impact but probably allow you to still lead a normal life after that. Spending all day looking at that sort of thing over an extended period of time is much more likely to have serious long-lasting effects. I don’t think there’s a magical line where it suddenly has an impact on mental health, and the details will vary from person to person, but there is most certainly a point where you go from functioning adult to justifying a formal diagnosis of PTSD or something similar.
samstave 5y ago
Are you kidding me??? When I was at FB ~2012+ -- I commuted in with one of the content mods on the CP/abuse -- and shit I heard from him was HORRIFIC. and there is TONS of it. And with all the pedo-network shit coming to light these days.. child abuse is WAY WAY WAY worse than anyone can imagine on this planet. Ill take your downvotes without comment as a tacit acquiescence to child abuse.
- diamondhandle 5y ago
  I am really glad you posted this, as I’ve heard similar from people within the org. It becomes hard to justify working on such products when you know you’re perpetuating so much harm, with zero interest from leadership in resolving the issues. You might be getting downvoted by your former colleagues. Would you be surprised?
- commandlinefan 5y ago
  That's surprising to me. I believe you, but I'm surprised. I would have assumed that this information would have been turned over to the police and the perpetrators whittled out of the general population in short order.
  - samstave 5y ago
    They do turn stuff over, I have no idea of what % is though... They should be transparent about it, IMO - as we really need a growing outcry on child safety rights. We constantly talk about the rights and protections for all these various groups, yet, in really really dire situations like this, sex trafficking etc, by comparison is way more important than kvetching over identity pronouns and such. We need a global anti-pedo-abuse force.
sethammons 5y ago
I don’t know about you, but I’ve accidentally came across a few pics on an old hard drive I was given second hand. I couldn’t even use my computer for a couple of weeks and I can still feel the revulsion decades later.
- sethammons 5y ago
  I'll burn some karma here: I know one is not supposed to question down votes, but here it goes. Why would someone down vote my comment? If someone doesn't think it adds to the narrative, skip the vote. If you feel it actively detracts or presents a false narrative, down vote. > I'm shocked by the implication that there's so much of it - that there's enough to damage somebody's mental health And I reply that just a few images had an outsized impact on me. And down voted. Normally, anecdotal evidence is replied to with the general response that "the plural of anecdote is not data." Maybe the down note is that I'm not referencing mental health? Am I supposed to connect all the dots in my reply and go from "if a few images can have a visceral impact on me, that it is within reason that much more exposure especially over time could lead to mental health issues"?
  - Anthony-G 5y ago
    It seems like some of your sibling comments have similarly been down-voted. God knows why. I’ve upvoted yours and theirs because I consider personal anecdotes to be useful contributions to discussion. If discussions were to be based solely on peer-reviewed meta-analyses of empirical data, they would be poorer for missing the colour and detail – and far, far drier.
pdpi 5y ago
Low frequency outcomes over large number of events can produce surprising numbers. 0.01% of 2B people is still 200k people. If each of them posts a single image once a year you’re still going to get around 600 pictures of child abuse per day.
timthorn 5y ago
Here's a currently open job role for a content assessor at the Internet Watch Foundation: https://www.iwf.org.uk/what-we-do/who-we-are/work-us https://www.iwf.org.uk/what-we-do/who-we-are/work-us The stats there say they've dealt with 1.2m reports over 23 years, with 560,000 takedowns. That's certainly more than a couple a month.

carlhjerpe 5y ago

I don't understand why on earth anyone would use Facebook as a platform for sharing this kind of content, they know It'll be removed and that Facebook is quite good at tracking you, so unless you're using a VM connected to a VPN with a browser soley for this content you'll have the police called on you? It just doesn't make sense to me. Also, why is it that facebooks content moderators have to look at everything while Google just removed content and says nothing more about it? Does Google have more confidence in their AI while Facebook doesn't and therefore need people to manually check it? Or is it that there might be more content on Facebook uploaded that an AI would misinterpret as child abuse (baby photos and the likes that people seem so desperate to share on Facebook these days).

rscoots 5y ago
This podcast from last year answers all these questions. Although the answers are not encouraging... Very eye-opening stuff: https://youtu.be/qv_hokG2oSo https://youtu.be/qv_hokG2oSo
fighterpilot 5y ago
They use a number of platforms for this but FB is quite aggressive about cracking down on it, unlike AWS among others who mostly turn a blind eye.

chrisjc 5y ago

> They are the ones who should be forced to pay for the long term mental health of these workers. I don't disagree... However, this doesn't do anything to address the issue that someone still has to sift through this disturbing content, and society as a whole probably still pays the price. I don't think any sort of compensation or mental help would prevent the kinds of side effects this sort of role is exposed to.

mc32 5y ago
I don’t know why FB believes it should aim to be the repository for all of everyone’s media. Most photos and videos are crappy. They could throttle it to 10 per person per day and increase quality. I’m sure people would complain and FB would think they lose out to a competitor with unlimited uploads. However, a reduced load means easier job to address the bad stuff. It’s probably unworkable, but making people more discerning rather than less isn’t a bad thing.

stickfigure 5y ago

You know who else has to deal with these images - and much worse, real-world living cases? Law enforcement. Lawyers. Juries. Do we just sit back and say "let's stop subjecting those poor officers from having to see evil things"? Of course not. We hire people with strong stomachs and a strong sense of duty. Not everyone who sees this stuff is scarred for life. See also: Soldiers. Firefighters. Emergency medical workers. Personally, I'd rather moderate gross things than, say, work in a coal mine. But that's me. I don't appreciate the insinuation that this makes me some sort of monster.

ceejayoz 5y ago
"We hire people with strong stomachs and a strong sense of duty. Not everyone who sees this stuff is scarred for life." https://www.theatlantic.com/national/archive/2014/09/police-officers-who-hit-their-wives-or-girlfriends/380329/ https://www.theatlantic.com/national/archive/2014/09/police-... > Two studies have found that at least 40 percent of police officer families experience domestic violence, in contrast to 10 percent of families in the general population. A third study of older and more experienced officers found a rate of 24 percent, indicating that domestic violence is two to four times more common among police families than American families in general.
- baumy 5y ago
  Those studies have been thoroughly discredited. Not only do they qualify things like "shouting" and "loss of temper" as domestic violence (which of course do not meet the legal threshold for that term), they have never been able to be reproduced. When the same exact researchers attempted to reproduce, they got a wildly different rate of 24% using the exact same methodology. Other researchers using better methodology and appropriate definitions of "domestic violence" have found numbers generally between 7% and 11%. The 40% study itself even says that the victims reported a 10% rate of physical domestic violence from their partner, but specifically does not indicate who the aggressor is - the officer or their spouse. The study is also 30+ years old at this point. [0] More recent research from 2009, which imo has a far superior methodology and a larger sample size, says that "over 87 percent of officers reported never having engaged in physical domestic violence in their lifetime" [1] Other studies [2] [3] report rates between about 2% and 11% for mixtures of one time and recurring domestic violence, which are in line with population averages. [0] https://babel.hathitrust.org/cgi/pt?id=umn.31951003089863c https://babel.hathitrust.org/cgi/pt?id=umn.31951003089863c [1] http://scholarcommons.usf.edu/etd/1862 http://scholarcommons.usf.edu/etd/1862 [2] http://webapp1.dlib.indiana.edu/virtual_disk_library/index.cgi/4951188/FID707/Root/New/030PG297.PDF http://webapp1.dlib.indiana.edu/virtual_disk_library/index.c... [3] https://digitalcommons.law.umaryland.edu/cgi/viewcontent.cgi?article=2519&context=fac_pubs https://digitalcommons.law.umaryland.edu/cgi/viewcontent.cgi...

coryrc 5y ago

Why not hire the sickos who like these things to do the filtering?

nonbirithm 5y ago

> Facebook wouldn’t have these problems if they weren’t obsessed with acquiring as many users as possible. Is there any venture capitalist that would generally believe "there is value in not trying to connect everyone on Earth together?" It feels like when you market your hot startup idea, connecting the world or enacting change is merely a cool new thing that carries no long-term downside. Find a market, connect with a target audience, grow. Maybe that's how Facebook started (or not). I feel that, because of human nature, if it weren't Facebook then some other company would have inevitably taken its place. Being able to connect to someone faster and easier than ever sounds like a tangible gain for average people. The idea, with no strings attached, was too tempting to pass up for someone that truly wants to change the world and be a hero of some kind. It worked for a single university, after all. It's the sense of starting a technology company when the future is completely unknown and the potential for change is exciting. I'm not sure there was anyone that would have understood the consequences of taking such an idea to its logical conclusion, and now Facebook has become irreversibly baked into human culture. The more people that want to connect, the more moderation staff is needed to match. Unfortunately, the desire to connect with others is wired into pretty much everyone on Earth at the deepest level (with exceptions, of course), and so your potential market is... pretty much everyone on Earth.

nitwit005 5y ago

> Facebook wouldn’t have these problems if they weren’t obsessed with acquiring as many users as possible. Even small sites have people upload child porn and other stuff into them, and some IT guy has to go handle it. The only thing that can really reduce the scale of this stuff is for the people doing it to suffer a consequence like jail time. That's clearly not happening in most cases, despite Facebook and others sharing information with the authorities. The same is true for other issues like people being hounded by death/rape threats and the like online. The police mostly ignore it as an issue, so moderators have to read a bunch of it, and the people doing it just get a suspension or ban from the one platform.