6 ms·

This might come off as a little rude, but it's sincere advice from someone who used to work in anti-spam (not at Facebook): I had a quick look through Dreamwid

by _ps6d 6y ago

This might come off as a little rude, but it's sincere advice from someone who used to work in anti-spam (not at Facebook): I had a quick look through Dreamwidth's "latest" page (https://www.dreamwidth.org/latest https://www.dreamwidth.org/latest) earlier today, and a major portion of the posts on there were blatant spam for things like credit card scams, "Work from home and make $1000/day!", and so on. You seem to be hosting a lot of spam, and those spam posts are also far more likely to be getting linked externally on sites like Facebook, since that's the reason they're being created. Because Dreamwidth is effectively free website hosting along with a free new subdomain for each account, blocking individual subdomains is futile, and it's difficult for external sites to distinguish between spam and legitimate blogs. I'm sure Facebook will unblock you fairly soon, but unless you get the spam on Dreamwidth under control, this will probably happen fairly often with different sites blocking it. It would be easy to end up with an impression of Dreamwidth being a spam-hosting site, and decide to block it (either manually or automatically). Blogspot has always been in a similar situation and would get blocked from a lot of sites due to the sheer amount of spam it hosts.

mindslight 6y ago
Controlling spam used to be about stopping unwanted messages sent to users. Now it has morphed into this idea that every site has the responsibility of content-policing their own users, lest what they publish be used to facilitate spam. Your advice may be pragmatic, but it shows how far we've slid down the slippery slope.
- jsnell 6y ago
  > Now it has morphed into this idea that every site has the responsibility of content-policing their own users, lest what they publish be linked from spam. Not sure what you mean here. The problem Deimorz was bringing up wasn't just about users writing something, and spammers linking to it. It was that this site was being used to host the spam payloads. By spammers, not by actual users. And this is how a lot of the early spam fighting worked: by finding hosts that allowed sending spam and publishing their IPs on blocklists. All mail traffic from those IPs, even if legit, would then be rejected by a large proportion of mail servers that subscribed to these blocklists.
  - mindslight 6y ago
    > this site was being used to host the spam payloads Calling these "spam payloads" is incorrect. The spam payloads are on Faceboot's servers. These are sites that are linked to by the spam, ostensibly for the purpose of funneling to whatever the spam is trying to market. Trying to police generic web pages, rather than the spam itself, seems like an exercise in futility given the basic philosophy of the Internet. > And this is how a lot of the early spam fighting worked: by finding hosts that allowed sending spam and publishing their IPs on blocklists The situation has a similar shape, but there is a distinction as Dreamwidth is not actively sending spam but rather responding to requests from viewers. Still, we can look at the outcome of what happened to the email ecosystem - increased centralization of providers - for a warning of what's to come.
    - thoraway1010 6y ago
      Malware and childporn reduction efforts also often go after the hosts of that content. I'm not sure why calling the folks hosting this stuff what it is incorrect. Sure, childporn folks don't actually necessarily "send" child porn, they just respond to requests from viewers. But they host it. These scam sites are like that - do you really think you can make $30,000 a week working 30 minutes a day from your home computer if you just send these idiot $25?
      - mindslight 6y ago
        You're just listing the earlier stops on the slippery slope. Make hosts responsible for policing information when it's viscerally-revolting child porn. Then make hosts police content when it's directly harmful to people's computers. Then make hosts police content when it's an attempt to scam. There's already a call to control political information when it has harmful effects on society. Next up is "your website was blacklisted because you allowed a user to link to Plandemic". I agree Plandemic has no redeeming purpose, but censorship is not the answer.
        thoraway1010 6y ago
- paulie_a 6y ago
  Sites have zero responsibility to monitor for spam. Other sites have no obligation to link to them. And that is what we are talking about. An obligation to link others
duskwuff 6y ago
Hmm, maybe it comes in waves? I just read over the last hour or so of posts on /latest/ and most of the posts there looked legitimate.
- AndrewDucker 6y ago
  Yeah, same. I can't tell with the Russian posts, but the ones in English were a mixture of journal updates and fanfic.
  - owl57 6y ago
    There's about a dozen Russian posts now, only one of which looks like spam.
zorkian 6y ago
You're definitely right -- this is an issue. I could very well believe that we tripped some FB spam measures. We have a very manual anti-spam process right now that relies on humans to detect it and action it. We have a couple of very dedicated folks who end up looking every few hours, but it's not automated, and we don't have full timezone coverage. It's definitely something I'd like to see us improve, but we've been focused on other projects (like switching from mid-90s HTML to a responsive design, which is a slow rewrite of the entire site). That said, if you have any advice on reasonably scalable ways of doing this in-house that don't involve sending our user content to a third party, I'd love to take any recommendations! Feel free to email me, mark@dreamwidth.org, if you would rather do that. And if not, don't worry about it, I appreciate the comment anyway :)
- Mehdi2277 6y ago
  The simplest spam filtering algorithm would be a naive bayes filter. It's essentially keep a count of words that appear in all posts, words that appear in spam posts, and words in non spam posts. Those counts + bayes rule will let you figure out the probability of spam given a word. It's called naive bayes because you assume each word in your post is independent of the others so probability the whole post is spam is just product of the probabilities. The nice thing about this is it's pretty computationally light and straightforward to implement for any language. I have no clue as to your stack, but if you have python for your backend then sklearn is a good library that has a naive bayes classifier (plus a lot of other better options). Any post with a high probability of being spam, I'd automatically flag and by default just remove with the option for a user to ask for manual review. Main thing you'd need for this or any fancier approach is some dataset of spam/non spam posts. If you have an easy way of retrieving past posts that were labelled spam that should allow you to make a fine dataset. If you don't want to train on your own user posts (although only information kept is word counts here), you can look online for spam datasets and use one of those to train your classifier.
  - gus_massa 6y ago
    I used SpamBayes a few years ago http://www.spambayes.org/ http://www.spambayes.org/ (Is the project dead now?) (It has a PSF licence https://en.wikipedia.org/wiki/Python_Software_Foundation_License https://en.wikipedia.org/wiki/Python_Software_Foundation_Lic... https://en.wikipedia.org/wiki/Comparison_of_free_and_open-source_software_licences https://en.wikipedia.org/wiki/Comparison_of_free_and_open-so...) The nice part is that SpamBayes gives you two numbers, the spam "probability" and the ham "probability". When one of them is very close to 1 (like > .99) and the other is very close to 0 (like <.01), there is a good chance that the message is really spam or ham. And this classify almost all the messages. But from time to time you get a message where the numbers are not so clear, or both are big or both are small, and this means the classifier is confused and you really must take a look at the message.

by _ps6d 6y ago

mindslight 6y ago

Controlling spam used to be about stopping unwanted messages sent to users. Now it has morphed into this idea that every site has the responsibility of content-policing their own users, lest what they publish be used to facilitate spam. Your advice may be pragmatic, but it shows how far we've slid down the slippery slope.

jsnell 6y ago
> Now it has morphed into this idea that every site has the responsibility of content-policing their own users, lest what they publish be linked from spam. Not sure what you mean here. The problem Deimorz was bringing up wasn't just about users writing something, and spammers linking to it. It was that this site was being used to host the spam payloads. By spammers, not by actual users. And this is how a lot of the early spam fighting worked: by finding hosts that allowed sending spam and publishing their IPs on blocklists. All mail traffic from those IPs, even if legit, would then be rejected by a large proportion of mail servers that subscribed to these blocklists.
- mindslight 6y ago
  > this site was being used to host the spam payloads Calling these "spam payloads" is incorrect. The spam payloads are on Faceboot's servers. These are sites that are linked to by the spam, ostensibly for the purpose of funneling to whatever the spam is trying to market. Trying to police generic web pages, rather than the spam itself, seems like an exercise in futility given the basic philosophy of the Internet. > And this is how a lot of the early spam fighting worked: by finding hosts that allowed sending spam and publishing their IPs on blocklists The situation has a similar shape, but there is a distinction as Dreamwidth is not actively sending spam but rather responding to requests from viewers. Still, we can look at the outcome of what happened to the email ecosystem - increased centralization of providers - for a warning of what's to come.
  - thoraway1010 6y ago
    Malware and childporn reduction efforts also often go after the hosts of that content. I'm not sure why calling the folks hosting this stuff what it is incorrect. Sure, childporn folks don't actually necessarily "send" child porn, they just respond to requests from viewers. But they host it. These scam sites are like that - do you really think you can make $30,000 a week working 30 minutes a day from your home computer if you just send these idiot $25?
    - mindslight 6y ago
      You're just listing the earlier stops on the slippery slope. Make hosts responsible for policing information when it's viscerally-revolting child porn. Then make hosts police content when it's directly harmful to people's computers. Then make hosts police content when it's an attempt to scam. There's already a call to control political information when it has harmful effects on society. Next up is "your website was blacklisted because you allowed a user to link to Plandemic". I agree Plandemic has no redeeming purpose, but censorship is not the answer.
      - thoraway1010 6y ago
paulie_a 6y ago
Sites have zero responsibility to monitor for spam. Other sites have no obligation to link to them. And that is what we are talking about. An obligation to link others

duskwuff 6y ago

Hmm, maybe it comes in waves? I just read over the last hour or so of posts on /latest/ and most of the posts there looked legitimate.

AndrewDucker 6y ago
Yeah, same. I can't tell with the Russian posts, but the ones in English were a mixture of journal updates and fanfic.
- owl57 6y ago
  There's about a dozen Russian posts now, only one of which looks like spam.

zorkian 6y ago

You're definitely right -- this is an issue. I could very well believe that we tripped some FB spam measures. We have a very manual anti-spam process right now that relies on humans to detect it and action it. We have a couple of very dedicated folks who end up looking every few hours, but it's not automated, and we don't have full timezone coverage. It's definitely something I'd like to see us improve, but we've been focused on other projects (like switching from mid-90s HTML to a responsive design, which is a slow rewrite of the entire site). That said, if you have any advice on reasonably scalable ways of doing this in-house that don't involve sending our user content to a third party, I'd love to take any recommendations! Feel free to email me, mark@dreamwidth.org, if you would rather do that. And if not, don't worry about it, I appreciate the comment anyway :)

Mehdi2277 6y ago
The simplest spam filtering algorithm would be a naive bayes filter. It's essentially keep a count of words that appear in all posts, words that appear in spam posts, and words in non spam posts. Those counts + bayes rule will let you figure out the probability of spam given a word. It's called naive bayes because you assume each word in your post is independent of the others so probability the whole post is spam is just product of the probabilities. The nice thing about this is it's pretty computationally light and straightforward to implement for any language. I have no clue as to your stack, but if you have python for your backend then sklearn is a good library that has a naive bayes classifier (plus a lot of other better options). Any post with a high probability of being spam, I'd automatically flag and by default just remove with the option for a user to ask for manual review. Main thing you'd need for this or any fancier approach is some dataset of spam/non spam posts. If you have an easy way of retrieving past posts that were labelled spam that should allow you to make a fine dataset. If you don't want to train on your own user posts (although only information kept is word counts here), you can look online for spam datasets and use one of those to train your classifier.
- gus_massa 6y ago
  I used SpamBayes a few years ago http://www.spambayes.org/ http://www.spambayes.org/ (Is the project dead now?) (It has a PSF licence https://en.wikipedia.org/wiki/Python_Software_Foundation_License https://en.wikipedia.org/wiki/Python_Software_Foundation_Lic... https://en.wikipedia.org/wiki/Comparison_of_free_and_open-source_software_licences https://en.wikipedia.org/wiki/Comparison_of_free_and_open-so...) The nice part is that SpamBayes gives you two numbers, the spam "probability" and the ham "probability". When one of them is very close to 1 (like > .99) and the other is very close to 0 (like <.01), there is a good chance that the message is really spam or ham. And this classify almost all the messages. But from time to time you get a message where the numbers are not so clear, or both are big or both are small, and this means the classifier is confused and you really must take a look at the message.