13 ms·

It actually is. I run a small video game forum with posts going back to 2008. We got absolutely smashed by bots scraping for training data for LLMs. So I put

by frameset 10mo ago

It actually is. I run a small video game forum with posts going back to 2008. We got absolutely smashed by bots scraping for training data for LLMs. So I put it behind Cloudflare and now it's down. Ho hum.

shaky-carrousel 10mo ago
It'd funny if these bots were run by Cloudflare.
- frameset 10mo ago
  Ha, yeah. They seemed to mostly be in SE Asia.
stevepotter 10mo ago
Can you please elaborate on “smashed”? I’m very interested
- frameset 10mo ago
  I took a screenshot of the graph in cloudflare when I switched on the bot challenges. https://i.ibb.co/qHCJyY7/image.png https://i.ibb.co/qHCJyY7/image.png I wrote the below to explain to our users what was happening, so apologies if the language is too simple for a HN reader. - 0630, we switched our DNS to proxy through CF, starting the collection of data, and implemented basic bot protections - Unfortunately whatever anti-bot magic they have isn't quite having the effect, even after two hours. - 0830, I sign in and take a look at the analytics. It seems like <SITE NAME> is very popular in Vietnam, Brazil, and Indonesia. - 0845, I make it so users from those countries have to pass a CF "challenge". This is similar to a CAPTCHA, but CF try to make it so there's no "choosing all the cars in an image" if they can help it. - So far 0% of our Asian audience have passed a challenge.
watermelon0 10mo ago
Have you tried Anubis or similar tools? I've had similar issues with bot scraping of a forum taking all server resources, and using PoW challenge solved the problem. https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis
- frameset 10mo ago
  I did! It's very cool tech. However for our config it was easier to slap CF in front of it. I will say one very appealing use of Anubis I'd love to try is using it as a Traefik middleware to protect services running in docker containers.
- gspr 10mo ago
  I've always wondered: has there been any effort to implement a PoW challenge like that at a lower level? I.e., TCP but the handshake requires solving a challenge, otherwise the connection is just closed? It seems like something that could benefit from being invisible on the application layer. Edit: To answer my own question, yes: http://www.arijuels.com/wp-content/uploads/2013/09/JB99.pdf http://www.arijuels.com/wp-content/uploads/2013/09/JB99.pdf Edit 2: Maybe TLS would be another reasonable place for it?
trollbridge 10mo ago
Same problem here. If I didn't use Cloudflare, nearly all of my traffic would be (apparently misconfigured) scraper bots.

shaky-carrousel 10mo ago

It'd funny if these bots were run by Cloudflare.

frameset 10mo ago
Ha, yeah. They seemed to mostly be in SE Asia.

stevepotter 10mo ago

Can you please elaborate on “smashed”? I’m very interested

frameset 10mo ago
I took a screenshot of the graph in cloudflare when I switched on the bot challenges. https://i.ibb.co/qHCJyY7/image.png https://i.ibb.co/qHCJyY7/image.png I wrote the below to explain to our users what was happening, so apologies if the language is too simple for a HN reader. - 0630, we switched our DNS to proxy through CF, starting the collection of data, and implemented basic bot protections - Unfortunately whatever anti-bot magic they have isn't quite having the effect, even after two hours. - 0830, I sign in and take a look at the analytics. It seems like <SITE NAME> is very popular in Vietnam, Brazil, and Indonesia. - 0845, I make it so users from those countries have to pass a CF "challenge". This is similar to a CAPTCHA, but CF try to make it so there's no "choosing all the cars in an image" if they can help it. - So far 0% of our Asian audience have passed a challenge.

watermelon0 10mo ago

Have you tried Anubis or similar tools? I've had similar issues with bot scraping of a forum taking all server resources, and using PoW challenge solved the problem. https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis

frameset 10mo ago
I did! It's very cool tech. However for our config it was easier to slap CF in front of it. I will say one very appealing use of Anubis I'd love to try is using it as a Traefik middleware to protect services running in docker containers.
gspr 10mo ago
I've always wondered: has there been any effort to implement a PoW challenge like that at a lower level? I.e., TCP but the handshake requires solving a challenge, otherwise the connection is just closed? It seems like something that could benefit from being invisible on the application layer. Edit: To answer my own question, yes: http://www.arijuels.com/wp-content/uploads/2013/09/JB99.pdf http://www.arijuels.com/wp-content/uploads/2013/09/JB99.pdf Edit 2: Maybe TLS would be another reasonable place for it?

trollbridge 10mo ago

Same problem here. If I didn't use Cloudflare, nearly all of my traffic would be (apparently misconfigured) scraper bots.