5 ms·
I don't mind Facebook crawling pages as long as it respects robots.txt, but for the last few weeks we've been hammered by requests from Facebook-owned IP addres
by Quppa 9y ago
I don't mind Facebook crawling pages as long as it respects robots.txt, but for the last few weeks we've been hammered by requests from Facebook-owned IP addresses (millions of hits daily, 50+ for the same URL at times). They don't even set the User-Agent header.
There's a bug report regarding the missing header here: https://developers.facebook.com/bugs/1654459311255613/ https://developers.facebook.com/bugs/1654459311255613/
Unfortunately it seems impossible to get in touch with Facebook devs directly.
- ikeboy 9y agoSend a cease and desist to the CTO. Wait 30 days, then sue under the CFAA. LinkedIn did it
- icebraining 9y agoNo, LinkedIn was sued by the recipient of the letter.
- AznHisoka 9y agowhat site do you own (if you can tell)?
- Quppa 9y agoNot sure I can say. On a positive note, it's given us an opportunity to focus on performance improvements :)