7 ms·

This sounds too good to be true in practice. If I deploy it on a server and continuously query it from all of my devices, will Google ban that serve rIP?

supah 5y ago
scraping yes. normal usage no.
- tyingq 5y ago
  Isn't normal usage scraping in this case? Looking at whoogle-search/request.py it's "scraping" google urls via the python requests module. I'm reasonably sure google fingerprints requests and assigns different weights for "probably scraping". I wouldn't be surprised if this has a lower threshold for triggering their captchas and/or blocking.
  - andai 5y ago
    I often trigger Google's captchas during normal usage. It seems to suspect the more "advanced" features like "intitle:" or "inurl:", or if I search too rapidly. I take being mistaken for a machine as a compliment!
    - ricardo81 5y ago
      That's understandable, a lot of exploit seekers use those features to find exploits e.g. "powered by [cms with known exploit]", Google (and Bing) are definitely are more prone to showing you a captcha for those searches, especially if you're looking beyond the first page.
spinax 5y ago
I'm on a shared IP with millions of people using the same public IP (T-Mobile CGNAT), there is one IP (many of them, actually) doing that right now from every T-Mobile customer. Your one server will be a blip on their radar if it even registers.
- izzytcp 5y ago
  Dude, thanks for letting me know that everyone uses Google. I'm talking about static IPs like servers in the cloud, not your home. That stuff is automated and I am sure static IPs get banned but here must be a quota of something.
  - spinax 5y ago
    Dude, CGNAT is handled at the ISP layer, I do not have an IPv4 address at all locally, it's a 464XLAT done on T-Mobile's side. All users come from a shared IPv4 on their network, not mine. Dude.
    - treesknees 5y ago
      Dude - Each of those users behind the NAT will have a different set of cookies, user agents, screen sizes, among other fingerprints that qualify them as unique. ISPs also routinely place their CGNAT addresses on specific whitelists so that services don't block them for abuse (you can look through the NANOG email list to find examples of this.) IP addresses are also classified as residential, cloud/server, etc. If Google sees rapid requests from the same IP classified as a server that's sending a Python Requests user-agent, they can absolutely block it.
      - 123pie123 5y ago
        how long should google block the IP? when the ISP reassigns it to a new home they would be blocked also blocked from google
        treesknees 5y ago
        Obviously I can't answer how long Google should block an abusive IP address since I'm not Google. A CGNAT IP address is not reassigned to a home, it's shared among many homes. If you meant from my example the cloud server IP, that is one issue that comes up pretty often on cloud services and there's not a clean way around it. For example I use Linode as my VPN server, I used to have all sorts of trouble with Google making me enter captchas or blocking my search just because of the abuse coming from the same IP range. I actually can't even login to some apps while on my VPN, and I've had this same IP address on my Linode for close to 10 years, so it's not an issue with my /32 specifically. You'll see the same thing on AWS, many of those ec2 instances can't be used for sending email or for VPN services because previous users of the IP space abused their way onto block lists.
packet_nerd 5y ago
I set this up for my family and I a few months ago and we set all our browsers and devices to use it as the default search engine. We've been really happy with it. I also set up a small script on a cron job that queries random search strings every few minutes and opens the first few hits in selenium. My theory is that if I can't completely stop them from tracking us, I can at least dilute their data with bogus searches. We haven't had any issues from Google.
DavideNL 5y ago
I guess it will be just like Searx: https://searx.me/ https://searx.me/ If an IP generates too much search queries, Google will block / throttle it…
johnchristopher 5y ago
So, I set it up this week-end on a vps and subdomain and today it's blocked :(. About this page Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot. Why did this happen? IP address: 137.x.x.x Time: 2021-08-30T16:56:45Z URL: https://www.google.com/search?gbv=1&q=broken&lr=&hl=en&safe=off

supah 5y ago

scraping yes. normal usage no.

tyingq 5y ago
Isn't normal usage scraping in this case? Looking at whoogle-search/request.py it's "scraping" google urls via the python requests module. I'm reasonably sure google fingerprints requests and assigns different weights for "probably scraping". I wouldn't be surprised if this has a lower threshold for triggering their captchas and/or blocking.
- andai 5y ago
  I often trigger Google's captchas during normal usage. It seems to suspect the more "advanced" features like "intitle:" or "inurl:", or if I search too rapidly. I take being mistaken for a machine as a compliment!
  - ricardo81 5y ago
    That's understandable, a lot of exploit seekers use those features to find exploits e.g. "powered by [cms with known exploit]", Google (and Bing) are definitely are more prone to showing you a captcha for those searches, especially if you're looking beyond the first page.

spinax 5y ago

I'm on a shared IP with millions of people using the same public IP (T-Mobile CGNAT), there is one IP (many of them, actually) doing that right now from every T-Mobile customer. Your one server will be a blip on their radar if it even registers.

izzytcp 5y ago
Dude, thanks for letting me know that everyone uses Google. I'm talking about static IPs like servers in the cloud, not your home. That stuff is automated and I am sure static IPs get banned but here must be a quota of something.
- spinax 5y ago
  Dude, CGNAT is handled at the ISP layer, I do not have an IPv4 address at all locally, it's a 464XLAT done on T-Mobile's side. All users come from a shared IPv4 on their network, not mine. Dude.
  - treesknees 5y ago
    Dude - Each of those users behind the NAT will have a different set of cookies, user agents, screen sizes, among other fingerprints that qualify them as unique. ISPs also routinely place their CGNAT addresses on specific whitelists so that services don't block them for abuse (you can look through the NANOG email list to find examples of this.) IP addresses are also classified as residential, cloud/server, etc. If Google sees rapid requests from the same IP classified as a server that's sending a Python Requests user-agent, they can absolutely block it.
    - 123pie123 5y ago
      how long should google block the IP? when the ISP reassigns it to a new home they would be blocked also blocked from google
      - treesknees 5y ago
        Obviously I can't answer how long Google should block an abusive IP address since I'm not Google. A CGNAT IP address is not reassigned to a home, it's shared among many homes. If you meant from my example the cloud server IP, that is one issue that comes up pretty often on cloud services and there's not a clean way around it. For example I use Linode as my VPN server, I used to have all sorts of trouble with Google making me enter captchas or blocking my search just because of the abuse coming from the same IP range. I actually can't even login to some apps while on my VPN, and I've had this same IP address on my Linode for close to 10 years, so it's not an issue with my /32 specifically. You'll see the same thing on AWS, many of those ec2 instances can't be used for sending email or for VPN services because previous users of the IP space abused their way onto block lists.

packet_nerd 5y ago

I set this up for my family and I a few months ago and we set all our browsers and devices to use it as the default search engine. We've been really happy with it. I also set up a small script on a cron job that queries random search strings every few minutes and opens the first few hits in selenium. My theory is that if I can't completely stop them from tracking us, I can at least dilute their data with bogus searches. We haven't had any issues from Google.

DavideNL 5y ago

I guess it will be just like Searx: https://searx.me/ https://searx.me/ If an IP generates too much search queries, Google will block / throttle it…

johnchristopher 5y ago

So, I set it up this week-end on a vps and subdomain and today it's blocked :(. About this page Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot. Why did this happen? IP address: 137.x.x.x Time: 2021-08-30T16:56:45Z URL: https://www.google.com/search?gbv=1&q=broken&lr=&hl=en&safe=off