Google respects robots.txt. Allowing Google scraping is optional. In fact many websites serve more content to google bot than to you before you log in, google b

Google respects robots.txt. Allowing Google scraping is optional. In fact many websites serve more content to google bot than to you before you log in, google bots clip through paywalls