Just FYI google and bing publish their user agent strings[1][2] for the crawlers. At least in my experience most of the typical ad-infested and paywalled news s

Just FYI google and bing publish their user agent strings[1][2] for the crawlers. At least in my experience most of the typical ad-infested and paywalled news sites wont display the paywall if you change the user agent to a crawler they prefer. [1] https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers https://developers.google.com/search/docs/crawling-indexing/... [2] https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0 https://www.bing.com/webmasters/help/which-crawlers-does-bin...