Sites will not let you crawl unless you are Google or maybe Microsoft. They might have, if it looked like you were likely going to become much better than Goog

Sites will not let you crawl unless you are Google or maybe Microsoft. They might have, if it looked like you were likely going to become much better than Google and would be competitive. But you will never look like that because they will not let you crawl. Is this true? I don't know, but it is one story. And gets to why there's a more fundamental economic mechanism to Google's monopoly, over Amazon's or Microsoft's.