6 ms·

Show HN: Tiny Web Crawler

• photonios 8y ago
  It's not entirely correct to remove query parameters from the URL. Query parameters are part of the URL and can make a website show entirely different content. The Googlebot for example considers these two URL's different pages: /mypage?name=name1 /mypage?name=name2