Instead of scraping HN directly, have you looked at the HN Search API, from the old HN search engine ? ( https://www.hnsearch.com/api https://www.hnsearch.com/a

Instead of scraping HN directly, have you looked at the HN Search API, from the old HN search engine ? ( https://www.hnsearch.com/api https://www.hnsearch.com/api ) The current search engine looks like it also has an API ( https://hn.algolia.com/api https://hn.algolia.com/api ). Presumably they have a full dataset on hand, which you may be able to get if you contact them.