5 ms·
One possible partial solution would be for HN to check submitted URLs for "majority matches" (say, 90% similar) and a check against submission titles over the l
by Rust 16y ago
One possible partial solution would be for HN to check submitted URLs for "majority matches" (say, 90% similar) and a check against submission titles over the last 48 hours, then offer the submitter an option to simply up-vote an existing story instead of submitting a brand new duplicate. If submitted pages were also scraped for headings and keywords, that could be used to provide similar matches as well.
- whimsy 16y agoThe problem proposed is not duplicates (i.e. 3 of the same story from Daring Fireball) but multiple writers on the same topic. URL matching would fail to identify these.
- Rust 16y agoTrue enough, but that's what title matching and content scraping are for. There would still be a percentage of misses (and false positives), but with the end decision to up-vote an existing, similar article or submit a new one left in the hands of the submitter, I think it's a reasonable feature.
- whimsy 16y agoContent scraping introduces a much higher load than mere URL regex magic...
- Rust 16y agoNo question about that. It's one of those "do it if we can" features. One could have a "light" version that just scraped the target page for header tags alone though. That might be a good compromise.