5 ms·
A commenter raises an interesting question: "How did you detect "decay"? Just based on HTTP codes, or by actually looking at the linked content? On my own blog
by splitbrain 11y ago
A commenter raises an interesting question:
"How did you detect "decay"? Just based on HTTP codes, or by actually looking at the linked content? On my own blog, I found more often than I like that old links still "work" per HTTP, but now refer to something rather different from the content that I originally intended to refer to."
What would be some good way to detect such spam sites in an automated way? Looking for the link's title in the remote HTML? Check for common domain placeholder page contents and spam words? Maybe Google has some API one could use?
- socket0 11y agoI've run a similar test on my own blog (links dating back to ~2000), and I had the same problem. Some sites do a permanent redirect on broken links, others don't even redirect but show generic content on the original URL. I guess your success in automating this would depend on the nature of the links, but from a completely random collection the only success I had was with visual inspection. (Someone with far too much time on their hands could probably write a script to attempt to retrieve a copy of the page from the Wayback Machine from around the time the link was posted, then calculate the percentage change compared to the current version. Not really reliable, but worth a try.)