5 ms·

The theory I've heard is related to 'crawl budget'. Google is only going to devote a finite amount of time to indexing your site. If the number of articles on y

The theory I've heard is related to 'crawl budget'. Google is only going to devote a finite amount of time to indexing your site. If the number of articles on your site exceeds that time, some portion of your site won't be indexed. So by 'pruning' undesirable pages, you might boost attention on the articles you want indexed. No clue how this ends up working in practice. Google's suggestion isn't to delete pages, but maybe mark some pages with a no index header. https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget https://developers.google.com/search/docs/crawling-indexing/...

nevi-me 3y ago
It could be better to opt those articles out of the crawler. Unless that's more effort. If articles included the year and month in the URL prefix, I would disallow /201* instead.
0cf8612b2e1e 3y ago
Even if that rule were true, why wouldn’t everything in the say, top NNN internet sites get an exemption? It is the Internet’s most hit content, why would it not be exhaustively indexed? Alternatively, other than ads, what is changing on a CNN article from 10 years ago? Why would that still be getting daily scans?
- kenjackson 3y ago
  That's a good point about the static nature of some pages. Is there any way to tell a crawler to crawl this page, but after this date don't crawl again, but keep anything you previously crawled.
- bhandziuk 3y ago
  CNET* not CNN. But everything you say is still true.
- em-bee 3y ago
  the ads are different. i am tracking rss feeds of many sites, and on some i get notifications for old articles because something irrelevant in the page changed.
- progmetaldev 3y ago
  Probably bad technology detecting a change. Things like current news showing up beneath the article, which changes whenever a new article is added. I've seen this happen on quite a few large websites. It might be technologically easier to drop old articles than the amount of time to fix whatever they use to determine if a page has changed. You would think a site like CNET wouldn't have to deal with something like that, but sometimes these sites that have been around for a long time have some serious outdated tech.
tedunangst 3y ago
How does Wikipedia manage to remain indexed?
- skissane 3y ago
  Very likely Google special-cases Wikipedia
- lmm 3y ago
  Purely speculating, Wikipedia has a huge number of inbound links (likely many more than CNet or even than more popular sites) which crawler allocation might be proportionate to. Even if it only crawled pages that had a specific link from an external site, that would be enough for Google to get pretty good coverage of Wikipedia.
- lkbm 3y ago
  If I were establishing a "crawl budget", it would be adjusted by value. If you're consistently serving up hits as I crawl, I'll keep crawling. If it's a hundred pages that will basically never be a first page result, maybe not. Wikipedia had a long tail of low-value content, but even the low-value content tends to be among the highest value for its given focus. e.g., I don't know how many people search "Danish trade monopoly in Iceland", and the Wikipedia article on it isn't fantastic, but it's a pretty good start[0]. Good enough to serve up as the main snippet on Google. [0] https://en.wikipedia.org/wiki/Danish_trade_monopoly_in_Iceland https://en.wikipedia.org/wiki/Danish_trade_monopoly_in_Icela...
- deleted 3y ago
  [deleted]
- pessimizer 3y ago
  Google is paying Wikipedia through "Wikimedia Enterprise." If Wikipedia weren't able to sucker people into thinking that they're poverty-stricken, Google would probably prop it up like they do Firefox.
- ericd 3y ago
  Your site isn’t worthy of the same crawl budget as Wikipedia.
crazygringo 3y ago
But as that linked guide explains, that's only relevant for sites with e.g. over a million pages changing once a week. That's for stuff like large e-commerce sites with constantly changing product info. Google is clear that if your content doesn't change often (in the way that news articles don't), then crawl budget is irrelevant.
- codedokode 3y ago
  To determine whether content changes Google has to spend budget as well, hasn't it? So it has to fetch that 20-years old article.
  - throw0101a 3y ago
    > So it has to fetch that 20-years old article. It doesn't have to fetch every article (statical sampling can give confidence intervals), and it doesn't have to fetch the full article: doing a "HEAD /" instead of a "GET /" will save on bandwidth, and throwing in ETag / If-Modified-Since / whatever headers can get the status of an article (200 versus 304 response) without bother with the full fetch.
    - voramok 3y ago
      There’s an obvious way this can be exploited. Bait and switch.
  - strken 3y ago
    If the content is literally the same, the crawler should be able to use If-Modified-Since, right? It still has to make a HTTP request, but not parse or index anything.
    - codedokode 3y ago
      If the content is dynamic (e.g. a list of popular articles in a sidebar has changed), then the page will be considered "updated".
jesprenj 3y ago
They could specify in the sitemap how often do old articles change. Or set a indefinite caching header.
- codedokode 3y ago
  Google might not trust the sitemap because it sometimes is wrong.
throw0101a 3y ago
> The theory I've heard is related to 'crawl budget'. Google is only going to devote a finite amount of time to indexing your site. Once a site has been indexed once, should it really be crawled again? Perhaps Google should search for RSS/Atom feeds on sites and poll those regularly for updates: that way they don't waste time doing to a site scrape multiple times. Old(er) articles, once crawled, don't really have to be babysat. If Google wants to double-check that an already-crawled site hasn't changed too much, they can do a statistical sampling of random links on it using ETag / If-Modified-Since / whatever.
- jszymborski 3y ago
  I can imagine a malicious actor changing an SEO-friendly page to something spammy and not SEO-friendly. Since E-Tag and If-Modified-Since are returned by the server, they can be manipulated. Just a guess though.
- influx 3y ago
  This should be what sitemap.xml provides already.
- jrochkind1 3y ago
  The SiteMap, which was invented by Google and designed to give information to crawlers, already includes last-updated info. No need to invent a new system based on RSS/Atom, there is already an actually existing and in-use system based on SiteMap. So, what you suggest is already happening -- or at least, the system is already there for it to happen. It's possible Google does not trust the last modified info given by site owners enough, or for other reasons does not use your suggested approach, I can't say. https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap https://developers.google.com/search/docs/crawling-indexing/...

nevi-me 3y ago

It could be better to opt those articles out of the crawler. Unless that's more effort. If articles included the year and month in the URL prefix, I would disallow /201* instead.

0cf8612b2e1e 3y ago

Even if that rule were true, why wouldn’t everything in the say, top NNN internet sites get an exemption? It is the Internet’s most hit content, why would it not be exhaustively indexed? Alternatively, other than ads, what is changing on a CNN article from 10 years ago? Why would that still be getting daily scans?

kenjackson 3y ago
That's a good point about the static nature of some pages. Is there any way to tell a crawler to crawl this page, but after this date don't crawl again, but keep anything you previously crawled.
bhandziuk 3y ago
CNET* not CNN. But everything you say is still true.
em-bee 3y ago
the ads are different. i am tracking rss feeds of many sites, and on some i get notifications for old articles because something irrelevant in the page changed.
progmetaldev 3y ago
Probably bad technology detecting a change. Things like current news showing up beneath the article, which changes whenever a new article is added. I've seen this happen on quite a few large websites. It might be technologically easier to drop old articles than the amount of time to fix whatever they use to determine if a page has changed. You would think a site like CNET wouldn't have to deal with something like that, but sometimes these sites that have been around for a long time have some serious outdated tech.

tedunangst 3y ago

How does Wikipedia manage to remain indexed?

skissane 3y ago
Very likely Google special-cases Wikipedia
lmm 3y ago
Purely speculating, Wikipedia has a huge number of inbound links (likely many more than CNet or even than more popular sites) which crawler allocation might be proportionate to. Even if it only crawled pages that had a specific link from an external site, that would be enough for Google to get pretty good coverage of Wikipedia.
lkbm 3y ago
If I were establishing a "crawl budget", it would be adjusted by value. If you're consistently serving up hits as I crawl, I'll keep crawling. If it's a hundred pages that will basically never be a first page result, maybe not. Wikipedia had a long tail of low-value content, but even the low-value content tends to be among the highest value for its given focus. e.g., I don't know how many people search "Danish trade monopoly in Iceland", and the Wikipedia article on it isn't fantastic, but it's a pretty good start[0]. Good enough to serve up as the main snippet on Google. [0] https://en.wikipedia.org/wiki/Danish_trade_monopoly_in_Iceland https://en.wikipedia.org/wiki/Danish_trade_monopoly_in_Icela...
deleted 3y ago
[deleted]
pessimizer 3y ago
Google is paying Wikipedia through "Wikimedia Enterprise." If Wikipedia weren't able to sucker people into thinking that they're poverty-stricken, Google would probably prop it up like they do Firefox.
ericd 3y ago
Your site isn’t worthy of the same crawl budget as Wikipedia.

crazygringo 3y ago

But as that linked guide explains, that's only relevant for sites with e.g. over a million pages changing once a week. That's for stuff like large e-commerce sites with constantly changing product info. Google is clear that if your content doesn't change often (in the way that news articles don't), then crawl budget is irrelevant.

codedokode 3y ago
To determine whether content changes Google has to spend budget as well, hasn't it? So it has to fetch that 20-years old article.
- throw0101a 3y ago
  > So it has to fetch that 20-years old article. It doesn't have to fetch every article (statical sampling can give confidence intervals), and it doesn't have to fetch the full article: doing a "HEAD /" instead of a "GET /" will save on bandwidth, and throwing in ETag / If-Modified-Since / whatever headers can get the status of an article (200 versus 304 response) without bother with the full fetch.
  - voramok 3y ago
    There’s an obvious way this can be exploited. Bait and switch.
- strken 3y ago
  If the content is literally the same, the crawler should be able to use If-Modified-Since, right? It still has to make a HTTP request, but not parse or index anything.
  - codedokode 3y ago
    If the content is dynamic (e.g. a list of popular articles in a sidebar has changed), then the page will be considered "updated".

jesprenj 3y ago

They could specify in the sitemap how often do old articles change. Or set a indefinite caching header.

codedokode 3y ago
Google might not trust the sitemap because it sometimes is wrong.

throw0101a 3y ago

> The theory I've heard is related to 'crawl budget'. Google is only going to devote a finite amount of time to indexing your site. Once a site has been indexed once, should it really be crawled again? Perhaps Google should search for RSS/Atom feeds on sites and poll those regularly for updates: that way they don't waste time doing to a site scrape multiple times. Old(er) articles, once crawled, don't really have to be babysat. If Google wants to double-check that an already-crawled site hasn't changed too much, they can do a statistical sampling of random links on it using ETag / If-Modified-Since / whatever.

jszymborski 3y ago
I can imagine a malicious actor changing an SEO-friendly page to something spammy and not SEO-friendly. Since E-Tag and If-Modified-Since are returned by the server, they can be manipulated. Just a guess though.
influx 3y ago
This should be what sitemap.xml provides already.
jrochkind1 3y ago
The SiteMap, which was invented by Google and designed to give information to crawlers, already includes last-updated info. No need to invent a new system based on RSS/Atom, there is already an actually existing and in-use system based on SiteMap. So, what you suggest is already happening -- or at least, the system is already there for it to happen. It's possible Google does not trust the last modified info given by site owners enough, or for other reasons does not use your suggested approach, I can't say. https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap https://developers.google.com/search/docs/crawling-indexing/...