6 ms·

In the same time it’s so practical to ask a question and it opens 25 pages to search and summarize the answer. Before that’s more or less what I was trying to d

In the same time it’s so practical to ask a question and it opens 25 pages to search and summarize the answer. Before that’s more or less what I was trying to do by hand. Maybe not 25 websites because of crap SEO the top 10 contains BS content so I curated the list but the idea is the same no ?

pm215 1y ago
Sure, but if the fetcher is generating "39,000 requests per minute" then surely something has gone wrong somewhere ?
- miohtama 1y ago
  Even if it is generating 39k req/minute I would expect most of the pages already be locally cached by Meta, or served statically by their respective hosts. We have been working hard on catching websites and it has been a solved problem for the last decade or so.
  - mrweasel 1y ago
    Many AI crawlers seems to go to great length to avoid caches, not sure why.
  - ndriscoll 1y ago
    Could be serving no-cache headers? Seems like yet another problem stemming from every website being designed as if it were some dynamic application when nearly all of them are static documents. nginx doing 39k req/min to cacheable pages on an n100 is what you might call "98% idle", not "unsustainable load on web servers". The data transfer, on the other hand, could be substantial and costly. Is it known whether these crawlers do respect caching at all? Provide If-Modified-Since/If-None-Match or anything like that?
- andai 1y ago
  They're not very good at web queries, if you expand the thinking box to see what they're searching for, like half of it is nonsense. e.g. they'll take an entire sentence the user said and put it in quotes for no reason. Thankfully search engines started ignoring quotes years ago, so it balances out...
rco8786 1y ago
My personal experience is that OpenAI's crawler was hitting a very, very low traffic website I manage 10s of 1000s of times a minute non-stop. I had to block it from Cloudflare.
- danaris 1y ago
  Same here. I run a very small browser game (~120 weekly users currently), and until I put its Wiki (utterly uninteresting to anyone who doesn't already play the game) behind a login-wall, the bots were causing massive amounts of spurious traffic. Due to some of the Wiki's data coming live from the game through external data feeds, the deluge of bots actually managed to crash the game several times, necessitating a restart of the MariaDB process.
  - mrweasel 1y ago
    Wikis seems to attract AI bots like crazy, especially the bad kind that will attempt any type of cache invalidation available to them.
- Leynos 1y ago
  Where is caching breaking so badly that this is happening? Are OpenAI failing to use etags or honour cache validity?
  - Analemma_ 1y ago
    Their crawler is vibe-coded.
  - deleted 1y ago
    [deleted]

pm215 1y ago

Sure, but if the fetcher is generating "39,000 requests per minute" then surely something has gone wrong somewhere ?

miohtama 1y ago
Even if it is generating 39k req/minute I would expect most of the pages already be locally cached by Meta, or served statically by their respective hosts. We have been working hard on catching websites and it has been a solved problem for the last decade or so.
- mrweasel 1y ago
  Many AI crawlers seems to go to great length to avoid caches, not sure why.
- ndriscoll 1y ago
  Could be serving no-cache headers? Seems like yet another problem stemming from every website being designed as if it were some dynamic application when nearly all of them are static documents. nginx doing 39k req/min to cacheable pages on an n100 is what you might call "98% idle", not "unsustainable load on web servers". The data transfer, on the other hand, could be substantial and costly. Is it known whether these crawlers do respect caching at all? Provide If-Modified-Since/If-None-Match or anything like that?
andai 1y ago
They're not very good at web queries, if you expand the thinking box to see what they're searching for, like half of it is nonsense. e.g. they'll take an entire sentence the user said and put it in quotes for no reason. Thankfully search engines started ignoring quotes years ago, so it balances out...

rco8786 1y ago

My personal experience is that OpenAI's crawler was hitting a very, very low traffic website I manage 10s of 1000s of times a minute non-stop. I had to block it from Cloudflare.

danaris 1y ago
Same here. I run a very small browser game (~120 weekly users currently), and until I put its Wiki (utterly uninteresting to anyone who doesn't already play the game) behind a login-wall, the bots were causing massive amounts of spurious traffic. Due to some of the Wiki's data coming live from the game through external data feeds, the deluge of bots actually managed to crash the game several times, necessitating a restart of the MariaDB process.
- mrweasel 1y ago
  Wikis seems to attract AI bots like crazy, especially the bad kind that will attempt any type of cache invalidation available to them.
Leynos 1y ago
Where is caching breaking so badly that this is happening? Are OpenAI failing to use etags or honour cache validity?
- Analemma_ 1y ago
  Their crawler is vibe-coded.
- deleted 1y ago
  [deleted]