6 ms·

Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries in

Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries instead of going directly. Is it really a bot if it's in response to a human asking for some aggregate information about charities, triggering a web search and then following the result links to get details for the human? Well, clearly yes it is, but it's a very different proposition from this article's implication that "they have no throttling on their scrapers"[1]. > Challenge 46 datacenter ASNs. Humans don't browse from AWS. People who have workstations in the cloud do. > The bots use 99% of the bill and I pay 100% of it. Running a site this way is always a wallet-DDoS risk. [1] though yes, by far most will be pure automation with no human in the loop. It's an assumption on my part, but feels like a safe one.

bakugo 1mo ago
> Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries instead of going directly. That's not absent from the article, it's right there in the section titled "The Claude ratio". ChatGPT, Claude, etc. use different user-agents for scraping vs user-initiated requests, and the author notes that user-initiated requests were an absolutely miniscule fraction of the total traffic.
- thomashabets2 1mo ago
  Oh, that's what "Anthropic's search crawler, had requested 420,680 pages in one week. That same week, Claude sent me 12 human visitors" meant? Yeah, googling it does seem like "Claude-User" is for user-initiated requests. By "Claude sent me" I thought the author meant referer header in real browser requests showed that they came from. I mean, that's what the section "Pages crawled per visitor referred" refers to, right?
sethops1 1mo ago
> Running a site this way What do you mean "this way". What other way is there to run the site?
- thomashabets2 1mo ago
  For the age of the Internet, running a website metered is a pretty new phenomenon. Or at least this directly metered. 20 years ago maybe you would pay for your monthly 95th percentile bandwidth use. Not saying that was great either, but it was certainly more managable than what's common now.
fooey 1mo ago
cloudflares lack of spending caps and and usage alerts is a seriously unnerving approach

bakugo 1mo ago

> Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries instead of going directly. That's not absent from the article, it's right there in the section titled "The Claude ratio". ChatGPT, Claude, etc. use different user-agents for scraping vs user-initiated requests, and the author notes that user-initiated requests were an absolutely miniscule fraction of the total traffic.

thomashabets2 1mo ago
Oh, that's what "Anthropic's search crawler, had requested 420,680 pages in one week. That same week, Claude sent me 12 human visitors" meant? Yeah, googling it does seem like "Claude-User" is for user-initiated requests. By "Claude sent me" I thought the author meant referer header in real browser requests showed that they came from. I mean, that's what the section "Pages crawled per visitor referred" refers to, right?

sethops1 1mo ago

> Running a site this way What do you mean "this way". What other way is there to run the site?

thomashabets2 1mo ago
For the age of the Internet, running a website metered is a pretty new phenomenon. Or at least this directly metered. 20 years ago maybe you would pay for your monthly 95th percentile bandwidth use. Not saying that was great either, but it was certainly more managable than what's common now.

fooey 1mo ago

cloudflares lack of spending caps and and usage alerts is a seriously unnerving approach