6 ms·

In a few years the VC money will dry up and this gross overspend on slurping data will end.

igor47 17d ago
Visions of vast data centers surrounded by fields of browning grass, in which aging, rusting, formerly extremely expensive hardware is spending billions of compute cycles looking at anime catgirls
- rpcope1 17d ago
  Sounds like an even shittier version of the Lorax. :(
pixl97 17d ago
Unfortunately we're apt to run into some kind of Jeavons Paradox where the hardware gets so much faster in those few years will be able to slurp massive amounts of data cheaply so the problem never really ends.
marginalia_nu 17d ago
It's very likely the last few years of bot behavior is the consequence of the residential proxy business booming. This is indirectly due to AI company crawling, but the fact that they are as cheap and available as they are changes the incentives for anyone using them toward reckless and unsustainable request behavior, as there is no risk of burning your IPs, and very small chances of seeing any consequences of essentially DDoS:ing a website.
- inigyou 17d ago
  And the residential proxy business was created by Cloudflare, who was created by us using Cloudflare. I've been on all three sides (user of RPs, getting paid to run an RP, and trying to block RPs from my site). Residential proxy service is nice. You can scrape anything, even with the dumbest curl command, and only get a Cloudflare block maybe 15% of the time, in which case you just try again. That's less often than I get a cloudflare block from using a privacy browser from a non-proxy address. Cloudflare does not stop bots, it stops humans.
  - roughly 17d ago
    Wait, so you’ve been the person who wanted to keep people from scraping your site, the person who’s trying to scrape your site, and the person getting paid to help someone scrape your site? Brother, what are you doing with your life?
    - inigyou 17d ago
      Welcome to capitalism. Welcome to game theory. Welcome to competition. Welcome to the real world. Welcome to being a grown adult.
      - roughly 16d ago
        Contingency exists, I get that, and if you're truly in that state, sure, I don't judge necessity. A lot of our cohort seems to confuse a studied disinterest in looking beyond the end of their own nose for the sage wisdom of adulthood, though.
        inigyou 16d ago
        How will you know anything about a system you stubbornly only look at one side of? Like saying planes are terrible because they're loud. They are, but have you never traveled to a different city?
        roughly 16d ago
        I’ve never mugged someone either, I suppose I should give it a shot before I go passing judgement.

Visions of vast data centers surrounded by fields of browning grass, in which aging, rusting, formerly extremely expensive hardware is spending billions of compute cycles looking at anime catgirls

rpcope1 17d ago
Sounds like an even shittier version of the Lorax. :(

pixl97 17d ago

Unfortunately we're apt to run into some kind of Jeavons Paradox where the hardware gets so much faster in those few years will be able to slurp massive amounts of data cheaply so the problem never really ends.

marginalia_nu 17d ago

It's very likely the last few years of bot behavior is the consequence of the residential proxy business booming. This is indirectly due to AI company crawling, but the fact that they are as cheap and available as they are changes the incentives for anyone using them toward reckless and unsustainable request behavior, as there is no risk of burning your IPs, and very small chances of seeing any consequences of essentially DDoS:ing a website.

inigyou 17d ago
And the residential proxy business was created by Cloudflare, who was created by us using Cloudflare. I've been on all three sides (user of RPs, getting paid to run an RP, and trying to block RPs from my site). Residential proxy service is nice. You can scrape anything, even with the dumbest curl command, and only get a Cloudflare block maybe 15% of the time, in which case you just try again. That's less often than I get a cloudflare block from using a privacy browser from a non-proxy address. Cloudflare does not stop bots, it stops humans.
- roughly 17d ago
  Wait, so you’ve been the person who wanted to keep people from scraping your site, the person who’s trying to scrape your site, and the person getting paid to help someone scrape your site? Brother, what are you doing with your life?
  - inigyou 17d ago
    Welcome to capitalism. Welcome to game theory. Welcome to competition. Welcome to the real world. Welcome to being a grown adult.
    - roughly 16d ago
      Contingency exists, I get that, and if you're truly in that state, sure, I don't judge necessity. A lot of our cohort seems to confuse a studied disinterest in looking beyond the end of their own nose for the sage wisdom of adulthood, though.
      - inigyou 16d ago
        How will you know anything about a system you stubbornly only look at one side of? Like saying planes are terrible because they're loud. They are, but have you never traveled to a different city?
        roughly 16d ago
        I’ve never mugged someone either, I suppose I should give it a shot before I go passing judgement.