5 ms·

Good news for curl users: https://github.com/mandatoryprogrammer/thermoptic https://github.com/mandatoryprogrammer/thermoptic

joshmn 1y ago
Work like this is incredible. I did not know this existed. Thank you.
- mandatory 1y ago
  Thanks :) if you have any issues with it let me know.
benatkin 1y ago
> NOTE: Due to many WAFs employing JavaScript-level fingerprinting of web browsers, thermoptic also exposes hooks to utilize the browser for key steps of the scraping process. See this section for more information on this. This reminds me of how Stripe does user tracking for fraude detection https://mtlynch.io/stripe-update/ https://mtlynch.io/stripe-update/ I wonder if thermoptic could handle that.
mips_avatar 1y ago
Cool project!
- mandatory 1y ago
  Thanks!
Symbiote 1y ago
Oh great /s In a month or two, I can be annoyed when I see some vibe-coded AI startup's script making five million requests a day to work's website with this. They'll have been ignoring the error responses: {"All data is public and available for free download": "https://example.edu/very-large-001.zip"} — a message we also write in the first line of every HTML page source. Then I will spend more time fighting this shit, and less time improving the public data system.
- mandatory 1y ago
  Feel free to read the README, this was already an ability that startups could pay for using private premium proxy services before thermoptic. Having an open source version allows regular people to do scraping and not just those rich in capital. Much of the best data services on the internet all start with scraping, the README lists many of them.
snowe2010 1y ago
People like you are why independent sites can’t afford to run on the internet anymore.
- mandatory 1y ago
  They can't? I've run many free independent sites for years, that's news to me.
- timbowhite 1y ago
  I run independent websites and I'm not broke yet.
- 1gn15 1y ago
  I block all humans (only robots are allowed) and I'm still able to run independent websites.

joshmn 1y ago

Work like this is incredible. I did not know this existed. Thank you.

mandatory 1y ago
Thanks :) if you have any issues with it let me know.

benatkin 1y ago

> NOTE: Due to many WAFs employing JavaScript-level fingerprinting of web browsers, thermoptic also exposes hooks to utilize the browser for key steps of the scraping process. See this section for more information on this. This reminds me of how Stripe does user tracking for fraude detection https://mtlynch.io/stripe-update/ https://mtlynch.io/stripe-update/ I wonder if thermoptic could handle that.

mips_avatar 1y ago

Cool project!

mandatory 1y ago
Thanks!

Symbiote 1y ago

Oh great /s In a month or two, I can be annoyed when I see some vibe-coded AI startup's script making five million requests a day to work's website with this. They'll have been ignoring the error responses: {"All data is public and available for free download": "https://example.edu/very-large-001.zip"} — a message we also write in the first line of every HTML page source. Then I will spend more time fighting this shit, and less time improving the public data system.

mandatory 1y ago
Feel free to read the README, this was already an ability that startups could pay for using private premium proxy services before thermoptic. Having an open source version allows regular people to do scraping and not just those rich in capital. Much of the best data services on the internet all start with scraping, the README lists many of them.

snowe2010 1y ago

People like you are why independent sites can’t afford to run on the internet anymore.

mandatory 1y ago
They can't? I've run many free independent sites for years, that's news to me.
timbowhite 1y ago
I run independent websites and I'm not broke yet.
1gn15 1y ago
I block all humans (only robots are allowed) and I'm still able to run independent websites.