4 ms·
Use Tor to proxy. Every time you get a connection error, send a SIGHUP to the Tor daemon, which will tell the Tor daemon to establish a new connection, thereby
by earthnail 20d ago
Use Tor to proxy. Every time you get a connection error, send a SIGHUP to the Tor daemon, which will tell the Tor daemon to establish a new connection, thereby giving you a new IP. You may not get as good results as a residential proxy, but these proxies are prohibitively expensive if you're doing this for your own personal or academic use, and the Tor method works surprisingly well.
- vivzkestrel 20d ago- am not familiar with tor except for the fact that it uses something called onion with multiple hops and is very hard to trace making it privacy friendly - are we talking here about using tor programmatically? - what I am referring to on my end is how to go about programmatically downloading a 100 million videos one after the other on s3 - what kind of bot detection / evasion / ip address / proxy rotation mechanisms are needed - are you familiar with libraries in the pypi ecosystem capable of tackling some of these concerns? - is aiohttp / asyncio python enough for this paired with multiprocessing or do we have better options?
- earthnail 20d ago- Use the python Library stem to control the Tor daemon and send sighup commands when necessary - yt dlp has builtin support for tor - write the retry logic using your favourite coding AI - spin up a cheap cloud instance and let it run
- m00dy 20d agoDude, Tor's exit nodes are flagged everywhere.