4 ms·

Anyone know the specifics of how this works under the hood? I assumed the people over at Twitter/X would aggressively block systems like this from working. Ar

Anyone know the specifics of how this works under the hood? I assumed the people over at Twitter/X would aggressively block systems like this from working. Are they doing something creative to get around that?

swyx 21d ago
sad that i come here for technical information and yours is the first lonely comment asking about how this works
- inigyou 20d ago
  On most of these sites you can just copy the requests the browser makes. There's a walls and ladders game going on, but all the easy to implement walls aren't very high so most sites don't bother building the tallest one possible. You can start by making a burner account and viewing some pages in a browser and watching the developer network tab. Either you'll see some dynamic JSON request that fetches the main page content, or it'll be in the initial page request. In either case you can copy that request and parse it. If it's in the initial page then it's in HTML so parse it with BeautifulSoup or similar. A minority of sites are using TLS fingerprinting, so try curl-impersonate if plain curl doesn't work. At least in Firefox I suggest right click "copy as cURL [command]" and then delete parts of the command to see what's actually needed to get the data, or copy the whole exact thing if you're worried about getting fingerprinted based on the parts you deleted. There's certainly some kind of auth cookie or token. How long does it last? Maybe 15 minutes, an hour, a day, forever? If it's not forever there's a way to refresh it so look through the network requests again, or maybe the network requests when you log in. If you get rate limited it's probably by IP address and you can try a few things. Rebooting your router to change IP? If you have IPv6 you can change your address arbitrarily within a /64 and if the site is really dumb it will consider that a new IP, but most know about that. If your ISP gives you a /48 or /56 you can change subnet. You can try running on cloudflare workers or Amazon lambda, which have big IP pools that continually churn. Some sites that use cloudflare may exempt all of cloudflare from rate limiting and you may greatly benefit from running on a cloudflare worker. You can try a mobile connection that may use CGNAT, forcing the site to choose between unblocking you or blocking your carrier. If all else fails, residential proxy access is either very cheap or very expensive to buy, depending on how much data you need to move. In the end you just need to try it and see.
inigyou 21d ago
No, and nitter the software still works too. Apparently the only barrier he could create was a cease and desist letter.
- mentalgear 21d ago
  TechnoKing!
vit787 21d ago
[flagged]
dawnerd 21d ago
Burner accounts are dirt, dirt cheap. I have several hundred I rotate out. I’m afraid to mention the library I’m using in case it’s not in their cross hairs, but it makes scraping trivial. At least at the scale I’m scraping I don’t even need a proxy. My residential up is enough to keep the blocks away. Only had a handful of accounts banned.
- GaryNumanVevo 20d ago
  Yup, this is the way. I have 5 residential IPs that I rotate dozens of accounts through to do sentiment analysis work.

swyx 21d ago

sad that i come here for technical information and yours is the first lonely comment asking about how this works

inigyou 20d ago
On most of these sites you can just copy the requests the browser makes. There's a walls and ladders game going on, but all the easy to implement walls aren't very high so most sites don't bother building the tallest one possible. You can start by making a burner account and viewing some pages in a browser and watching the developer network tab. Either you'll see some dynamic JSON request that fetches the main page content, or it'll be in the initial page request. In either case you can copy that request and parse it. If it's in the initial page then it's in HTML so parse it with BeautifulSoup or similar. A minority of sites are using TLS fingerprinting, so try curl-impersonate if plain curl doesn't work. At least in Firefox I suggest right click "copy as cURL [command]" and then delete parts of the command to see what's actually needed to get the data, or copy the whole exact thing if you're worried about getting fingerprinted based on the parts you deleted. There's certainly some kind of auth cookie or token. How long does it last? Maybe 15 minutes, an hour, a day, forever? If it's not forever there's a way to refresh it so look through the network requests again, or maybe the network requests when you log in. If you get rate limited it's probably by IP address and you can try a few things. Rebooting your router to change IP? If you have IPv6 you can change your address arbitrarily within a /64 and if the site is really dumb it will consider that a new IP, but most know about that. If your ISP gives you a /48 or /56 you can change subnet. You can try running on cloudflare workers or Amazon lambda, which have big IP pools that continually churn. Some sites that use cloudflare may exempt all of cloudflare from rate limiting and you may greatly benefit from running on a cloudflare worker. You can try a mobile connection that may use CGNAT, forcing the site to choose between unblocking you or blocking your carrier. If all else fails, residential proxy access is either very cheap or very expensive to buy, depending on how much data you need to move. In the end you just need to try it and see.

inigyou 21d ago

No, and nitter the software still works too. Apparently the only barrier he could create was a cease and desist letter.

mentalgear 21d ago
TechnoKing!

vit787 21d ago

[flagged]

dawnerd 21d ago

Burner accounts are dirt, dirt cheap. I have several hundred I rotate out. I’m afraid to mention the library I’m using in case it’s not in their cross hairs, but it makes scraping trivial. At least at the scale I’m scraping I don’t even need a proxy. My residential up is enough to keep the blocks away. Only had a handful of accounts banned.

GaryNumanVevo 20d ago
Yup, this is the way. I have 5 residential IPs that I rotate dozens of accounts through to do sentiment analysis work.