7 ms·

Interesting but seems less powerful than my current setup: - I have mitmproxy to capture the traffic / manipulate the traffic - I have Chrome opened with Sele

Interesting but seems less powerful than my current setup: - I have mitmproxy to capture the traffic / manipulate the traffic - I have Chrome opened with Selenium/Capybara/chromedriver and using mitmproxy - I then browse to the target pages, it records the selected requests and the selected responses - It then replays the requests until they fail (with a delay) I highly recommend mitmproxy, it's extremely powerful: capture traffic, send responses without hitting the server, block/hang requests, modify responses, modify requests/responses headers. Then higher level interfaces can be built on top, Selenium allows you to load Chrome extensions and execute Javascript on any page for instance. You can also manage many tabs at the same time. I could make a blog post/demo if people are interested

marvel_boy 8y ago
>I could make a blog post/demo if people are interested yes, please !
kuhhk 8y ago
A blog article would be nice. Sounds interesting, but I’m having a hard-time understanding. If it’s replaying requests, how do you get it to do things like go to the next pagination and click on all of the next paginated results?
- aboutruby 8y ago
  In my case I can't do the pagination automatically so I have to fetch the pages myself to then have them replayed. In most cases you would capture the request and change the "page=" parameter (either for an HTML page or an API). You could also use selenium to click on each "next page". Could be parallelized with multiple tabs / windows. The only website that blocks me is Bloomberg because they detect mitmproxy (I didn't care enough to make mitmproxy harder to detect). Another detail is that regular Chrome doesn't let you load insecure certificates while chromedriver allows that. Anyway, I will write about all that, I already posted some code on my Twitter: https://twitter.com/localhostdotdev https://twitter.com/localhostdotdev (that I will turn into a blog).

marvel_boy 8y ago

>I could make a blog post/demo if people are interested yes, please !

kuhhk 8y ago

A blog article would be nice. Sounds interesting, but I’m having a hard-time understanding. If it’s replaying requests, how do you get it to do things like go to the next pagination and click on all of the next paginated results?

aboutruby 8y ago
In my case I can't do the pagination automatically so I have to fetch the pages myself to then have them replayed. In most cases you would capture the request and change the "page=" parameter (either for an HTML page or an API). You could also use selenium to click on each "next page". Could be parallelized with multiple tabs / windows. The only website that blocks me is Bloomberg because they detect mitmproxy (I didn't care enough to make mitmproxy harder to detect). Another detail is that regular Chrome doesn't let you load insecure certificates while chromedriver allows that. Anyway, I will write about all that, I already posted some code on my Twitter: https://twitter.com/localhostdotdev https://twitter.com/localhostdotdev (that I will turn into a blog).