7 ms·

I've been working with scrapers quite a lot. I started with python requests, then to scrapy, then selenium, then selenium via undetected_chromedriver, and once

I've been working with scrapers quite a lot. I started with python requests, then to scrapy, then selenium, then selenium via undetected_chromedriver, and once that started being detected during a chrome update about a year ago, I've switched over to seleniumbase. It got by undetected, but to get it working with pre-downloaded drivers, I had to look into the code. I have never, and I mean never, in all my python years, seen such a horrible mess of code. We are talking 1000lines long methods, with 20-30 different flags and branches Just horrible. I have since switched to Playwright, which seems to be also undetected, and offers a much saner interface.

seleniumbase 2y ago
SeleniumBase modifies the webdriver so that it doesn't get detected when used alongside the CDP stealth mode and methods. It'll download chromedriver for you. Not sure what you mean by the multiple branches, as there's just the primary one. What 1000-line methods are you referring to? By "flags", do you mean the different command-line options available? As for Playwright, they aren't undetected: See https://github.com/microsoft/playwright/issues/23884#issuecomment-1606877360 https://github.com/microsoft/playwright/issues/23884#issueco... - "Playwright is an end-to-end testing framework, where we expect you test on your own environments. Bypassing any form of bot protection is not something we can act on. Thanks for your understanding." On the contrary, SeleniumBase is OK with bypassing bot detection: https://github.com/seleniumbase/SeleniumBase/blob/master/examples/cdp_mode/ReadMe.md https://github.com/seleniumbase/SeleniumBase/blob/master/exa...
- cyanmagenta 2y ago
  Not the commenter, but “multiple branches” in this context is referring to if/else statements in the code, not source-control branches. Similarly, “flags” is referring to function arguments like a boolean “is_original.” More generally, they are just saying that the code has long, complicated, bug-prone functions. That said, I just spent a few minutes browsing the SeleniumBase repro, and honestly it didn’t seem that unusual to me. Would be interested in seeing a specific example of what the commenter had in mind.
mdaniel 2y ago
rather than point-by-point rebuttal as the sibling requests, I think this sums up the coding style pretty well: https://github.com/seleniumbase/SeleniumBase/blob/v4.33.11/seleniumbase/core/capabilities_parser.py#L29-L173 https://github.com/seleniumbase/SeleniumBase/blob/v4.33.11/s...
- seleniumbase 2y ago
  That method came from code that I accepted in a PR from December 31, 2019: https://github.com/seleniumbase/SeleniumBase/pull/459 https://github.com/seleniumbase/SeleniumBase/pull/459 Not a true representation of most of the code today.
  - parineum 2y ago
    The code is in the code base. Presumably, it still gets run. It doesn't make a difference if new code doesn't look like that.
    - seleniumbase 2y ago
      Call it "legacy code" if you'd like. That specific part is from a less common feature for setting options when running on a Selenium Grid. The new CDP Mode isn't compatible with The Grid (since CDP Mode makes direct CDP API calls without making Selenium API calls).
      - MstWntd 2y ago
        it's always easier for people today to look at the work of other people in the past and draw stupid conclusions.. don't mind them..
    - wisty 2y ago
      Bad old code has been battle tested. Bad new code has not, and is more likely to have the show stopper bugs you want to avoid.
edm0nd 2y ago
Not sure if you have explored rolling captcha solving services into your code. Its easy as fuck and you can do it in a few lines of code. Check out DeathByCaptcha or AntiCaptcha. It's like $2.99 per 1,000 successfully solved captchas. I guess my point is, you dont have to be undetected nor write 1000 lines of code to scrape or do whatever you are needing to do always. Saved me a ton of headaches and time when captchas are involved.
- mintzworld 2y ago
  SeleniumBase is free, open-source, can bypass CAPTCHAs with a few lines of code, and it works from the free tier of GitHub Actions.
  - edm0nd 2y ago
    It cant bypass all captchas and thats what im talking about.
    - mintzworld 2y ago
      According to live demos seen in https://www.youtube.com/watch?v=Mr90iQmNsKM https://www.youtube.com/watch?v=Mr90iQmNsKM, it'll bypass Cloudflare, Akamai, Shape Security, DataDome, Incapsula, Kasada, and PerimeterX.
      - edm0nd 2y ago
        Okay, and? DeathByCaptcha can bypass all of those + all other captchas. Write a ton of code or just roll in a solving service API. Ez decision and save a ton of time + get to scraping faster.
        seleniumbase 2y ago
        With SeleniumBase, you can bypass CAPTCHAs with one line of code: `sb.uc_gui_click_captcha()`
pryelluw 2y ago
Enterprise Python code. Somehow ends up being worse than Java enterprise code. I’m too used to it at this point.
- seleniumbase 2y ago
  The "Python vs Java" debate is probably one for a different Hacker News post. :)
  - pryelluw 2y ago
    I meant that some of the code reminds me of enterprise python. The kicker is that code that works > pretty code. People here act as if ugly code is somehow lesser just because it’s ugly. Meanwhile there’s a lot of ugly code making millions of dollars. Didn’t mean to bash your project. Sorry if it came across that way.
    - seleniumbase 2y ago
      It's OK. No offense was taken. It almost looked like the conversation was expanding into a "Python vs Java" debate, but (thankfully) it did not. I've seen both worlds. I've seen advantages to both. I decided to stay in the Python world.
      - pryelluw 2y ago
        Same. Although enterprise python is akin to wrestling a boa constrictor.
bryanrasmussen 2y ago
Maybe I am just a cynic but I would expect Playwright to be detected when using Chrome, I mean I would expect it was to the benefit of Google to make that happen for the sake of making reCaptcha detect bots better. That's actually why I've been scrapping my Playwright automation (because I expect I will encounter problems even if hasn't happened yet, cynical and paranoid) and moving towards writing a browser extension to automate Firefox. Basically my use case is automating tedious things for myself not running bots at scale, so that's why it is imperative not to get caught being "not human", because then risk account problems.
- robertlagrant 2y ago
  How can Google make that happen? Playwright's made by Microsoft. It can use Firefox as a browser as well as Chrome.
  - bryanrasmussen 2y ago
    well I said when using Chrome, how would they make it happen? well it's not like it's using AutoHotkey to automate things, it must be using underlying browser apis to move to move the mouse to mouseover something etc. as opposed to actually using the mouse, as an example naive workflow - I would think the browser sends message to google that instance (unique id) is being automated, recaptcha is detected by chrome on page, chrome calls hidden recaptcha method .setUniqueId(uniqueID) uniqueID is sent back to Google response tells it this is actually an automated browser that is being used as opposed to recaptcha, recaptcha gives 90% chance browser is automated to site, site stops browser access. Site happy it uses recaptcha because they stopped automation. Sure, Playwright can use FF, but most often people just use Chrome.

seleniumbase 2y ago

SeleniumBase modifies the webdriver so that it doesn't get detected when used alongside the CDP stealth mode and methods. It'll download chromedriver for you. Not sure what you mean by the multiple branches, as there's just the primary one. What 1000-line methods are you referring to? By "flags", do you mean the different command-line options available? As for Playwright, they aren't undetected: See https://github.com/microsoft/playwright/issues/23884#issuecomment-1606877360 https://github.com/microsoft/playwright/issues/23884#issueco... - "Playwright is an end-to-end testing framework, where we expect you test on your own environments. Bypassing any form of bot protection is not something we can act on. Thanks for your understanding." On the contrary, SeleniumBase is OK with bypassing bot detection: https://github.com/seleniumbase/SeleniumBase/blob/master/examples/cdp_mode/ReadMe.md https://github.com/seleniumbase/SeleniumBase/blob/master/exa...

cyanmagenta 2y ago
Not the commenter, but “multiple branches” in this context is referring to if/else statements in the code, not source-control branches. Similarly, “flags” is referring to function arguments like a boolean “is_original.” More generally, they are just saying that the code has long, complicated, bug-prone functions. That said, I just spent a few minutes browsing the SeleniumBase repro, and honestly it didn’t seem that unusual to me. Would be interested in seeing a specific example of what the commenter had in mind.

mdaniel 2y ago

rather than point-by-point rebuttal as the sibling requests, I think this sums up the coding style pretty well: https://github.com/seleniumbase/SeleniumBase/blob/v4.33.11/seleniumbase/core/capabilities_parser.py#L29-L173 https://github.com/seleniumbase/SeleniumBase/blob/v4.33.11/s...

seleniumbase 2y ago
That method came from code that I accepted in a PR from December 31, 2019: https://github.com/seleniumbase/SeleniumBase/pull/459 https://github.com/seleniumbase/SeleniumBase/pull/459 Not a true representation of most of the code today.
- parineum 2y ago
  The code is in the code base. Presumably, it still gets run. It doesn't make a difference if new code doesn't look like that.
  - seleniumbase 2y ago
    Call it "legacy code" if you'd like. That specific part is from a less common feature for setting options when running on a Selenium Grid. The new CDP Mode isn't compatible with The Grid (since CDP Mode makes direct CDP API calls without making Selenium API calls).
    - MstWntd 2y ago
      it's always easier for people today to look at the work of other people in the past and draw stupid conclusions.. don't mind them..
  - wisty 2y ago
    Bad old code has been battle tested. Bad new code has not, and is more likely to have the show stopper bugs you want to avoid.

edm0nd 2y ago

Not sure if you have explored rolling captcha solving services into your code. Its easy as fuck and you can do it in a few lines of code. Check out DeathByCaptcha or AntiCaptcha. It's like $2.99 per 1,000 successfully solved captchas. I guess my point is, you dont have to be undetected nor write 1000 lines of code to scrape or do whatever you are needing to do always. Saved me a ton of headaches and time when captchas are involved.

mintzworld 2y ago
SeleniumBase is free, open-source, can bypass CAPTCHAs with a few lines of code, and it works from the free tier of GitHub Actions.
- edm0nd 2y ago
  It cant bypass all captchas and thats what im talking about.
  - mintzworld 2y ago
    According to live demos seen in https://www.youtube.com/watch?v=Mr90iQmNsKM https://www.youtube.com/watch?v=Mr90iQmNsKM, it'll bypass Cloudflare, Akamai, Shape Security, DataDome, Incapsula, Kasada, and PerimeterX.
    - edm0nd 2y ago
      Okay, and? DeathByCaptcha can bypass all of those + all other captchas. Write a ton of code or just roll in a solving service API. Ez decision and save a ton of time + get to scraping faster.
      - seleniumbase 2y ago
        With SeleniumBase, you can bypass CAPTCHAs with one line of code: `sb.uc_gui_click_captcha()`

pryelluw 2y ago

Enterprise Python code. Somehow ends up being worse than Java enterprise code. I’m too used to it at this point.

seleniumbase 2y ago
The "Python vs Java" debate is probably one for a different Hacker News post. :)
- pryelluw 2y ago
  I meant that some of the code reminds me of enterprise python. The kicker is that code that works > pretty code. People here act as if ugly code is somehow lesser just because it’s ugly. Meanwhile there’s a lot of ugly code making millions of dollars. Didn’t mean to bash your project. Sorry if it came across that way.
  - seleniumbase 2y ago
    It's OK. No offense was taken. It almost looked like the conversation was expanding into a "Python vs Java" debate, but (thankfully) it did not. I've seen both worlds. I've seen advantages to both. I decided to stay in the Python world.
    - pryelluw 2y ago
      Same. Although enterprise python is akin to wrestling a boa constrictor.

bryanrasmussen 2y ago

Maybe I am just a cynic but I would expect Playwright to be detected when using Chrome, I mean I would expect it was to the benefit of Google to make that happen for the sake of making reCaptcha detect bots better. That's actually why I've been scrapping my Playwright automation (because I expect I will encounter problems even if hasn't happened yet, cynical and paranoid) and moving towards writing a browser extension to automate Firefox. Basically my use case is automating tedious things for myself not running bots at scale, so that's why it is imperative not to get caught being "not human", because then risk account problems.

robertlagrant 2y ago
How can Google make that happen? Playwright's made by Microsoft. It can use Firefox as a browser as well as Chrome.
- bryanrasmussen 2y ago
  well I said when using Chrome, how would they make it happen? well it's not like it's using AutoHotkey to automate things, it must be using underlying browser apis to move to move the mouse to mouseover something etc. as opposed to actually using the mouse, as an example naive workflow - I would think the browser sends message to google that instance (unique id) is being automated, recaptcha is detected by chrome on page, chrome calls hidden recaptcha method .setUniqueId(uniqueID) uniqueID is sent back to Google response tells it this is actually an automated browser that is being used as opposed to recaptcha, recaptcha gives 90% chance browser is automated to site, site stops browser access. Site happy it uses recaptcha because they stopped automation. Sure, Playwright can use FF, but most often people just use Chrome.