mnmkng

7 ms·

1.
by mnmkng 2y ago
Crawlee isn’t any less configurable than Scrapy. It just uses different, in my personal opinion more approachable, patterns. It makes it easier to start with, but you can tweak whatever you want. Btw, you can add middleware in Crawlee Route
2.
by mnmkng 2y ago
Technically it can. You can log in with the PlaywrightCrawler class without issue. The question is if there’s 2FA as well and how that’s handled. Crawlee does not have any abstraction for handling 2FA as it depends a lot on what verificatio
3.
by mnmkng 2y ago
We tried a self hosted OCR model a few years ago, but the quality and speed wasn’t great. From experience, it’s usually better to reverse engineer the APIs. The more complicated they are, the less they change. So it can sometimes be painful
4.
by mnmkng 2y ago
In one word. Nothing. But I personally think it does some things a little easier, a little faster and little more conveniently than the other libraries and tools out there. Although there’s one thing that the JS version of Crawlee has which
5.
by mnmkng 2y ago
It uses Playwright under the hood, so yes, it can do all of that, and more.
6.
by mnmkng 2y ago
It’s an “old” law that did not consider many intricacies of internet and the platforms that exist on it and it’s mostly made obsolete by EU case law, which has shrunk the definition of a protected database under this law so much that it’s p
7.
by mnmkng 2y ago
Sorry about the confusion. Some features, like the tiered proxies, are not documented properly. You’re absolutely right. Updates will come soon. We wanted to have as many features in the initial release as possible, because we have a local
8.
by mnmkng 2y ago
Oh wow, thanks! Will fix it right away. Crawlee is originally a JS library.
9.
by mnmkng 4y ago
finally!
10.
by mnmkng 4y ago
Yeah I agree, keeping the source HTML is great for debugging or retro-fixing issues. We also like to take screenshots on important errors, when running headless.
11.
by mnmkng 4y ago
Exactly. The dynamic websites need to pull the data from somewhere as well. There's no magic behind it. Either all the data is in the initial payload in some form (not necessarily HTML), or it's downloaded later, again, over HTTP.
12.
by mnmkng 4y ago
If it doesn’t, please make an issue. We know it works from the community but we don’t have tests specifically for Lambda. But it should work, so we’ll help if it doesn’t.
13.
by mnmkng 4y ago
Yes
14.
by mnmkng 4y ago
I get your point and I don't have an objective answer to it. We believe that internet is an open medium and there's immense value for humankind waiting to be discovered and unlocked in all its data. After all, many of the big tech
15.
by mnmkng 4y ago
Yeah, sure. We don't advertise it, but if you get in touch with us on support@apify.com or through the chat widget, we can create a proxy-only plan for you.
16.
by mnmkng 4y ago
With fingerprints it's a tradeoff between having enough of them for large scale scraping and staying consistent with your environment. E.g. you can get exponentially more combinations if you also use Firefox, Webkit, MacOS and Windows
17.
by mnmkng 4y ago
Nice! Good luck with your project. The parsers are available under the utils.social namespace: https://crawlee.dev/api/utils/namespace/social The headless browser utils are under puppeteer and playwright util
18.
by mnmkng 4y ago
The example uses Crawlee already, you can just remove the import { Actor } from 'apify'; and then all references to Actor and either remove them or replace them with Crawlee functions. E.g. await Actor.openKeyValueStore() should b
19.
by mnmkng 4y ago
You can use a headless browser (would recommend PlaywrightCrawler) to log in once and then use the session cookie until it expires in any crawler. When it expires, you can re-login and repeat the process.
20.
by mnmkng 4y ago
Yeah, exactly. Using pure HTTP needs much less resources than running headless browsers, so unless you really need to use a browser, you can save a lot of compute power (and money) by using plain HTTP.
21.
by mnmkng 4y ago
Funny, I never knew about node-osmosis. I like the API in the example. It's a shame that it's no longer maintained.
22.
by mnmkng 4y ago
Thanks! We really love open source and wanted to give back to the community. Crawlee is built on top of other great open-source libraries and projects. It's the best thing about building software.
23.
by mnmkng 4y ago
word!
24.
by mnmkng 4y ago
You can use any captcha solving service with Crawlee, but we plan to add a plugin to make its use much easier. It's on our roadmap.
25.
by mnmkng 4y ago
The ideal approach would depend on your architecture. It's really easy and cheap to create new queues on the Apify platform (we create ~500k every day) so we usually run a crawler per domain. It performs the best and it's the easi
26.
by mnmkng 4y ago
In crawlee, you can use the useIncognitoPages option to create a separate context for each page https://crawlee.dev/api/browser-pool/class/LaunchContext#use... Not sure if it will be enough to offset your RAM
27.
by mnmkng 4y ago
Thank you ;)
28.
by mnmkng 4y ago
Stealth is already included by default, but I love the name :D
29.
by mnmkng 4y ago
Thank you! :)
30.
by mnmkng 4y ago
> While I'm not particularly happy to see JavaScript begin taking over another field as it truly is an awful language, more choice is always better and this project looks valuable enough to make dealing with JS a worthwhile tradeoff

More ›