mateuszbuda

6 ms·

1.
by mateuszbuda 1y ago
Welcome to the zoo! ScrapingFish :)
2.
by mateuszbuda 1y ago
Here you can find statistics based on data scraped from Indie Hackers, but it’s not split by the tool size: https://scrapingfish.com/blog/indie-hackers-revenue
3.
by mateuszbuda 1y ago
There are many different methods used by proxy providers to unethically source their IPs: https://scrapingfish.com/how-ips-for-web-scraping-are-source...
4.
by mateuszbuda 2y ago
There’s AWS Bedrock Knowledge Base (Amazon proprietary RAG solution) which can digest PDFs and, as far as I tested it on real world documents, it works pretty well and is cost effective.
5.
by mateuszbuda 2y ago
You can use a USB hub (example with 20 ports: https://www.sipolar.com/product/a-805p-20-ports-usb-2-0-hub/ ) and attach multiple USB dongles to it. This blog post describes a setup for web scraping: https:/&#
6.
by mateuszbuda 2y ago
Do you have an idea what would be my external IP address? On my phone connected to a mobile network, I get assigned mobile IP address which is my external IP address. It's not attached to the SIM card because it changes when I reconnec
7.
by mateuszbuda 2y ago
Can I use this to run my own mobile network? Is there something like a blank SIM card which I could use for it? I don't need global coverage but is it possible to create my own BTS on a PC (with some antenna connected to it) and then h
8.
by mateuszbuda 2y ago
What exactly doesn't work well? Did you consider playwright?
9.
by mateuszbuda 2y ago
I don’t really know. I don’t write posts to optimize for SEO (include FAQ at the end or something like that) and hope it’s just good content people will share. There are also SEO pages which do not have any useful content. I think I should
10.
by mateuszbuda 2y ago
Content marketing - blog posts with useful content posted around the Internet. Most traffic from HN and Reddit.
11.
by mateuszbuda 2y ago
I think that LLM costs, even GPT-4o, are probably lower compared to proxy costs usually required for web scraping at scale. The cost of residential/mobile proxies is a few $ per GB. If I were to process cleaned data obtained using 1GB
12.
by mateuszbuda 2y ago
I agree that web scraping is a shady business in many cases but there is definitely a difference between setting up a few mobile proxies for yourself and using devices and networks which belong to other people without them even knowing this
13.
by mateuszbuda 2y ago
Here are some insights into how proxies are sourced: https://scrapingfish.com/how-ips-for-web-scraping-are-source... There's also an option to build your own mobile proxy pool which gives you very good reputation IPs f
14.
by mateuszbuda 2y ago
I would also include deceptive credits systems used by SaaS which have usage-based like subscriptions. It’s a bait and switch variant. First, you think one call to the API is one credit but it always turns out that you need calls which cons
15.
by mateuszbuda 2y ago
I tried not buying food with added sugar but it’s surprisingly difficult. Here is an interesting analysis I did some time ago which shows that for half of the food items, sugar is the main ingredient: https://scrapingfish.com&#x2
16.
by mateuszbuda 2y ago
At https://scrapingfish.com/ we have both options, usage based https://scrapingfish.com/buy and subscriptions (monthly unlimited requests plan) https://scrapingfish.com/unlimited . Despite s
17.
by mateuszbuda 2y ago
I’m still working on a web scraping API ( https://scrapingfish.com/ ). For some people it’s evil bot but for others it’s enabler for public data access. I think it’s useful.
18.
by mateuszbuda 2y ago
Anyone can share experience with https://ollama.com/ ?
19.
by mateuszbuda 2y ago
We keep working on web scraping API with custom-made mobile proxy pool: https://scrapingfish.com/ There is no AI in it so far but we consider adding support for parsing the result to extract data using LLM.
20.
by mateuszbuda 3y ago
Two main differentiators. 1. Pricing. We charge for requests and the cost of each successful request is the same as opposed to misleading API credits system used by others. Also, we sell request pack which are valid up to 1 year as opposed
21.
by mateuszbuda 3y ago
To give you a reference point, at https://scrapingfish.com/ we charge $0.002 per API call but in our case, the API call gives you the value by itself: access to mobile proxies and cluster of browsers. For you, I would reco
22.
by mateuszbuda 3y ago
Only around 5% of the products from IndieHackers generate a monthly revenue exceeding ~$8,333 (around ~$100k/year). Source: https://scrapingfish.com/blog/indie-hackers-revenue
23.
by mateuszbuda 3y ago
A web scraping API: https://scrapingfish.com/
24.
by mateuszbuda 3y ago
I've created a mobile proxy pool for a personal project of web scraping real estate data: https://scrapingfish.com/blog/byo-mobile-proxy-for-web-scrap... Then, I expanded the infrastructure and built a web scrapin
25.
by mateuszbuda 3y ago
0.2 cents is how much a single request costs for well-protect website where web scrapers look for emails (e.g. LinkedIn): https://scrapingfish.com/#pricing Paying additional 0.2 cents per request, if it can significantly im
26.
by mateuszbuda 3y ago
Here is an example of how you can scrape data from job boards: https://scrapingfish.com/blog/are-most-rust-jobs-in-crypto
27.
by mateuszbuda 3y ago
In case you need a web scraping tool: https://scrapingfish.com
28.
by mateuszbuda 3y ago
Some parts are outdated now. Bot detection vs web scraping is an ever evolving arms race. Anyway, incolumnitas blog is a great source of information about bot detection and techniques to avoid being detected. We have implemented many succes
29.
by mateuszbuda 3y ago
Exactly, did you build your own solution for this, maybe based on open source projects, or do you rely on external service for scraping? I'm curious too since I'm developing a web scraping API product ( https://scrapingf
30.
by mateuszbuda 3y ago
How did you collect this data? Did you just use webscraping? What tools did you use for this?

More ›