fbouvier

6 ms·

1.
Lightpanda Agent and PandaScript – LLM at buildtime, not runtime(lightpanda.io)
8 points by fbouvier 3mo ago|0 comments
2.
by fbouvier 8mo ago
Keeping some kind of style information is definitely important to understand the semantics of the webpage.
3.
by fbouvier 8mo ago
Thanks Steeve!
4.
by fbouvier 8mo ago
Yes HTML is too heavy and too expensive for LLM. We are working on a text-based format more suitable for AI.
5.
by fbouvier 8mo ago
Thanks for the feedback, we will try to make this clearer on the website. Lightpanda works with Playwright, and we have some docs[1] and examples[2] available. Web APIs and CDP specifications are huge, so this is still a work in progess. Ma
6.
by fbouvier 8mo ago
Hi, I am Francis, founder of Lightpanda. We wrote a full article explaining why we choose Zig over Rust or C++, if you are interested: https://lightpanda.io/blog/posts/why-we-built-lightpanda-in-... Our goal is to
7.
by fbouvier 2y ago
And some lightweight alternatives like Bellard's QuickJS ( https://bellard.org/quickjs/ ) in C and Kiesel ( https://kiesel.dev/ ) in Zig.
8.
by fbouvier 2y ago
Yes, argentic workflows are one of our use cases for Lightpanda. We skip the graphical rendering of the web page for instant startup, fast execution and low resources usage.
9.
by fbouvier 2y ago
It might works if you need to handle a few websites. But this retro engineering approach is not maintainable if you want to handle hundreds or thousands of websites.
10.
by fbouvier 2y ago
I don't expect it to change a lot. All the main components are there, it's mainly a question of coverage now.
11.
by fbouvier 2y ago
Not now but we might do it in the future. It's easy to export a Zig project as a C ABI library.
12.
by fbouvier 2y ago
Thank you! Happy if you use it for your e2e tests in your servers, it's an open-source project! Of course it's quite easy to spin a local instance of a headless browser for occasional use. But having a production platform is anoth
13.
by fbouvier 2y ago
It was my first idea. Forking Chromium has obvious advantages (compatibility). But it's not architectured for that. The renderer is everywhere. I'm not saying it's impossible, just that it did look more difficult to me than s
14.
by fbouvier 2y ago
For now we just support CDP. But Selenium is definitely in our roadmap.
15.
by fbouvier 2y ago
Yes but WebKit is not a browser per se, it's a rendering engine. It's less resource-intensive than Chrome, but here we are talking orders of magnitude between Lightpanda and Chrome. If you are ~10x faster while using ~10x less RAM
16.
by fbouvier 2y ago
There are a lot of uses cases: - LLM training (RAG, fine tuning) - AI agents - scraping - SERP - testing - any kind of web automation basically Bot protection of course might be a problem but it depends also on the volume of requests, IP, a
17.
Show HN: Lightpanda, an open-source headless browser in Zig(github.com)
319 points by fbouvier 2y ago|137 comments
18.
by fbouvier 2y ago
Thanks! No we haven't worked on bot detection.
19.
by fbouvier 2y ago
Every tool can be used in a good or bad way, Chrome, Firefox, cURL, etc. It's not the browser who doesn't play nicely, it's the user. It's the user's responsibility to behave well, like in life :)
20.
by fbouvier 2y ago
We did not run benchmarks with chrome-headless-shell (aka the old headless mode) but I guess that performance wise it's on the same scale as the new headless mode.
21.
by fbouvier 2y ago
I fully understand your concern and agree that scrapers shouldn't be hurting web servers. I don't think they are using our browser :) But in my opinion, blocking a browser as such is not the right solution. In this case, it's
22.
by fbouvier 2y ago
WASM support is not there yet but it's on the roadmap and we had it in our mind since the beginning of the project, and have made our dev choices accordingly. So yes it could be used in a serverless platform like Cloudflare workers. Ou
23.
by fbouvier 2y ago
We had some discussions about it. It seems to us that AGPL will ensure that a company running our browser in a cloud managed offer will have to keep its modifications open for the community. We might be wrong, maybe AGPL will damage the pro
24.
by fbouvier 2y ago
Yes, that's true. It's a balance to find between RAM and speed. I was thinking more on use cases that require to disable JIT anyway (WASM, iOS integration, security).
25.
by fbouvier 2y ago
In the past I've run hundreds of headful instances of Chrome in a server environment using Xvfb. It was not a pleasant experience :)
26.
by fbouvier 2y ago
I guess it depends on the scale of your requests. When you want to browse a few websites from time to time, a local headful browser might be a solution. But when you have thousands or millions of webpages, you need a server environment and
27.
by fbouvier 2y ago
You're right, the debugging part is a good use case for graphical rendering in a headless environment. I see it as a build time/runtime question. At build (dev) time I want to have a graphical response (debugging, computer vision,
28.
by fbouvier 2y ago
I believe it will be still significantly lower as we skip the graphical rendering. But to validate that we need to increase our Web APIs coverage.
29.
by fbouvier 2y ago
The cost of the browser part is still a problem. In our previous startup, we were scraping >20 millions of webpages per day, with thousands of instances of Chrome headless in parallel. Regarding the RAM usage, it's still ~10x better
30.
by fbouvier 2y ago
The most important "feature" is to increase our Web APIs coverage :) But of course we plan to add others features, including - tight integration with LLM - embed mode (as a C library and as a WASM module) so you can add a real bro

More ›