8 ms·

Show HN: Lightpanda, an open-source headless browser in Zig

319 points by fbouvier 2y ago

We’re Francis and Pierre, and we're excited to share Lightpanda (https://lightpanda.io https://lightpanda.io), an open-source headless browser we’ve been building for the past 2 years from scratch in Zig (not dependent on Chromium or Firefox). It’s a faster and lighter alternative for headless operations without any graphical rendering. Why start over? We’ve worked a lot with Chrome headless at our previous company, scraping millions of web pages per day. While it’s powerful, it’s also heavy on CPU and memory usage. For scraping at scale, building AI agents, or automating websites, the overheads are high. So we asked ourselves: what if we built a browser that only did what’s absolutely necessary for headless automation? Our browser is made of the following main components: - an HTTP loader - an HTML parser and DOM tree (based on Netsurf libs) - a Javascript runtime (v8) - partial web APIs support (currently DOM and XHR/Fetch) - and a CDP (Chrome Debug Protocol) server to allow plug & play connection with existing scripts (Puppeteer, Playwright, etc). The main idea is to avoid any graphical rendering and just work with data manipulation, which in our experience covers a wide range of headless use cases (excluding some, like screenshot generation). In our current test case Lightpanda is roughly 10x faster than Chrome headless while using 10x less memory. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them. It's a beta version, so expect most websites to fail or crash. The plan is to increase coverage over time. We chose Zig for its seamless integration with C libs and its comptime feature that allow us to generate bi-directional Native to JS APIs (see our zig-js-runtime lib https://github.com/lightpanda-io/zig-js-runtime https://github.com/lightpanda-io/zig-js-runtime). And of course for its performance :) As a company, our business model is based on a Managed Cloud, browser as a service. Currently, this is primarily powered by Chrome, but as we integrate more web APIs it will gradually transition to Lightpanda. We would love to hear your thoughts and feedback. Where should we focus our efforts next to support your use cases?

fbouvier 2y ago
Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or crash. The plan is to increase coverage over time. Happy to answer any questions.
- niutech 2y ago
  Congratulations! But does it support Google Account login? And ReCAPTCHA?
- toobulkeh 2y ago
  I’d love to see better optimized web socket support and “save” features that cache LLM queries to optimize fallback
- JoelEinbinder 2y ago
  When I've talked to people running this kind of ai scraping/agent workflow, the costs of the AI parts dwarf that of the web browser parts. This causes computational cost of the browser to become irrelevant. I'm curious what situation you got yourself in where optimizing the browser results in meaningful savings. I'd also like to be in that place! I think your ram usage benchmark is deceptive. I'd expect a minimal browser to have much lower peak memory usage than chrome on a minimal website. But it should even out or get worse as the websites get richer. The nature of web scraping is that the worst sites take up the vast majority of your cpu cycles. I don't think lowering the ram usage of the browser process will have much real world impact.
  - refulgentis 2y ago
    Generally, for consumer use cases, it's best to A) do it locally, preserving some of the original web contract B) run JS to get actual content C) post-process to reduce inference cost D) get latency as low as possible Then, as the article points out, the Big Guns making the LLMs are a big use case for this because they get a 10x speedup and can begin contemplating running JS. It sounds like the people you've talked to are in a messy middle: no incentive to improve efficiency of loading pages, simply because there's something else in the system that has a fixed cost to it. I'm not sure why that would rule out improving anything else, it doesn't seem they should be stuck doing nothing other than flailing around for cheaper LLM inference. > I think your ram usage benchmark is deceptive. I'd expect a minimal browser to have much lower peak memory usage than chrome on a minimal website. I'm a bit lost, the ram usage benchmark says its ~10x less, and you feel its deceptive because you'd expect ram usage to be less? Steelmanning: 10% of Chrome's usage is still too high?
cropcirclbureau 2y ago
Pretty cool. Do you have a list of features you plan to support and plan to cut? Also, how much does this differ from the DOM impls that test frameworks use? I recall Jest or someone sporting such a feature.
- fbouvier 2y ago
  The most important "feature" is to increase our Web APIs coverage :) But of course we plan to add others features, including - tight integration with LLM - embed mode (as a C library and as a WASM module) so you can add a real browser to your project the same way you add libcurl
  - andrethegiant 2y ago
    Could it potentially fit in a Cloudflare worker? Workers are also V8 and can run wasm, but are constrained to 128MB RAM and 10MB zipped bundle size
    - fbouvier 2y ago
      WASM support is not there yet but it's on the roadmap and we had it in our mind since the beginning of the project, and have made our dev choices accordingly. So yes it could be used in a serverless platform like Cloudflare workers. Our startup time is a huge advantage here (20ms vs 600ms for Chrome headless in our local tests). Regarding v8 in Cloudflare workers I think we can not used directly, ie. we still need to embed a JS engine in the wasm module.
m3kw9 2y ago
How does this work because the browser needs to render a page and the vision model needs to know where a button is, so it still needs to see an image. How does headless make it easier?
- katiehallett 2y ago
  Headless mode skips the visual rendering meant for humans, but the DOM structure and layout still exist, allowing the model to parse elements programmatically (e.g. button locations). Instead of 'seeing' an image, the model interacts with the page's underlying structure, which is faster and more efficient. Our browser removes the rendering engine as well, so it won't handle 100% of automation use cases, but it's also what allows us to be faster and lighter than Chrome in headless mode.
  - wiradikusuma 2y ago
    But what if the human programmer needs to visually verify that their code works by eyeballing which element got selected, etc?
    - fbouvier 2y ago
      You're right, the debugging part is a good use case for graphical rendering in a headless environment. I see it as a build time/runtime question. At build (dev) time I want to have a graphical response (debugging, computer vision, etc.). And then, when the script is ready, I can use Lightpanda at runtime as a lightweight alternative.
      - codetrotter 2y ago
        I was doing a personal side project for a while where I was trying to make my own little Wayback Machine-alike. Mine was very rudimentary, built on top of Firefox and WebDriver plus Squid proxy. For debugging purposes you could have your headless browser function as a HTTP Proxy Server, maybe? And in your headless browser you could capture a static snapshot of the DOM after your JavaScript runtime has executed the scripts for the page. Similar to how the archive.today guy serves static snapshots of websites. And then developers using your headless browser could point their Firefox or Chrome browser to the HTTP Proxy server hosted by your headless browser program, in order to get a static snapshot view of what the DOM is like after your headless browser has executed JavaScript from the page. And then Firefox or Chrome will render static HTML view of what the page looked like to your headless browser, that the developer can inspect to make decisions about further interactions with the page. As a tool for debugging.
weinzierl 2y ago
If I don't need JavaScript or any interactivity, just modern HTML + modern CSS, is there any modern lightweight renderer to png or svg? Something in the spirit of wkhtmltoimage or WeasyPrint that does not require a full blown browser but more modern with support of recent HTML and CSS? In a sense this is Lightpanda's complement to a "full panda". Just the fully rendered DOM to pixels.
- nicoburns 2y ago
  We're working on this here: https://github.com/DioxusLabs/blitz https://github.com/DioxusLabs/blitz See the "screenshot" example for rendering to png. There's no SVG backend currently, but one could be added. (proper announcement of project coming soon)
Kathc 2y ago
An open-source browser built from scratch is bold. What inspired the development of Lightpanda?
- katiehallett 2y ago
  Thanks! The three of us worked together at our former company - ecomm saas start up where we spent a ton of $ on scraping infrastructure spinning up headless Chrome instances. It started out as more of an R&D thesis - is it possible to strip out graphical rendering from Chrome headless? Turns out no - so we tried to build it from scratch. And the beta results validated the thesis. I wrote a whole thing about it here if you're interested in delving deeper https://substack.thewebscraping.club/p/rethinking-the-web-browser https://substack.thewebscraping.club/p/rethinking-the-web-br...
  - corford 2y ago
    Not sure what category of ecomm sites you were scraping but I scrape >10million ecomm URLs daily and, honestly, in my experience the compute is not a major issue (8 times out of 10 you can either use API endpoints and/or session stuffing to avoid needing a browser for every request; and in the 2 out of 10 sites where you really need a browser for all requests it's usually to circumvent aggressive anti-bot which means you're very likely going to need full chrome or FF anyway - and you can parallelise quite effectively across tabs). One niche where I could definitely see a use for this though is scraping terribly coded sites that need some JS execution to safely get the data you want (e.g. they do some bonkers client side calculations that you don't want to reverse engineer). It would be nice to not pay the perf tax of chrome in these cases. Having said all of that, I have to say from a geek perspective it's super neat what you guys are hacking on! Zig+V8+CDP bindings is very cool.
    - zlagen 2y ago
      fully agree here, using a browser for everything is the dumb way. You just usually use it to circumvent the blocking and then reuse the cookies to call the endpoints directly.
monkmartinez 2y ago
This is pretty neat, but I have to ask; Why does everyone want to build and/or use a headless browser? When I use pyautogui and my desktop chrome app I never have problems with captchas or trigger bot detectors. When I use a "headless" playwright, selenium, or puppeteer, I almost always run into problems. My conclusion is that "headless" scrapping creates more problems than it solves. Why don't we use the chrome, firefox, safari, or edge that we are using on a day to day basis?
- deleted 2y ago
  [deleted]
- fbouvier 2y ago
  I guess it depends on the scale of your requests. When you want to browse a few websites from time to time, a local headful browser might be a solution. But when you have thousands or millions of webpages, you need a server environment and a headless browser.
  - fbouvier 2y ago
    In the past I've run hundreds of headful instances of Chrome in a server environment using Xvfb. It was not a pleasant experience :)
kavalg 2y ago
Why AGPL? I am not blaming you. I am just curious about the reasoning behind your choice.
- fbouvier 2y ago
  We had some discussions about it. It seems to us that AGPL will ensure that a company running our browser in a cloud managed offer will have to keep its modifications open for the community. We might be wrong, maybe AGPL will damage the project more than eg. Apache2. In that case we will reconsider our choice. It's always easier this way :) Our underlying library https://github.com/lightpanda-io/zig-js-runtime https://github.com/lightpanda-io/zig-js-runtime is licensed with Apache2.
cratermoon 2y ago
So is this the scraper we need to block? https://news.ycombinator.com/item?id=42750420 https://news.ycombinator.com/item?id=42750420
- fbouvier 2y ago
  I fully understand your concern and agree that scrapers shouldn't be hurting web servers. I don't think they are using our browser :) But in my opinion, blocking a browser as such is not the right solution. In this case, it's the user who should be blocked, not the browser.
  - jjcoffman 2y ago
    If your browser doesn't play nicely and obey robots.txt when its headless I don't think it's that crazy to block the browser and not the user.
    - fbouvier 2y ago
      Every tool can be used in a good or bad way, Chrome, Firefox, cURL, etc. It's not the browser who doesn't play nicely, it's the user. It's the user's responsibility to behave well, like in life :)
    - slt2021 2y ago
      it is trivial to spoof user-agent, if you want to stop a motivated scraper, you need a different solution that exploits the fact that robots use headless browser
      - sangnoir 2y ago
        > it is trivial to spoof user-agent It's also trivial to detect spoofed user agents via fingerprinting. The best defense against scrapers is done in layers, with user-agent name block as the bare minimum.
    - hansvm 2y ago
      The first thing that came to mind when I saw this project wasn't scraping (where I'd typically either want a less detectible browser or a more performant option), but as a browser engine that's actually sane to link against if I wanted to, e.g., write a modern TUI browser. Banning the root library (even if you could with UA spoofing and whatnot) is right up there with banning Chrome to keep out low-wage scraping centers and their armies of employees. It's not even a little effective also risks significant collateral damage.
surfmike 2y ago
Another browser in this space is https://ultralig.ht/ https://ultralig.ht/, it's geared for in-game UI but I wonder how easy it would be to retool it for a similar use case.
gwittel 2y ago
Interesting. Looks really neat! How do you deal with anti bot stuff like Fingerprintjs, Cloudflare turnstile, etc? Maybe you’re new enough to not get flagged but I find this (and CDP) a challenge at times with these anti-bot systems.
frankgrecojr 2y ago
The hello world example does not work. In fact, no website I've tried works. It's usually always panics. For the example in the readme, the errors are: ``` ./lightpanda-aarch64-macos --host 127.0.0.1 --port 9222 info(websocket): starting blocking worker to listen on 127.0.0.1:9222 info(server): accepting new conn... info(server): client connected info(browser): GET https://wikipedia.com/ https://wikipedia.com/ 200 info(browser): fetch https://wikipedia.com/portal/wikipedia.org/assets/js/index-24c3e2ca18.js https://wikipedia.com/portal/wikipedia.org/assets/js/index-2...: http.Status.ok info(browser): eval script portal/wikipedia.org/assets/js/index-24c3e2ca18.js: ReferenceError: location is not defined info(browser): fetch https://wikipedia.com/portal/wikipedia.org/assets/js/gt-ie9-ce3fe8e88d.js https://wikipedia.com/portal/wikipedia.org/assets/js/gt-ie9-...: http.Status.ok error(events): event handler error: error.JSExecCallback info(events): event handler error try catch: TypeError: Cannot read properties of undefined (reading 'length') info(server): close cmd, closing conn... info(server): accepting new conn... thread 5274880 panic: attempt to use null value zsh: abort ./lightpanda-aarch64-macos --host 127.0.0.1 --port 9222 ```
- lbotos 2y ago
  Not OP -- do you have some kind of proxy or firewall? Looks like you couldn't download https://wikipedia.com/portal/wikipedia.org/assets/js/gt-ie9-ce3fe8e88d.js https://wikipedia.com/portal/wikipedia.org/assets/js/gt-ie9-... for some reason. In my contributions to joplin s3 backend "Cannot read properties of undefined (reading 'length')" was usually when you were trying to access an object that wasn't instantiated. (Can't figure out length of <undefined>) So for some reason it seems you can't execute JS?
- zelcon 2y ago
  That's Zig for you. A ``modern'' systems programming language with no borrow checker or even RAII.
  - hansvm 2y ago
    Those statements are mostly true and also worth talking about, but they're not pertinent to that error (remotely provided JS not behaving correctly), or the eventual crash (which you'd cause exactly the same way for the same reason in Rust with a .unwrap() call).
dang 2y ago
(This was on the frontpage as https://news.ycombinator.com/item?id=42812859 https://news.ycombinator.com/item?id=42812859 but someone pointed out to me that it had been a Show HN a few weeks ago: https://news.ycombinator.com/item?id=42430629 https://news.ycombinator.com/item?id=42430629, so I've made a fresh copy of that submission and moved the comments hither. I hope that's ok with everyone!)
zlagen 2y ago
what do you think would be the use cases for this project? being lightweight is awesome but usually you need a real browser for most use cases. Testing sites and scraping for example. It may work for some scraping use cases but I think that if the site uses any kind of bot blocking this is not going to cut it.
- fbouvier 2y ago
  There are a lot of uses cases: - LLM training (RAG, fine tuning) - AI agents - scraping - SERP - testing - any kind of web automation basically Bot protection of course might be a problem but it depends also on the volume of requests, IP, and other parameters. AI agents will do more and more actions on behalf of humans in the future and I believe the bot protection mechanism will evolve to include them as legit.
  - zlagen 2y ago
    thanks, it doesn't seem like it's the direction it's going at the moment. If you look at the robots.txt of many websites, they are actually banning AI bots from crawling the site. To me it seems more likely that each site will have its own AI agent to perform operations but controlled by the site.
deleted 2y ago
[deleted]
the__alchemist 2y ago
I have a meta question from browsing the repo: Why do C, C++, and Zig code bases, by convention, include a license at the top of every module" IMO it makes more sense to insetead include of an overview of the module's purpose, and how it fits in with the rest of the program, and one license at the top-level, as the project already has.
- AndyKelley 2y ago
  100% of my projects, including the Zig compiler itself, have only the license file at the root of the project tree, except of course for files that were copy pasted from other projects.
zelcon 2y ago
Why didn't you just fork Chromium and strip out the renderer? This is guaranteed to bitrot when the web standards change unless you keep up with it forever and have perpetual funding. Yes, modifying Chromium is hard, but this seems harder.
- tetris11 2y ago
  Why do anything: because it shows what's possible, and makes the next effort that much more easier. I call this process of frontier effort and discovery: "science"
  - zelcon 2y ago
    Redoing what others have already done is not what I think of when I hear "frontier effort"
- cxr 2y ago
  > modifying Chromium is hard, but this seems harder Prove it.
- fbouvier 2y ago
  It was my first idea. Forking Chromium has obvious advantages (compatibility). But it's not architectured for that. The renderer is everywhere. I'm not saying it's impossible, just that it did look more difficult to me than starting over. And starting from scratch has other benefits. We own the codebase and thus it's easier for us to add new features like LLM integrations. Plus reducing binary size and startup time, mandatory for embedding it (as a WASM module or as C lib).
  - oever 2y ago
    The Chromium/Webkit renderer used to have multiple rendering backends. You might use or add a no-op backend.
stuckkeys 2y ago
How does it do against captchas?
psanchez 2y ago
I think this is a really cool project. Scrapping aside, I would definitely use this with playwright for end2end tests if it had 100% compatibility with chrome and ran with a fraction of the time/memory. At my company we have a small project where we are running the equivalent of 6.5 hours of end2end tests daily using playwright. Running the tests in parallel takes around half an hour. Your project is still in very early stages, but assuming 10x speed, that would mean we could pass all our tests in roughtly 3 min (best case scenario). That being said, I would make use of your browser, but would likely not make use of your business offering (our tests require internal VPN, have some custom solution for reporting, would be a lot of work to change for little savings; we run all tests currently in spot/preemptible instances which are already 80% cheaper). Business-wise I found very little info on your website. "4x the efficiency at half the cost" is a good catch phrase, but compared to what? I mean, you can have servers in Hetzner or in AWS and one is already a fraction of the cost of the other. How convenient is to launch things on your remote platform vs launch them locally or setting it up? does it provide any advantages in the case of web scrapping compared to other solutions? how parallelizable is it? Do you have any paying customers already? Supercool tech project. Best of luck!
- fbouvier 2y ago
  Thank you! Happy if you use it for your e2e tests in your servers, it's an open-source project! Of course it's quite easy to spin a local instance of a headless browser for occasional use. But having a production platform is another story (monitoring, maintenance, security and isolation, scalability), so there are business use cases for a managed version.
randomMatrix101 2y ago
Very cool project, congrats guys!
optixyt 2y ago
The second social media botters find this.
evanjrowley 2y ago
I'm interested to see if this could be made to work as a drop-in replacement for the headless Chromium that Hoarder uses to archive web content. I don't have a problem with the current Hoarder solution, but it would be nice to use something that requires less RAM.
deleted 2y ago
[deleted]

319 points by fbouvier 2y ago

fbouvier 2y ago

Author here. The browser is made from scratch (not based on Chromium/Webkit), in Zig, using v8 as a JS engine. Our idea is to build a lightweight browser optimized for AI use cases like LLM training and agent workflows. And more generally any type of web automation. It's a work in progress, there are hundreds of Web APIs, and for now we just support some of them (DOM, XHR, Fetch). So expect most websites to fail or crash. The plan is to increase coverage over time. Happy to answer any questions.

niutech 2y ago
Congratulations! But does it support Google Account login? And ReCAPTCHA?
toobulkeh 2y ago
I’d love to see better optimized web socket support and “save” features that cache LLM queries to optimize fallback
JoelEinbinder 2y ago
When I've talked to people running this kind of ai scraping/agent workflow, the costs of the AI parts dwarf that of the web browser parts. This causes computational cost of the browser to become irrelevant. I'm curious what situation you got yourself in where optimizing the browser results in meaningful savings. I'd also like to be in that place! I think your ram usage benchmark is deceptive. I'd expect a minimal browser to have much lower peak memory usage than chrome on a minimal website. But it should even out or get worse as the websites get richer. The nature of web scraping is that the worst sites take up the vast majority of your cpu cycles. I don't think lowering the ram usage of the browser process will have much real world impact.
- refulgentis 2y ago
  Generally, for consumer use cases, it's best to A) do it locally, preserving some of the original web contract B) run JS to get actual content C) post-process to reduce inference cost D) get latency as low as possible Then, as the article points out, the Big Guns making the LLMs are a big use case for this because they get a 10x speedup and can begin contemplating running JS. It sounds like the people you've talked to are in a messy middle: no incentive to improve efficiency of loading pages, simply because there's something else in the system that has a fixed cost to it. I'm not sure why that would rule out improving anything else, it doesn't seem they should be stuck doing nothing other than flailing around for cheaper LLM inference. > I think your ram usage benchmark is deceptive. I'd expect a minimal browser to have much lower peak memory usage than chrome on a minimal website. I'm a bit lost, the ram usage benchmark says its ~10x less, and you feel its deceptive because you'd expect ram usage to be less? Steelmanning: 10% of Chrome's usage is still too high?

cropcirclbureau 2y ago

Pretty cool. Do you have a list of features you plan to support and plan to cut? Also, how much does this differ from the DOM impls that test frameworks use? I recall Jest or someone sporting such a feature.

fbouvier 2y ago
The most important "feature" is to increase our Web APIs coverage :) But of course we plan to add others features, including - tight integration with LLM - embed mode (as a C library and as a WASM module) so you can add a real browser to your project the same way you add libcurl
- andrethegiant 2y ago
  Could it potentially fit in a Cloudflare worker? Workers are also V8 and can run wasm, but are constrained to 128MB RAM and 10MB zipped bundle size
  - fbouvier 2y ago
    WASM support is not there yet but it's on the roadmap and we had it in our mind since the beginning of the project, and have made our dev choices accordingly. So yes it could be used in a serverless platform like Cloudflare workers. Our startup time is a huge advantage here (20ms vs 600ms for Chrome headless in our local tests). Regarding v8 in Cloudflare workers I think we can not used directly, ie. we still need to embed a JS engine in the wasm module.

m3kw9 2y ago

How does this work because the browser needs to render a page and the vision model needs to know where a button is, so it still needs to see an image. How does headless make it easier?

katiehallett 2y ago
Headless mode skips the visual rendering meant for humans, but the DOM structure and layout still exist, allowing the model to parse elements programmatically (e.g. button locations). Instead of 'seeing' an image, the model interacts with the page's underlying structure, which is faster and more efficient. Our browser removes the rendering engine as well, so it won't handle 100% of automation use cases, but it's also what allows us to be faster and lighter than Chrome in headless mode.
- wiradikusuma 2y ago
  But what if the human programmer needs to visually verify that their code works by eyeballing which element got selected, etc?
  - fbouvier 2y ago
    You're right, the debugging part is a good use case for graphical rendering in a headless environment. I see it as a build time/runtime question. At build (dev) time I want to have a graphical response (debugging, computer vision, etc.). And then, when the script is ready, I can use Lightpanda at runtime as a lightweight alternative.
    - codetrotter 2y ago
      I was doing a personal side project for a while where I was trying to make my own little Wayback Machine-alike. Mine was very rudimentary, built on top of Firefox and WebDriver plus Squid proxy. For debugging purposes you could have your headless browser function as a HTTP Proxy Server, maybe? And in your headless browser you could capture a static snapshot of the DOM after your JavaScript runtime has executed the scripts for the page. Similar to how the archive.today guy serves static snapshots of websites. And then developers using your headless browser could point their Firefox or Chrome browser to the HTTP Proxy server hosted by your headless browser program, in order to get a static snapshot view of what the DOM is like after your headless browser has executed JavaScript from the page. And then Firefox or Chrome will render static HTML view of what the page looked like to your headless browser, that the developer can inspect to make decisions about further interactions with the page. As a tool for debugging.

weinzierl 2y ago

If I don't need JavaScript or any interactivity, just modern HTML + modern CSS, is there any modern lightweight renderer to png or svg? Something in the spirit of wkhtmltoimage or WeasyPrint that does not require a full blown browser but more modern with support of recent HTML and CSS? In a sense this is Lightpanda's complement to a "full panda". Just the fully rendered DOM to pixels.

nicoburns 2y ago
We're working on this here: https://github.com/DioxusLabs/blitz https://github.com/DioxusLabs/blitz See the "screenshot" example for rendering to png. There's no SVG backend currently, but one could be added. (proper announcement of project coming soon)

Kathc 2y ago

An open-source browser built from scratch is bold. What inspired the development of Lightpanda?

katiehallett 2y ago
Thanks! The three of us worked together at our former company - ecomm saas start up where we spent a ton of $ on scraping infrastructure spinning up headless Chrome instances. It started out as more of an R&D thesis - is it possible to strip out graphical rendering from Chrome headless? Turns out no - so we tried to build it from scratch. And the beta results validated the thesis. I wrote a whole thing about it here if you're interested in delving deeper https://substack.thewebscraping.club/p/rethinking-the-web-browser https://substack.thewebscraping.club/p/rethinking-the-web-br...
- corford 2y ago
  Not sure what category of ecomm sites you were scraping but I scrape >10million ecomm URLs daily and, honestly, in my experience the compute is not a major issue (8 times out of 10 you can either use API endpoints and/or session stuffing to avoid needing a browser for every request; and in the 2 out of 10 sites where you really need a browser for all requests it's usually to circumvent aggressive anti-bot which means you're very likely going to need full chrome or FF anyway - and you can parallelise quite effectively across tabs). One niche where I could definitely see a use for this though is scraping terribly coded sites that need some JS execution to safely get the data you want (e.g. they do some bonkers client side calculations that you don't want to reverse engineer). It would be nice to not pay the perf tax of chrome in these cases. Having said all of that, I have to say from a geek perspective it's super neat what you guys are hacking on! Zig+V8+CDP bindings is very cool.
  - zlagen 2y ago
    fully agree here, using a browser for everything is the dumb way. You just usually use it to circumvent the blocking and then reuse the cookies to call the endpoints directly.

monkmartinez 2y ago

This is pretty neat, but I have to ask; Why does everyone want to build and/or use a headless browser? When I use pyautogui and my desktop chrome app I never have problems with captchas or trigger bot detectors. When I use a "headless" playwright, selenium, or puppeteer, I almost always run into problems. My conclusion is that "headless" scrapping creates more problems than it solves. Why don't we use the chrome, firefox, safari, or edge that we are using on a day to day basis?

deleted 2y ago
[deleted]
fbouvier 2y ago
I guess it depends on the scale of your requests. When you want to browse a few websites from time to time, a local headful browser might be a solution. But when you have thousands or millions of webpages, you need a server environment and a headless browser.
- fbouvier 2y ago
  In the past I've run hundreds of headful instances of Chrome in a server environment using Xvfb. It was not a pleasant experience :)

kavalg 2y ago

Why AGPL? I am not blaming you. I am just curious about the reasoning behind your choice.

fbouvier 2y ago
We had some discussions about it. It seems to us that AGPL will ensure that a company running our browser in a cloud managed offer will have to keep its modifications open for the community. We might be wrong, maybe AGPL will damage the project more than eg. Apache2. In that case we will reconsider our choice. It's always easier this way :) Our underlying library https://github.com/lightpanda-io/zig-js-runtime https://github.com/lightpanda-io/zig-js-runtime is licensed with Apache2.

cratermoon 2y ago

So is this the scraper we need to block? https://news.ycombinator.com/item?id=42750420 https://news.ycombinator.com/item?id=42750420

fbouvier 2y ago
I fully understand your concern and agree that scrapers shouldn't be hurting web servers. I don't think they are using our browser :) But in my opinion, blocking a browser as such is not the right solution. In this case, it's the user who should be blocked, not the browser.
- jjcoffman 2y ago
  If your browser doesn't play nicely and obey robots.txt when its headless I don't think it's that crazy to block the browser and not the user.
  - fbouvier 2y ago
    Every tool can be used in a good or bad way, Chrome, Firefox, cURL, etc. It's not the browser who doesn't play nicely, it's the user. It's the user's responsibility to behave well, like in life :)
  - slt2021 2y ago
    it is trivial to spoof user-agent, if you want to stop a motivated scraper, you need a different solution that exploits the fact that robots use headless browser
    - sangnoir 2y ago
      > it is trivial to spoof user-agent It's also trivial to detect spoofed user agents via fingerprinting. The best defense against scrapers is done in layers, with user-agent name block as the bare minimum.
  - hansvm 2y ago
    The first thing that came to mind when I saw this project wasn't scraping (where I'd typically either want a less detectible browser or a more performant option), but as a browser engine that's actually sane to link against if I wanted to, e.g., write a modern TUI browser. Banning the root library (even if you could with UA spoofing and whatnot) is right up there with banning Chrome to keep out low-wage scraping centers and their armies of employees. It's not even a little effective also risks significant collateral damage.

surfmike 2y ago

Another browser in this space is https://ultralig.ht/ https://ultralig.ht/, it's geared for in-game UI but I wonder how easy it would be to retool it for a similar use case.

gwittel 2y ago

Interesting. Looks really neat! How do you deal with anti bot stuff like Fingerprintjs, Cloudflare turnstile, etc? Maybe you’re new enough to not get flagged but I find this (and CDP) a challenge at times with these anti-bot systems.

frankgrecojr 2y ago

The hello world example does not work. In fact, no website I've tried works. It's usually always panics. For the example in the readme, the errors are: ``` ./lightpanda-aarch64-macos --host 127.0.0.1 --port 9222 info(websocket): starting blocking worker to listen on 127.0.0.1:9222 info(server): accepting new conn... info(server): client connected info(browser): GET https://wikipedia.com/ https://wikipedia.com/ 200 info(browser): fetch https://wikipedia.com/portal/wikipedia.org/assets/js/index-24c3e2ca18.js https://wikipedia.com/portal/wikipedia.org/assets/js/index-2...: http.Status.ok info(browser): eval script portal/wikipedia.org/assets/js/index-24c3e2ca18.js: ReferenceError: location is not defined info(browser): fetch https://wikipedia.com/portal/wikipedia.org/assets/js/gt-ie9-ce3fe8e88d.js https://wikipedia.com/portal/wikipedia.org/assets/js/gt-ie9-...: http.Status.ok error(events): event handler error: error.JSExecCallback info(events): event handler error try catch: TypeError: Cannot read properties of undefined (reading 'length') info(server): close cmd, closing conn... info(server): accepting new conn... thread 5274880 panic: attempt to use null value zsh: abort ./lightpanda-aarch64-macos --host 127.0.0.1 --port 9222 ```

lbotos 2y ago
Not OP -- do you have some kind of proxy or firewall? Looks like you couldn't download https://wikipedia.com/portal/wikipedia.org/assets/js/gt-ie9-ce3fe8e88d.js https://wikipedia.com/portal/wikipedia.org/assets/js/gt-ie9-... for some reason. In my contributions to joplin s3 backend "Cannot read properties of undefined (reading 'length')" was usually when you were trying to access an object that wasn't instantiated. (Can't figure out length of <undefined>) So for some reason it seems you can't execute JS?
zelcon 2y ago
That's Zig for you. A ``modern'' systems programming language with no borrow checker or even RAII.
- hansvm 2y ago
  Those statements are mostly true and also worth talking about, but they're not pertinent to that error (remotely provided JS not behaving correctly), or the eventual crash (which you'd cause exactly the same way for the same reason in Rust with a .unwrap() call).

dang 2y ago

(This was on the frontpage as https://news.ycombinator.com/item?id=42812859 https://news.ycombinator.com/item?id=42812859 but someone pointed out to me that it had been a Show HN a few weeks ago: https://news.ycombinator.com/item?id=42430629 https://news.ycombinator.com/item?id=42430629, so I've made a fresh copy of that submission and moved the comments hither. I hope that's ok with everyone!)

zlagen 2y ago

what do you think would be the use cases for this project? being lightweight is awesome but usually you need a real browser for most use cases. Testing sites and scraping for example. It may work for some scraping use cases but I think that if the site uses any kind of bot blocking this is not going to cut it.

fbouvier 2y ago
There are a lot of uses cases: - LLM training (RAG, fine tuning) - AI agents - scraping - SERP - testing - any kind of web automation basically Bot protection of course might be a problem but it depends also on the volume of requests, IP, and other parameters. AI agents will do more and more actions on behalf of humans in the future and I believe the bot protection mechanism will evolve to include them as legit.
- zlagen 2y ago
  thanks, it doesn't seem like it's the direction it's going at the moment. If you look at the robots.txt of many websites, they are actually banning AI bots from crawling the site. To me it seems more likely that each site will have its own AI agent to perform operations but controlled by the site.

deleted 2y ago

[deleted]

the__alchemist 2y ago

I have a meta question from browsing the repo: Why do C, C++, and Zig code bases, by convention, include a license at the top of every module" IMO it makes more sense to insetead include of an overview of the module's purpose, and how it fits in with the rest of the program, and one license at the top-level, as the project already has.

AndyKelley 2y ago
100% of my projects, including the Zig compiler itself, have only the license file at the root of the project tree, except of course for files that were copy pasted from other projects.

zelcon 2y ago

Why didn't you just fork Chromium and strip out the renderer? This is guaranteed to bitrot when the web standards change unless you keep up with it forever and have perpetual funding. Yes, modifying Chromium is hard, but this seems harder.

tetris11 2y ago
Why do anything: because it shows what's possible, and makes the next effort that much more easier. I call this process of frontier effort and discovery: "science"
- zelcon 2y ago
  Redoing what others have already done is not what I think of when I hear "frontier effort"
cxr 2y ago
> modifying Chromium is hard, but this seems harder Prove it.
fbouvier 2y ago
It was my first idea. Forking Chromium has obvious advantages (compatibility). But it's not architectured for that. The renderer is everywhere. I'm not saying it's impossible, just that it did look more difficult to me than starting over. And starting from scratch has other benefits. We own the codebase and thus it's easier for us to add new features like LLM integrations. Plus reducing binary size and startup time, mandatory for embedding it (as a WASM module or as C lib).
- oever 2y ago
  The Chromium/Webkit renderer used to have multiple rendering backends. You might use or add a no-op backend.

stuckkeys 2y ago

How does it do against captchas?

psanchez 2y ago

I think this is a really cool project. Scrapping aside, I would definitely use this with playwright for end2end tests if it had 100% compatibility with chrome and ran with a fraction of the time/memory. At my company we have a small project where we are running the equivalent of 6.5 hours of end2end tests daily using playwright. Running the tests in parallel takes around half an hour. Your project is still in very early stages, but assuming 10x speed, that would mean we could pass all our tests in roughtly 3 min (best case scenario). That being said, I would make use of your browser, but would likely not make use of your business offering (our tests require internal VPN, have some custom solution for reporting, would be a lot of work to change for little savings; we run all tests currently in spot/preemptible instances which are already 80% cheaper). Business-wise I found very little info on your website. "4x the efficiency at half the cost" is a good catch phrase, but compared to what? I mean, you can have servers in Hetzner or in AWS and one is already a fraction of the cost of the other. How convenient is to launch things on your remote platform vs launch them locally or setting it up? does it provide any advantages in the case of web scrapping compared to other solutions? how parallelizable is it? Do you have any paying customers already? Supercool tech project. Best of luck!

fbouvier 2y ago
Thank you! Happy if you use it for your e2e tests in your servers, it's an open-source project! Of course it's quite easy to spin a local instance of a headless browser for occasional use. But having a production platform is another story (monitoring, maintenance, security and isolation, scalability), so there are business use cases for a managed version.

randomMatrix101 2y ago

Very cool project, congrats guys!

optixyt 2y ago

The second social media botters find this.

evanjrowley 2y ago

I'm interested to see if this could be made to work as a drop-in replacement for the headless Chromium that Hoarder uses to archive web content. I don't have a problem with the current Hoarder solution, but it would be nice to use something that requires less RAM.

deleted 2y ago

[deleted]