9 ms·
Show HN: An API that takes a URL and returns a file with browser screenshots
- ranger_danger 2y agoNo license?
- gkamer8 2y agoOh wow, totally forgot. Just added MIT.
- _nolram 2y agoI'm working on a project that requires automated website screenshots, and I've hit the cookie banner problem. I initially tried a brute-force approach, cataloging common button classes and text to simulate clicks, but the sheer variety of implementations makes it unmanageable. So many different classes, button texts etc. I've resorted to "https://screenshotone.com https://screenshotone.com", because it takes a perfect screenshot every time, never had a single cookie banner visible on the screenshots. I would really like to know how this is handled. Maybe there is someone here that can share some knowledge.
- tantaman 2y agous ai?
- ge96 2y agothe very same
- bangaladore 2y agoThe website [1] is very strange. What does U.S. stand for? If I were to stumble on this I'd assume it was a fishing / scam website trying to impersonate the government. Bad vibes all around. [1] - https://us.ai/ https://us.ai/
- gkamer8 2y agoI'll try to improve the vibes :( I've been working at this startup for almost two years now and that page and branding etc has been changing a lot as you can imagine ...
- bangaladore 2y agoBut what is the branding? United States AI? Like the premise of the company name is bad. Real bad.
- bbor 2y agoA) Thanks for sharing your OSS with the world!! B) I'm also a little confused. Surely that domain cost(s) $$$ -- why not go with a cute "us" branding rather than "U.S."? Unless you're looking to sell in other countries where maybe U.S. expertise is a selling point, this definitely comes across like you're pretending to be part of the government. EDIT: For comparison, we.ai costs $500,000/y (!!!) EDIT2: It looks like you're positioning yourself as a defense/govt contractor, thus the branding? That's certainly cool, but IMHO, if I were you and owned that domain, I'd offer it to Palantir for $$$$$ and just go with your second choice. They're currently starting in on a whole genocide/global war thing, so they have cash to burn!
- gkamer8 2y agoHi thanks! The domain actually used to be a redirect link to U.S. Automotive Industries (a trade publication). I reached out to them and got a deal, so it was a lot for me but not, like, we.ai expensive lol. The name was always a corporate placeholder and I liked the idea of US Steel or General Electric type names. Some startups have done similar things, and many people actually like the name a ton. But I know it's controversial and so any products I made have their own names and branding that's pretty separate (see: Abbey). Over the past few months I've gone the gov contracting route and the name actually made some sense, so I've used it raw. Still, the plan is to get a DBA in the near future and switch it up. Thanks for the advice!
- 2y ago
- wildzzz 2y agoIt's one guy running his little AI startup fresh out of college. Claims to be a former national security analyst but makes no such claim on his LinkedIn.
- gkamer8 2y agoThanks for the catch on my LinkedIn, I really should have that there now. It was originally something I kept private.
- wildzzz 2y agoThanks, I'm always on the lookout for people with suspicious or over-exaggerated credentials cough-Lex Friedman-cough. Is the national security paper public? Is it something about Ufimstev?
- gkamer8 2y agoHi, it is unfortunately not public and cannot be made so to my understanding. It was frustrating to talk about in job interviews for that reason and therefore was not on the LinkedIn.
- throwaway314155 2y ago> cough-Lex Friedman-cough Oh please elaborate!
- standardly 2y agolisten to any episode and it's evident
- throwaway314155 2y agoYeah I mean I gathered that, still would be interesting to know what specifically he lied about.
- xnx 2y agoFor anyone who might not be aware, Chrome also has the ability to save screenshots from the command line using: chrome --headless --screenshot="path/to/save/screenshot.png" --disable-gpu --window-size=1280,720 "https://www.example.com https://www.example.com"
- martinbaun 2y agoOh man, I needed this so many times didn't even think of doing it like this. I tried using Selenium and all different external services. Thank you! Works in chromium as well.
- azhenley 2y agoVery nice, I didn't know this. I used pyppeteer and selenium for this previously which seemed excessive.
- Onavo 2y agoWhat features won't work without GPU?
- xnx 2y ago[flagged]
- dingnuts 2y agooh good an AI summary with none of the facts checked, literally more useless than the old lmgtfy and somehow more rude "here's some output that looks relevant to your question but I couldn't even be arsed to look any of it up, or copy paste it, or confirm its validity"
- kylecazar 2y agoThis flag isn't valid anymore in the new chrome headless. Disable GPU doesn't exist unless your on the old version (and then, it was meant as a workaround for Windows users only). I've used this via selenium not too long ago
- 2y ago
- aspeckt-112 2y agoI’m looking forward to giving this a go. Great idea!
- manmal 2y agoBeing a bit frustrated with Linkwarden’s resource usage, I’ve thought about making my own self hosted bookmarking service. This could be a low effort way of loading screenshots for these links, very cool! It‘ll be interesting how many concurrent requests this can process.
- synthomat 2y agoThat's nice and everything but what to do about the EU cookie banners? Does hosting outside of the EU help?
- gkamer8 2y agoYeah the EU cookie banners are annoying, I'm hoping to do some automation to click out of them before taking the screenshots
- cjr 2y agoThere are browser extensions you could run like consent-o-matic to try to click and hide the cookies from your screenshots: https://chromewebstore.google.com/detail/consent-o-matic/mdjildafknihdffpkfmmpnpoiajfjnjd https://chromewebstore.google.com/detail/consent-o-matic/mdj... Otherwise using a combination of well-known class names, ‘accept’ strings, and heuristics such as z-index, position: fixed/sticky etc can also narrow down the number of likely elements that could be modals/banners. You could also ask a vision model whether a screenshot has a cookie banner, and ask for co-ordinates to remove it, although this could get expensive at scale!
- gkamer8 2y agoThanks, that's a great idea! I was originally going to go the vision model route because I'd also like people to be able to send instructions to sign in with some credentials (like when visiting the nytimes or something).
- artur_makly 2y agoyeah that's what we basically did here at https://VisualSitemaps.com https://VisualSitemaps.com, but it can also be quickly become over-the-top, and you may end up removing important content. That's why in the end we added a second option to just manually enter CSS classes.
- cess11 2y agoNo. Tell the services you're using to stop with the malicious compliance.
- quink 2y ago> SCREENSHOT_JPEG_QUALITY Not two words that should be near each other, and JPEG is the only option. Almost like it’s designed to nerd-snipe someone into a PR to change the format based on Accept headers.
- gkamer8 2y ago> Almost like it's designed to nerd-snipe someone into a PR to change the format based on Accept headers pls
- mpetrovich 2y agoReminds me of this open source library I wrote to do the same thing: https://github.com/nextbigsoundinc/imagely https://github.com/nextbigsoundinc/imagely It uses puppeteer and chrome headless behind the scenes.
- joshstrange 2y agoThis is cool but at this point MCP is the clear choice for exposing tools to LLMs, I'm sure someone will write a wrapper around this to provide the same functionality as an MCP-SSE server. I want to try this out though and see how I like it compared to the MCP Puppeteer I'm using now (which does a great job of visiting pages, taking screenshots, interacting with the page, etc).
- jot 2y agoIf you’re worried about the security risks, edge cases, maintenance pain and scaling challenges of self hosting there are various solid hosted alternatives: - https://browserless.io https://browserless.io - low level browser control - https://scrapingbee.com https://scrapingbee.com - scraping specialists - https://urlbox.com https://urlbox.com - screenshot specialists* They’re all profitable and have been around for years so you can depend on the businesses and the tech. * Disclosure: I work on this one and was a customer before I joined the team.
- edm0nd 2y agohttps://www.scraperapi.com/ https://www.scraperapi.com/ is good too. Been using them to scrape via their API on websites that have a lot of captchas or anti scraping tech like DataDome.
- rustdeveloper 2y agoHappy to suggest another web scraping API alternative I rely on: https://scrapingfish.com https://scrapingfish.com
- xeornet 2y agoWhat’s the chance you’re affiliated? Almost every one of your comments links to it. And curiously similar interest in Rust from the official HN page and yours. No need to be sneaky.
- bbor 2y agoDo these services respect norobot manifests? Isn't this all kinda... illegal...? Or at least non-consensual?
- basilgohar 2y agorobots.txt isn't legally binding. I am interested to know if and how services even interact with it. It's more like a clue on when the interesting content for scrapers is on your site. This is how I imagine it goes: "Hey, don't scrape the data here." "You know what? I'm scrape it even harder!"
- morbusfonticuli 2y agoSimilar project: gowitness [1]. A really cool tool i recently discovered. Next to scraping and performing screenshots of websites and saving it in multiple formats (including sqlite3), it can grab and save the headers, console logs & cookies and has a super cool web GUI to access all data and compare e.g the different records. I'm planning to build my personal archive.org/waybackmachine-like web-log tool via gowitness in the not-so-distant future. [1] https://github.com/sensepost/gowitness https://github.com/sensepost/gowitness
- westurner 2y agosimonw/shot-scraper has a number of cli args, a GitHub actions repo template, and docs: https://shot-scraper.datasette.io/en/stable/ https://shot-scraper.datasette.io/en/stable/ From https://news.ycombinator.com/item?id=30681242 https://news.ycombinator.com/item?id=30681242 : > Awesome Visual Regression Testing > lists quite a few tools and online services: https://github.com/mojoaxel/awesome-regression-testing https://github.com/mojoaxel/awesome-regression-testing > "visual-regression": https://github.com/topics/visual-regression https://github.com/topics/visual-regression
- kevinsundar 2y agoI'm looking for something similar that can also extract the diff of content on the page over time, in addition to screenshots. Any suggestions? I have a homegrown solution using an LLM and scrapegraphai for https://getchangelog.com https://getchangelog.com but would rather offload that to a service that does a better job rendering websites. There's some websites that I get error pages from using playwright, but they load fine in my usual Chrome browser.
- caelinsutch 2y agoThe easiest solution to this is probably extracting / formatting the content, then running a diff on that. Otherwise you could use snapshot testing algorithms as a diffing method. We use browserbase and olostep which both have strong proxies (first one gives you a playwright instance, second one just screenshot + raw HTML).
- arnoldcjones 2y agoGood point on offloading it as for the amount of work that's required in setting up a wrapper for something like Puppeteer, Playwright etc that also works with a probably quite specific setup, I've found the best way to get a quality image consistently is to just subscribe to one of the many SASS' out there that already do this well. Some of the comments above suggest some decent screenshot-as-a-service products. Really depends on how valuable your time is over your (or your companies) money. I prefer going for the quality (and more $) solution rather than the solution that boasts cheap prices, as I tend to avoid headaches of unreliable services. Sam Vines Boots theory and all that. For image comparison I've always found using pixelmatch by Mapbox works well for PNG's https://github.com/mapbox/pixelmatch https://github.com/mapbox/pixelmatch
- mlunar 2y agoSimilar one I wrote a while ago using Pupetteer for the IoT low power display purposes. Neat trick is that it learns the refresh interval, so that it takes a snapshot just before it's requested :) https://github.com/SmilyOrg/website-image-proxy https://github.com/SmilyOrg/website-image-proxy
- rpastuszak 2y agoCool! In using sth similar on my site to generate screenshots of tweets (for privacy purposes): https://untested.sonnet.io/notes/xitterpng-privacy-friendly-embeds-and-one-way-mirrors/ https://untested.sonnet.io/notes/xitterpng-privacy-friendly-...
- jchw 2y agoOne thing to be cognizant of: if you're planning to run this sort of thing against potentially untrusted URLs, the browser might be able to make requests to internal hosts in whatever network it is on. It would be wise, on Linux, to use network namespaces, and block any local IP range in the namespace, or use a network namespace to limit the browser to a wireguard VPN tunnel to some other network.
- remram 2y agoVery important note! This is called Server-Side Request Forgery (SSRF).
- jot 2y agoToo many developers learn this the hard way. It’s one of the top reasons larger organisations prefer to use hosted services rather than doing it themselves.
- leptons 2y agoThis is true for practically every web browser anyone uses on any site that they don't personally control.
- jchw 2y agoThis is true, although I think in a home environment, there aren't as many interesting things to hit, and you're limited by Same Origin Policy, as well as certain mitigations that web browsers deploy against attacks like DNS Rebinding. However, if you're running this on a server, there's a much greater likelihood that interesting services are under the firewall, e.g. maybe the Kubernetes API server. Code execution could potentially be a form post away.
- anonzzzies 2y agoIs there a self hosted version that does this properly?
- deleted 2y ago[deleted]
- hedora 2y agoIt'd be nice if it produced a list of bounding boxes + URL's you'd get if you clicked on the bounding box. Then it'd be close to my dream of a serverless web browser service, where the client just renders a clickmap .png or .webp, and the requests go to a farm of "one request per page load" ephemeral web browser instances. The web browsers could cache the images + clickmaps they return in an S3 bucket. Assuming the farm of browsers had a large number of users, this would completely defeat fingerprinting + cookies. It'd also provide an archive (as in durable, not as in high quality) of the browsed static content.
- robertclaus 2y agoWe developed a service like this internally at a previous company. It was nice to have a generic "preview" generating service.
- cchance 2y agoThe fact the github doesn't have a screenshot seems... like a sad omission
- codenote 2y agoI thought it was a scale of code that could have been included in Abbe. https://github.com/US-Artificial-Intelligence/abbey https://github.com/US-Artificial-Intelligence/abbey Was the motivation for separating it based on security considerations, as stated in the "Security Considerations"? https://github.com/US-Artificial-Intelligence/ScrapeServ?tab=readme-ov-file#security-considerations https://github.com/US-Artificial-Intelligence/ScrapeServ?tab...
- Mani_Pathak 2y ago[dead]
- nottorp 2y agoWhy is the repo called artificial intelligence when it just runs browsers?
- s09dfhks 2y agohmm cant defeat cloudflare unfortunately, otherwise not bad