6 ms·

Show HN: Crawl a modern website to a zip, serve the website from the zip

223 points by unlog 2y ago

unlog 2y ago
I'm a big fan of modern JavaScript frameworks, but I don't fancy SSR, so have been experimenting with crawling myself for uploading to hosts without having to do SSR. This is the result
- mikeqq2024 2y ago
  for a long crawling task, if exited/broken for any reason, does it save and restore at the next run?
  - eXpl0it3r 2y ago
    The README says: > Can resume if process exit, save checkpoint every 250 urls
    - mikeqq2024 2y ago
      nice, better make it as a commandline option with default value. 250 is too many for large files and slow connection.
ryanwaldorf 2y ago
What's the benefit of this approach?
- unlog 2y ago
  That the page HTML is indexable by search engines without having to render in the server. Such unzipping to a directory served by nginx. You may also use it for archiving purposes, or for having backups.
- brudgers 2y ago
  One possible advantage I see is it creates a 1:1 correspondence between a website and a file. If what I care about is the website (and that's usually going to be the case), then there's a single familiar box containing all the messy details. I don't have to see all the files I want to ignore. That might not be a benefit for you and not having used it, it is only a theoretical benefit in an unlikely future for me. But just from the title of the post, I had a very clear piccture of the mechanism and it was not obvious why I would want to start with a different mechanism (barring ordinary issues with open source projects). But that's me and your mileage may vary.
earleybird 2y ago
Understood that this is early times, are you considering a licence to release it under?
- unlog 2y ago
  Sure, I forgot about that detail, what license do you suggest?
  - yodon 2y ago
    MIT and BSD seem to be by far the most common these days (I generally do MIT personally)
    - unlog 2y ago
      added
  - meiraleal 2y ago
    AGPL
- yodon 2y ago
  +1 can you add a license file with an MIT or BSD or whatever your preference is? (It's very cool. I'd love to help with this project, I'm guessing others would as well)
jayemar 2y ago
Is the output similar to a web archive file (warc)?
- unlog 2y ago
  That's something I haven't explored, sounds interesting. Right now, the zip file contains a mirror of the files found on the website when loaded in a browser. I've ended with a zip file by luck, as mirroring to the file system gives predictable problems with file/folder names.
  - toomuchtodo 2y ago
    https://news.ycombinator.com/item?id=40628958 https://news.ycombinator.com/item?id=40628958 https://github.com/internetarchive/warctools https://github.com/internetarchive/warctools
jll29 2y ago
Microsoft Interne Explorer (no, I'm not using it personally) had a file format called *.mht that could save a HTML page together with all the files referenced from it like inline images. I believe you could not store more than one page in one *.mht file, though, so your work could be seen as an extension. Although UNIX philosophy posits that it's good to have many small files, I like your idea for its contribution to reduceing clutter (imagine running 'tree' in both scenarios) and also avoiding running out of inodes in some file systems (maybe less of a problem nowadays in general, not sure as I haven't generated millions of tiny files recently).
- jdougan 2y ago
  .mht us alive and well. It is a MIME wrapper on the files and is generated by Chrome, Opera, and Edge's save option "Webpage as single file" and defaults to an extension of .mhtml. When I last looked Firefox didn't support it natively but it was a requested feature.
  - rrr_oh_man 2y ago
    > When I last looked Firefox didn't support it natively but it was a requested feature. That sounds familiar, unfortunately
    - jdougan 2y ago
      There are firefox plug-ins that claim to support saving as mhtml, I have no experience with them.
  - felipefar 2y ago
    Unfortunately it's not supported by Safari either.
  - venusenvy47 2y ago
    I use SingleFile on Firefox quite often for this purpose. https://addons.mozilla.org/en-US/firefox/addon/single-file/ https://addons.mozilla.org/en-US/firefox/addon/single-file/
kitd 2y ago
Nice work! Obligatory mention for RedBean, the server that you can package along with all assets (incl db, scripting and TLS support) into a single multi-platform binary. https://redbean.dev/ https://redbean.dev/
- ilrwbwrkhv 2y ago
  Wow this is so cool. I like these types of single binary things. Even Pocketbase is like that where you can just compile your whole app into one Go binary and just rsync that over to your server and run it.
  - djbusby 2y ago
    Go makes a binary for each target. RedBean uses magic from @jart to run the same binary on multiple platforms.
tamimio 2y ago
How is it different from HTTrack? And what about the media extension, which one is supported and which one isn’t? Sometimes when I download some sites with HTTrack, some files just get ignored because by default it looks only for default types, and you have to manually add them there.
- unlog 2y ago
  Big fan of HTTrack! reminds me of the old days and makes me sad of the current state of the web. I am not sure if HTTTrack progressed from fetching resources, long time since I used it for last time, but what my project does, is spin a real web-browser(chrome in headless mode which means it's hidden) and then it lets the JavaScript on that website execute, which means it will display/generate some fancy HTML that you can then save it as is into an index.html. It saves all kind of files, it doesn't care the extension or mime types of files, it tries to save them all.
  - tamimio 2y ago
    > It saves all kind of files, it doesn't care the extension or mime types of files, it tries to save them all. That’s awesome to know, I will give it a try. One website I remember I tried to download and has all sorts of animations with .riv extension and it didn’t work well with HTTrack, will try it with this soon, thanks for sharing it!
    - unlog 2y ago
      let me know how that goes I am interested!
renegat0x0 2y ago
My 5 cents: - status codes 200-299 are all OK - status codes 300-399 are redirects, and also can be OK eventually - 403 in my experience occurs quite often, where it is not an error, but suggestion that your user agent is not OK - robots.txt should be scanned to check if any resource is prohibited, or if there are speed requirements. It is always better to be _nice_. I plan to add something like that and also missing it in my project - It would be interesting to generate hash from app, and update only if hash is different?
- unlog 2y ago
  Status codes, I am displaying the list because mostly on a JavaScript driven application you don't want other codes than 200 (besides media). I thought about robots.txt but as this is a software that you are supposed to run against your own website I didn't consider it worthy. You have a point on speed requirements and prohibited resources (but is not like skipping over them will add any security). I haven't put much time/effort into an update step. Currently, it resumes if the process exited via checkpoints(it saves current state every 250 URLs, if any is missing then it can continue, else it will be done) Thanks, btw what's your project!? Share!
  - renegat0x0 2y ago
    I agree with your points. You might be interested in reddit webscraping thread https://www.reddit.com/r/webscraping/ https://www.reddit.com/r/webscraping/ My passion project is https://github.com/rumca-js/Django-link-archive https://github.com/rumca-js/Django-link-archive Currently I use only one thread for scraping, I do not require more. It gets the job done. Also I know too little to play more with python "celery" threads. My project can be used for various things. Depends on needs. Recently I am playing with using it as a 'search engine'. I am scraping the Internet to find cool stuff. Results are in https://github.com/rumca-js/Internet-Places-Database https://github.com/rumca-js/Internet-Places-Database. No all domains are interesting though.
  - PenguinCoder 2y ago
    > Status codes, I am displaying the list because mostly on a JavaScript driven application you don't want other codes than 200 (besides media). What? Why? Regardless of the programming language used to generate content, the standard, well known HTTP status codes should be returned as expected . If your JS served site, gives me a 200 code when it should be a 404, you're wrong.
meiraleal 2y ago
Seems like a very useful tool to impersonate websites. Useful to scammers. Why would someone crawl their own website?
- kej 2y ago
  Scammers don't need this to copy an existing website, and I could see plenty of legitimate uses. Maybe you're redoing the website but want to keep the previous site around somewhere, or you want an easy way to archive a site for future reference. Maybe you're tired of paying for some hosted CMS but you want to keep the content.
  - meiraleal 2y ago
    All the scenarios you described can be achieved by having access to the source code, assuming you own it.
    - kej 2y ago
      Lots of things are possible with access to source code that are still easier when someone writes a tool for that scenario.
      - meiraleal 2y ago
        Crawling a build you already have isn't one of them
        p4bl0 2y ago
        The website in question may be a dynamic website (e.g., WordPress, MediaWiki, or whatever other CMS or custom web app) and you either want a snapshot of it for backup, or you run it locally and want un static copy to host it elsewhere that only support static files.
- unlog 2y ago
  > Why would someone crawl their own website? My main use case is that the docs site https://pota.quack.uy/ https://pota.quack.uy/ , Google cannot index it properly. On here https://www.google.com/search?q=site%3Apota.quack.uy https://www.google.com/search?q=site%3Apota.quack.uy you will see some tiles/descriptions won't match what the content of the page is about. As the full site is rendered client side, via JavaScript, I can just crawl myself and save the html output to actual files. Then, I can serve that content with nginx or any other web server without having to do the expensive thing of SSR via nodejs. Not to mention, that being able to do SSR with modern JavaScript frameworks is not trivial, and requires engineering time.
ivolimmen 2y ago
So a modern chm (Microsoft Compiled HTML help file)
nox101 2y ago
I'm curious about this vs a .har file In Chrome Devtools, network tab, last icon that looks like an arrow pointing into a dish (Export har file) I guess a .har file as ton more data though I used it to extract data from sites that either intensionally or unintentionally make it hard to get data. For example, signing up for an apartment the apartment management site used pdf.js and provided no way to save the PDF. So saved the .har file and extracted the PDF.
- jraph 2y ago
  IIUC HAR files contain an awful lot of data that you would not want to end up being stored in a web page archive: - irrelevant http headers (including cache control) taking up too much space - auth data / cookies, credentials, personal infos that you don't want saved for privacy and security reasons especially if you want to share your archive. HAR is also not very efficient for this use case: it's a list of request represented in json. A folder representation is far better, for storage efficiency as well as for reading the archive (you basically need to implement some custom logic that rcan read the page replaying the requests).
ProtoAES256 2y ago
Wow! I never knew things like this existed! I always used wget (full below) but nowadays seemingly all sites are behind cloudflare so I need to pass a cookie too. Glad to see easier methods! wget \ --header "Cookie: <cf or other>" --user-agent="<UA>" --recursive \ --level 5 \ --no-clobber \ --page-requisites \ --adjust-extension \ --span-hosts \ --convert-links \ --domains <example.com> \ --no-parent \ <example.com\sub>
szhabolcs 2y ago
How does it work with single page apps? If the data is loaded from the server, does it save the page contents as full, or just the source of the page?
- unlog 2y ago
  It saves the generated/rendered html, but I have just added a `spa` mode, that will save the original HTML without modifications. This makes most simple web app work. I have also updated the local server for fetching from origin missing resources. For example, a webapp may load some JS modules only when you click buttons or links, when that happens and the requested file is not on the zip, it will fetch it from origin and update the zip. So mostly you can back up an SPA by crawling it first and then using it for a bit for fetching the missing resources/modules.
  - zb3 2y ago
    Note I've recently released a tool that can find all chunks of a SPA that works for popular configurations (like webpack or ES modules): https://github.com/zb3/getfrontend https://github.com/zb3/getfrontend
    - unlog 2y ago
      Thanks for sharing!
CGamesPlay 2y ago
I like the approach here! Saving to a simple zip file is elegant. I worked on a similar idea years ago [0], but made the mistake of building it as a frontend. In retrospect, I would make this crawl using a headless browser and serve it via a web application, like you're doing. I would love to see better support for SPAs, where we can't just start from a sitemap. If you're interested in, you can check out some of the code from my old app for inspiration on how to crawl pages (it's Electron, so it will share a lot of interfaces with Puppeteer) [1]. [0] https://github.com/CGamesPlay/chronicler/tree/master https://github.com/CGamesPlay/chronicler/tree/master [1] https://github.com/CGamesPlay/chronicler/blob/master/src/main/ScrapeRunner.js#L123 https://github.com/CGamesPlay/chronicler/blob/master/src/mai...
- unlog 2y ago
  It tries to fetch a sitemap for in case there's some missing link. But it starts from the root and crawls internal links. There's a new mode added this morning for spa with the option `--spa` that will write the original HTML instead of the generated/rendered one. That way some apps _will_ work better.
billpg 2y ago
https://example.com/droste.zip/droste.zip/droste.zip/droste.zip/droste.zip/droste.zip/droste.zip/droste.zip/ https://example.com/droste.zip/droste.zip/droste.zip/droste.......
sedawk 2y ago
I used to use MAFF (Mozilla Archive Format)[1] a lot back in the day. I was very upset when they ended the support[2]. I never dug deeper whether I can unzip and decode the packing, but saving as simple ZIP does somewhat guarantee future-proofing. [1] https://en.wikipedia.org/wiki/Mozilla_Archive_Format https://en.wikipedia.org/wiki/Mozilla_Archive_Format [2] https://support.mozilla.org/en-US/questions/1180271 https://support.mozilla.org/en-US/questions/1180271
Per_Bothner 2y ago
The libwebsockets server (https://libwebsockets.org https://libwebsockets.org) supports serving directly from zip archives. Furthermore, if a URL is mapped to a compressed archive member, and assuming the browser can accept gzip-compressed files (as most can), then the compressed data is copied from archive over http to the browser, without de-compressing or conversion by the server. The server does a little bit of header fiddling but otherwise sends the raw bytes to the browser, which automatically decompresses it.

223 points by unlog 2y ago

I'm a big fan of modern JavaScript frameworks, but I don't fancy SSR, so have been experimenting with crawling myself for uploading to hosts without having to do SSR. This is the result

mikeqq2024 2y ago
for a long crawling task, if exited/broken for any reason, does it save and restore at the next run?
- eXpl0it3r 2y ago
  The README says: > Can resume if process exit, save checkpoint every 250 urls
  - mikeqq2024 2y ago
    nice, better make it as a commandline option with default value. 250 is too many for large files and slow connection.

ryanwaldorf 2y ago

What's the benefit of this approach?

unlog 2y ago
That the page HTML is indexable by search engines without having to render in the server. Such unzipping to a directory served by nginx. You may also use it for archiving purposes, or for having backups.
brudgers 2y ago
One possible advantage I see is it creates a 1:1 correspondence between a website and a file. If what I care about is the website (and that's usually going to be the case), then there's a single familiar box containing all the messy details. I don't have to see all the files I want to ignore. That might not be a benefit for you and not having used it, it is only a theoretical benefit in an unlikely future for me. But just from the title of the post, I had a very clear piccture of the mechanism and it was not obvious why I would want to start with a different mechanism (barring ordinary issues with open source projects). But that's me and your mileage may vary.

earleybird 2y ago

Understood that this is early times, are you considering a licence to release it under?

unlog 2y ago
Sure, I forgot about that detail, what license do you suggest?
- yodon 2y ago
  MIT and BSD seem to be by far the most common these days (I generally do MIT personally)
  - unlog 2y ago
    added
- meiraleal 2y ago
  AGPL
yodon 2y ago
+1 can you add a license file with an MIT or BSD or whatever your preference is? (It's very cool. I'd love to help with this project, I'm guessing others would as well)

jayemar 2y ago

Is the output similar to a web archive file (warc)?

unlog 2y ago
That's something I haven't explored, sounds interesting. Right now, the zip file contains a mirror of the files found on the website when loaded in a browser. I've ended with a zip file by luck, as mirroring to the file system gives predictable problems with file/folder names.
- toomuchtodo 2y ago
  https://news.ycombinator.com/item?id=40628958 https://news.ycombinator.com/item?id=40628958 https://github.com/internetarchive/warctools https://github.com/internetarchive/warctools

jll29 2y ago

Microsoft Interne Explorer (no, I'm not using it personally) had a file format called *.mht that could save a HTML page together with all the files referenced from it like inline images. I believe you could not store more than one page in one *.mht file, though, so your work could be seen as an extension. Although UNIX philosophy posits that it's good to have many small files, I like your idea for its contribution to reduceing clutter (imagine running 'tree' in both scenarios) and also avoiding running out of inodes in some file systems (maybe less of a problem nowadays in general, not sure as I haven't generated millions of tiny files recently).

jdougan 2y ago
.mht us alive and well. It is a MIME wrapper on the files and is generated by Chrome, Opera, and Edge's save option "Webpage as single file" and defaults to an extension of .mhtml. When I last looked Firefox didn't support it natively but it was a requested feature.
- rrr_oh_man 2y ago
  > When I last looked Firefox didn't support it natively but it was a requested feature. That sounds familiar, unfortunately
  - jdougan 2y ago
    There are firefox plug-ins that claim to support saving as mhtml, I have no experience with them.
- felipefar 2y ago
  Unfortunately it's not supported by Safari either.
- venusenvy47 2y ago
  I use SingleFile on Firefox quite often for this purpose. https://addons.mozilla.org/en-US/firefox/addon/single-file/ https://addons.mozilla.org/en-US/firefox/addon/single-file/

kitd 2y ago

Nice work! Obligatory mention for RedBean, the server that you can package along with all assets (incl db, scripting and TLS support) into a single multi-platform binary. https://redbean.dev/ https://redbean.dev/

ilrwbwrkhv 2y ago
Wow this is so cool. I like these types of single binary things. Even Pocketbase is like that where you can just compile your whole app into one Go binary and just rsync that over to your server and run it.
- djbusby 2y ago
  Go makes a binary for each target. RedBean uses magic from @jart to run the same binary on multiple platforms.

tamimio 2y ago

How is it different from HTTrack? And what about the media extension, which one is supported and which one isn’t? Sometimes when I download some sites with HTTrack, some files just get ignored because by default it looks only for default types, and you have to manually add them there.

unlog 2y ago
Big fan of HTTrack! reminds me of the old days and makes me sad of the current state of the web. I am not sure if HTTTrack progressed from fetching resources, long time since I used it for last time, but what my project does, is spin a real web-browser(chrome in headless mode which means it's hidden) and then it lets the JavaScript on that website execute, which means it will display/generate some fancy HTML that you can then save it as is into an index.html. It saves all kind of files, it doesn't care the extension or mime types of files, it tries to save them all.
- tamimio 2y ago
  > It saves all kind of files, it doesn't care the extension or mime types of files, it tries to save them all. That’s awesome to know, I will give it a try. One website I remember I tried to download and has all sorts of animations with .riv extension and it didn’t work well with HTTrack, will try it with this soon, thanks for sharing it!
  - unlog 2y ago
    let me know how that goes I am interested!

renegat0x0 2y ago

My 5 cents: - status codes 200-299 are all OK - status codes 300-399 are redirects, and also can be OK eventually - 403 in my experience occurs quite often, where it is not an error, but suggestion that your user agent is not OK - robots.txt should be scanned to check if any resource is prohibited, or if there are speed requirements. It is always better to be _nice_. I plan to add something like that and also missing it in my project - It would be interesting to generate hash from app, and update only if hash is different?

unlog 2y ago
Status codes, I am displaying the list because mostly on a JavaScript driven application you don't want other codes than 200 (besides media). I thought about robots.txt but as this is a software that you are supposed to run against your own website I didn't consider it worthy. You have a point on speed requirements and prohibited resources (but is not like skipping over them will add any security). I haven't put much time/effort into an update step. Currently, it resumes if the process exited via checkpoints(it saves current state every 250 URLs, if any is missing then it can continue, else it will be done) Thanks, btw what's your project!? Share!
- renegat0x0 2y ago
  I agree with your points. You might be interested in reddit webscraping thread https://www.reddit.com/r/webscraping/ https://www.reddit.com/r/webscraping/ My passion project is https://github.com/rumca-js/Django-link-archive https://github.com/rumca-js/Django-link-archive Currently I use only one thread for scraping, I do not require more. It gets the job done. Also I know too little to play more with python "celery" threads. My project can be used for various things. Depends on needs. Recently I am playing with using it as a 'search engine'. I am scraping the Internet to find cool stuff. Results are in https://github.com/rumca-js/Internet-Places-Database https://github.com/rumca-js/Internet-Places-Database. No all domains are interesting though.
- PenguinCoder 2y ago
  > Status codes, I am displaying the list because mostly on a JavaScript driven application you don't want other codes than 200 (besides media). What? Why? Regardless of the programming language used to generate content, the standard, well known HTTP status codes should be returned as expected . If your JS served site, gives me a 200 code when it should be a 404, you're wrong.

meiraleal 2y ago

Seems like a very useful tool to impersonate websites. Useful to scammers. Why would someone crawl their own website?

kej 2y ago
Scammers don't need this to copy an existing website, and I could see plenty of legitimate uses. Maybe you're redoing the website but want to keep the previous site around somewhere, or you want an easy way to archive a site for future reference. Maybe you're tired of paying for some hosted CMS but you want to keep the content.
- meiraleal 2y ago
  All the scenarios you described can be achieved by having access to the source code, assuming you own it.
  - kej 2y ago
    Lots of things are possible with access to source code that are still easier when someone writes a tool for that scenario.
    - meiraleal 2y ago
      Crawling a build you already have isn't one of them
      - p4bl0 2y ago
        The website in question may be a dynamic website (e.g., WordPress, MediaWiki, or whatever other CMS or custom web app) and you either want a snapshot of it for backup, or you run it locally and want un static copy to host it elsewhere that only support static files.
unlog 2y ago
> Why would someone crawl their own website? My main use case is that the docs site https://pota.quack.uy/ https://pota.quack.uy/ , Google cannot index it properly. On here https://www.google.com/search?q=site%3Apota.quack.uy https://www.google.com/search?q=site%3Apota.quack.uy you will see some tiles/descriptions won't match what the content of the page is about. As the full site is rendered client side, via JavaScript, I can just crawl myself and save the html output to actual files. Then, I can serve that content with nginx or any other web server without having to do the expensive thing of SSR via nodejs. Not to mention, that being able to do SSR with modern JavaScript frameworks is not trivial, and requires engineering time.

ivolimmen 2y ago

So a modern chm (Microsoft Compiled HTML help file)

nox101 2y ago

I'm curious about this vs a .har file In Chrome Devtools, network tab, last icon that looks like an arrow pointing into a dish (Export har file) I guess a .har file as ton more data though I used it to extract data from sites that either intensionally or unintentionally make it hard to get data. For example, signing up for an apartment the apartment management site used pdf.js and provided no way to save the PDF. So saved the .har file and extracted the PDF.

jraph 2y ago
IIUC HAR files contain an awful lot of data that you would not want to end up being stored in a web page archive: - irrelevant http headers (including cache control) taking up too much space - auth data / cookies, credentials, personal infos that you don't want saved for privacy and security reasons especially if you want to share your archive. HAR is also not very efficient for this use case: it's a list of request represented in json. A folder representation is far better, for storage efficiency as well as for reading the archive (you basically need to implement some custom logic that rcan read the page replaying the requests).

ProtoAES256 2y ago

Wow! I never knew things like this existed! I always used wget (full below) but nowadays seemingly all sites are behind cloudflare so I need to pass a cookie too. Glad to see easier methods! wget \ --header "Cookie: <cf or other>" --user-agent="<UA>" --recursive \ --level 5 \ --no-clobber \ --page-requisites \ --adjust-extension \ --span-hosts \ --convert-links \ --domains <example.com> \ --no-parent \ <example.com\sub>

szhabolcs 2y ago

How does it work with single page apps? If the data is loaded from the server, does it save the page contents as full, or just the source of the page?

unlog 2y ago
It saves the generated/rendered html, but I have just added a `spa` mode, that will save the original HTML without modifications. This makes most simple web app work. I have also updated the local server for fetching from origin missing resources. For example, a webapp may load some JS modules only when you click buttons or links, when that happens and the requested file is not on the zip, it will fetch it from origin and update the zip. So mostly you can back up an SPA by crawling it first and then using it for a bit for fetching the missing resources/modules.
- zb3 2y ago
  Note I've recently released a tool that can find all chunks of a SPA that works for popular configurations (like webpack or ES modules): https://github.com/zb3/getfrontend https://github.com/zb3/getfrontend
  - unlog 2y ago
    Thanks for sharing!

CGamesPlay 2y ago

I like the approach here! Saving to a simple zip file is elegant. I worked on a similar idea years ago [0], but made the mistake of building it as a frontend. In retrospect, I would make this crawl using a headless browser and serve it via a web application, like you're doing. I would love to see better support for SPAs, where we can't just start from a sitemap. If you're interested in, you can check out some of the code from my old app for inspiration on how to crawl pages (it's Electron, so it will share a lot of interfaces with Puppeteer) [1]. [0] https://github.com/CGamesPlay/chronicler/tree/master https://github.com/CGamesPlay/chronicler/tree/master [1] https://github.com/CGamesPlay/chronicler/blob/master/src/main/ScrapeRunner.js#L123 https://github.com/CGamesPlay/chronicler/blob/master/src/mai...

unlog 2y ago
It tries to fetch a sitemap for in case there's some missing link. But it starts from the root and crawls internal links. There's a new mode added this morning for spa with the option `--spa` that will write the original HTML instead of the generated/rendered one. That way some apps _will_ work better.

billpg 2y ago

https://example.com/droste.zip/droste.zip/droste.zip/droste.zip/droste.zip/droste.zip/droste.zip/droste.zip/ https://example.com/droste.zip/droste.zip/droste.zip/droste.......

sedawk 2y ago

I used to use MAFF (Mozilla Archive Format)[1] a lot back in the day. I was very upset when they ended the support[2]. I never dug deeper whether I can unzip and decode the packing, but saving as simple ZIP does somewhat guarantee future-proofing. [1] https://en.wikipedia.org/wiki/Mozilla_Archive_Format https://en.wikipedia.org/wiki/Mozilla_Archive_Format [2] https://support.mozilla.org/en-US/questions/1180271 https://support.mozilla.org/en-US/questions/1180271

Per_Bothner 2y ago

The libwebsockets server (https://libwebsockets.org https://libwebsockets.org) supports serving directly from zip archives. Furthermore, if a URL is mapped to a compressed archive member, and assuming the browser can accept gzip-compressed files (as most can), then the compressed data is copied from archive over http to the browser, without de-compressing or conversion by the server. The server does a little bit of header fiddling but otherwise sends the raw bytes to the browser, which automatically decompresses it.