8 ms·
I use ZSTD a ton in my programming work where efficiency matters. But for sharing files with other people, ZIP is still king. Even 7z or RAR is niche. Everyone
by dikei 1y ago
I use ZSTD a ton in my programming work where efficiency matters.
But for sharing files with other people, ZIP is still king. Even 7z or RAR is niche. Everyone can open a ZIP file, and they don't really care if the file is a few MBs bigger.
- cesarb 1y ago> Everyone can open a ZIP file, and they don't really care if the file is a few MBs bigger. You can use ZSTD with ZIP files too! It's compression method 93 (see https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT which is the official ZIP file specification). Which reveals that "everyone can open a ZIP file" is a lie. Sure, everyone can open a ZIP file, as long as that file uses only a limited subset of the ZIP format features. Which is why formats which use ZIP as a base (Java JAR files, OpenDocument files, new Office files) standardize such a subset; but for general-purpose ZIP files, there's no such standard. (I have encountered such ZIP files in the wild; "unzip" can't decompress them, though p7zip worked for these particular ZIP files.)
- dikei 1y agoWell, only a lunatic would use ZIP with anything but DEFLATE/DEFLATE64
- redeeman 1y agothere are A LOT of zip files using lzma in the wild. also, how about people learn to use updated software? should newer video compression technologies not be allowed in mkv/mp4. if you cant open it, well.. then stop using 90ies winzip
- 1over137 1y ago>how about people learn to use updated software? How about software developers learn to keep software working on old OSes and old hardware?
- tiagod 1y agoWhat stops you from running updated zip/unzip on an old OS or on old hardware?
- krapht 1y agoNothing, but what stops you from using DEFLATE64? Installing new software has a real time and hassle cost, and how much time are you actually saving over the long run? It depends on your usage patterns.
- RealStickman_ 1y agoSupporting old APIs and additional legacy ways of doing things has a real cost in maintenance.
- mananaysiempre 1y agoSo does not supporting them, but the developer gets to externalize those.
- redeeman 1y agothe developer is hired by someone that gets to make that decision. Ultimately the customer does. Thats why some people spend extreme resources on legacy crap, because someone has deemed it worth it.
- redeeman 1y agowhat stops you from installing win95 and winzip?
- redeeman 1y agowhat software doesnt support OSs that are in active SECURITY support?
- 1y ago
- Am4TIfIsER0ppos 1y agomkv or mp4 with h264 and aac is good enough. mp3 is good enough. jpeg is good enough. zip with deflate is also good enough.
- redeeman 1y agoh264 is not good enough for many things
- e4m2 1y ago"Good enough" is not good enough.
- homebrewer 1y agoIn the middle of San Francisco, with Silicon Valley level incomes, very possible. In the real world I still exchange files with users on rustic ADSL, where every megabyte counts. Many areas out there, in rural Mongolia or in the middle of Africa that's just got access to the internet, are even worse in that regard.
- dahrkael 1y agoI started using winrar because winzip wouldnt fit in a floppy disk. so even in the 90s zip wasnt good enough
- landl0rd 1y agoNo. You can't get people to use updated software. You can't get a number of people to update past windows 7. This has been and will likely remain a persistent issue, and it's sure not one you're going to fix. All it will do is limit your ability to work with people. This isn't a hill on which you should die.
- redeeman 1y agoif they want to open certain files, they will update
- landl0rd 1y agoNo, they're just not going to work with you.
- TkTech 1y agoYep. Half the world's finances still spin on CSVs and FTP (no, not SFTP, FTP) If your customers request a format, that's the format you're using.
- nick238 1y agoAnd if they don't request a format (or if you ask, and the response is "what's a format"), you need to use one that's 99.99% supported.
- redeeman 1y agoim okay with that. That being said, I have not had a single issue delivering zip files with lzma, and i KNOW that I have gotten MANY from the random sources. I would also expect people to be able to decode h265 in an mp4 file. Your proposal seems, to word it bluntly, retarded. You would have mp4 frozen for h264 for ETERNITY, and then invent a new format as replacement? or you would just say "god has bestowed upon the world h264, and it shall be the LAST CODEC EVER!". get with the program. Things change, you cannot expect to be forwards compatible for ever. Sometimes people have to switch to newer versions of software.
- easton 1y ago> new Office files I know what you mean, I’m not being pedantic, but I just realized it’s been 19 years. I wonder when we’ll start calling them “Office files”.
- mauvehaus 1y ago> I wonder when we’ll start calling them “Office files”. Probably around the same time the save icon becomes something other than a 3 1/2" floppy disk.
- kevinventullo 1y agoNowadays I’ve noticed fewer applications have a save icon at all, relying instead on auto-save.
- ale42 1y agoAnd some only save to the cloud, whence a cloud icon with an arrow. (Not that I like that, but... that's what we get)
- jl6 1y agoEnglish is evolving as a hieroglyphic language. That floppy disk icon stands a good chance of becoming simply the glyph meaning "save". The UK still uses an icon of an 1840s-era bellows camera for its speed camera road signs. The origin story will be filed away neatly and only its residual meaning will be salient.
- guappa 1y agoYou can and I've done it… but you can't expect anything to be able to decompress it unless you wrote it yourself.
- justin66 1y ago> Copyright (c) 1989 - 2014, 2018, 2019, 2020, 2022 Mostly it seems nutty that, after all these years, they’re still updating the zip spec instead of moving on to a newer format.
- pornel 1y agoThe English language is awful, and we keep updating it instead of moving to a newer language. Some things are used for interoperability, and switching to a newer incompatible thing loses all of its value.
- deleted 1y ago[deleted]
- 6SixTy 1y ago.7z and .tar.* have existed for at least 20 years now, but you are unlikely to see a wild 7z file and .tar.* is isolated to the UNIX space
- danudey 1y agoTar files also have the miserable limitation of having no index; this means that to extract an individual file requires scanning through the entire archive until you find it, and then continuing to scan through the rest of the archive because a tar file can have the same file path added multiple times. That makes them useful for transferring an entire set of files that someone will want all or none of, e.g. source code, but terrible for a set of files that someone might want to access arbitrary files from.
- justin66 1y agoSure, but that's not really a reason to futilely try to spooge oddball algorithms that nobody is going to adopt into the .zip standard.
- deleted 1y ago[deleted]
- sidewndr46 1y agoSame thing with "WAV" files. There's at least 3 popular formats for the audio data out there.
- martinald 1y agoMore 'useful' one is webp. It has both a lossy and lossless compression algorithm, which have very different strengths and weaknesses. I think nearly every device supports reading both, but so many 'image optimization' libraries and packages don't - often just doing everything as lossy when it could be lossless (icons and what not).
- LegionMammal978 1y agoIt's similarly annoying how many websites take the existence of the lossy format as a license to recompress all WebP uploads, or sometimes other filetypes converted to WebP, even when it causes the filesize to increase. It's like we're returning to ye olden days of JPEG artifacts on every screenshot.
- danudey 1y agoI was thinking about this with YouTube as an example. A lot of people complain about the compression on YouTube videos making things look awful, but I bet there's a reasonable number of high-end content creators out there who would run a native(-ish, probably Electron) app on their local system to do a higher-quality encoding to YouTube's specifications before uploading. In many (most?) cases, it's possible to get better compression and higher quality if you're willing to spend the CPU cycles on it, meaning that YouTube could both reduce their encoding load and increase quality at the same time, and content creators could put out better quality videos that maintain better detail. It would certainly take longer to upload the multiple multiple versions of everything, and definitely it would take longer to encode, but it would also ease YouTube's burden and produce a better result. Ah well, a guy can dream.
- martinald 1y ago
- throw0101d 1y ago> You can use ZSTD with ZIP files too! Support for which was added in 2020: > On 15 June 2020, Zstandard was implemented in version 6.3.8 of the zip file format with codec number 93, deprecating the previous codec number of 20 as it was implemented in version 6.3.7, released on 1 June.[36][37] * https://en.wikipedia.org/wiki/Zstd#Usage https://en.wikipedia.org/wiki/Zstd#Usage So I'm not sure how widely deployed it would be.
- xxs 1y agoMost linux distributions have zip support with zstd.
- danudey 1y agoThe `zip` command on Ubuntu is 6.0, which was released in 2009 and does not support zstd. It does support bzip2 though!
- cesarb 1y ago> The `zip` command on Ubuntu is 6.0, which was released in 2009 and does not support zstd. It does support bzip2 though! You probably mean the "unzip" command, which https://infozip.sourceforge.net/UnZip.html https://infozip.sourceforge.net/UnZip.html lists as 6.0 being the latest, released on 20 April 2009. Relevant to this discussion, new in that release are support for 64-bit file sizes, bzip2 compression method, and UTF-8 filenames. The "zip" command is listed at https://infozip.sourceforge.net/Zip.html https://infozip.sourceforge.net/Zip.html as 3.0 being the latest, released on 7 July 2008. New in that release are also support for 64-bit file sizes, bzip2 compression method, and UTF-8 filenames. It would be great if both (or at least unzip) were updated to also support LZMA/XZ/ZSTD as compression methods, but given that there have been no new releases for over fifteen years, I'm not too hopeful.
- hulitu 1y ago> It would be great if both (or at least unzip) were updated to also support LZMA/XZ/ZSTD as compression methods, Why ? Xz supports xz, zstd supports zstd. Why should unzip support xz or rar or gz ?
- Akronymus 1y agoSo thats why rarely the customer cant open one of the zip files we send over.
- notepad0x90 1y agoI don't know about, had a dicey situation recently where powershell's compress-archive couldn't handle archives >4GB and had to use 7zip. it is more reliable and you can ship 7za.exe or create self-extracting archives (wish those were more of a thing outside of the windows world).
- chasil 1y agoIn the realm of POSIX.2 and UNIX relatives, the closest analog would be a "shar" archive. They are not regarded kindly. https://en.wikipedia.org/wiki/Shar_(file_format) https://en.wikipedia.org/wiki/Shar_(file_format)
- deleted 1y ago[deleted]
- landl0rd 1y agoI understand that security has to compromise for the real world, but a self-extracting archive is possibly one of the worst things one could use in terms of security.
- notepad0x90 1y agowhy? and why does it have to be a compromise? You're assuming things because things are already done insecurely. You can authenticate the self-extractor as well as the extracted content. The user gets a nice message "This is a 7zip self-extracting archive sent to you by Bob containing the files below". As an incident responder, I've seen much more of regular archives being used to social engineer users than self-extracting archives, because self-extracting is not "content executing". it is better for social engineering for users to establish trust in the payload first by having them manually open the archive. if something "weird" like self-extraction happens first, it might feel less trustworthy. Oh and by the way, things like PyInstaller or electron apps are already self-extracting and self-executing archives. So are JAR files and android APK's.
- fsckboy 1y ago
- sidewndr46 1y agoWhat are you compressing with zstd? I had to do this recently and the "xz" utility still blows it away in terms of compression ratio. In terms of memory and CPU usage, zstd wins by a large margin. But in my case I only really cared about compression ratio
- vlovich123 1y agopeople tend to care about decompression speed - xz can be quite slow decompressing super compressed files whereas zstd decompression speed is largely independent of that. People also tend to care about how much time they spend on compression for each incremental % of compression performance and zstd tends to be a Pareto frontier for that (at least for open source algorithms)
- bracketfocus 1y agoThis makes sense. A lot of end-users have internet speeds that can outpace the decompression speeds of heavily compressed files. Seems like there would be an irrational psychological aspect to it as well. Unfortunately for the hoster, they either have to eat the cost of the added bandwidth from a larger file or have people complain about slow decompression.
- vlovich123 1y agoWell the difference is quite a bit more manageable in practice since you’re talking about single digit space difference vs a 2-100x performance in decompression.
- sidewndr46 1y agoI definitely agree, I basically have unlimited time and unlimited CPU for decompressing. Available memory is huge too. The gains from xz were significant enough that I went with it.
- xxs 1y agodo you have examples where xz 'blows it away', not just zstd -3?
- jart 1y agoUse the pigz command for parallel gzip. Mark Adler also has an example floating around somewhere about how to implement basically the same thing using Z_BLOCK.
- mrWiz 1y agoMy main use case for 7z is bypassing corporate filters that block ZIPs from being sent.
- psyclobe 1y agozip is such a shit standard, hell there are parts of it that are still undocumented and sharing documents between system zip implementations across mac and windows sometimes fails.