12 ms·

Preventing ZIP parser confusion attacks on Python package installers

48 points by miketheman 1y ago

jspiner 1y ago
Thank you for the interesting article.
captn3m0 1y ago
Now I am curious at whether these ZIP confusion attacks are mitigated at other registries that use ZIPs? Are there any such?
- calebbrown 1y ago
  Apart from Python Wheels, the other popular ecosystems using zip files are Java jar files, and NuGet. Of these Java is the most interesting as there a few JDKs commonly in use. But I’m also interested in various security scanners that are built in other languages that can be fooled.
  - zahlman 1y ago
    Does NPM not use zip files? (Search results for `npm package format` are entirely not useful for figuring out what an NPM package actually consists of, beyond containing a `package.json` file. `pypi package format` results look wildly different; the first result I get is https://packaging.python.org/en/latest/discussions/package-formats/ https://packaging.python.org/en/latest/discussions/package-f... which is quite comprehensive about the exact information I want — disregarding for a moment the fact that I already know this stuff ;) The NPM search results, for me, start with a Geeks4Geeks tutorial on creating a package. Is there even anything analogous to the Python Packaging Authority — misunderstood and not-actually-authoritative as it is — for NPM?)
    - pornel 1y ago
      npm and Cargo use gzipped tarballs. Tar is an awful format that has multiple ways of specifying file names and file sizes, so there could be some shenanigans happening. It's also possible to make archives have different content based on case-sensitivity of the file system.
      - zahlman 1y ago
        Ah. Python source distributions are the same, so there may be additional considerations there. Though in general it doesn't seem like there's much concern in the Python ecosystem about that, considering that building them will run arbitrary code anyway....
zahlman 1y ago
> This has been done in response to the discovery that the popular installer uv has a different extraction behavior to many Python-based installers that use the ZIP parser implementation provided by the zipfile standard library module. > For maintainers of installer projects: Ensure that your ZIP implementation follows the ZIP standard and checks the Central Directory before proceeding with decompression. See the CPython zipfile module for a ZIP implementation that implements this logic. Begin checking the RECORD file against ZIP contents and erroring or warning the user that the wheel is incorrectly formatted. Good to know that I won't need to work around any issues with `zipfile` — and it would be rather absurd for any Python-based installer to use anything else to do the decompression. (Checking RECORD for consistency is straightforward, although of course it takes time.) ... but surely uv got its zip-decompression logic from a crate rather than hand-rolling it? How many other Rust projects out there might have questionable handling of zip files? > PyPI already implements ZIP and tarball compression-bomb detection as a part of upload processing. ... The implication is that `zipfile` doesn't handle this. But perhaps it can't really? Are there valid uses for zips that work that way? (Or maybe there isn't a clear rule for what counts as a "bomb", and PyPI has to choose a threshold value?)
- lexicality 1y ago
  > but surely uv got its zip-decompression logic from a crate rather than hand-rolling it? well... https://github.com/astral-sh/rs-async-zip https://github.com/astral-sh/rs-async-zip
  - zahlman 1y ago
    Interesting. (I have neither the familarity with Rust, nor the willingness to spend time on it, to decide how much of this is the fault of the original vs the fork.)
- woodruffw 1y ago
  > and it would be rather absurd for any Python-based installer to use anything else to do the decompression. You'd reasonably think, but it's difficult to assert this: a lot of people use third-party tooling (uv, but also a lot of hand-rolled stuff), and Python packages aren't always processed in a straight-line-from-the-index manner. (I think a good reference example of this is security scanners: a scanner might fetch a wheel ZIP and analyze it, and use whatever ZIP implementation it pleases.) It's also worth noting that one of the differentials here concerns the Central Directory, but the other one is more pernicious: the ZIP APPNOTE[1] isn't really clear about how implementations should key from to EOCDR back to the local file entries, and implementations have (reasonably, IMO) interpreted the language differently. Python's zipfile chooses to do it in one way that I think is justifiable, but it's a "true" differential in the sense that there's no golden answer. > (Or maybe there isn't a clear rule for what counts as a "bomb", and PyPI has to choose a threshold value?) Yes, it's this. There are legitimate uses for high-ratio archives (e.g. compressed OS images), but Python package distributions are (generally) not one of them. PyPI has its own compression ratio that's intended to be a sweet spot between "that was compressed really well" and "someone is trying to ZIP-bomb the index." [1]: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
burnt-resistor 1y ago
Related to multiple .zip formats: I've found macOS Archive Utility sometimes refuses to extract early pkzip .zips created on MS-DOS, but yet Info-ZIP handles them just fine. And, the macOS Archive Utility will complain that a proper .tar.bz2 is "corrupt" created using bzip2. In general, be liberal in input and be conservative in output. Sometimes, this means using less features or certain older formats so that all/most things work without issues.
- quietbritishjim 1y ago
  > In general, be liberal in input and be conservative in output. That is a dangerous maxim in a world with malicious players. In fact this PyPI problem is precisely because zip files are being too readily accepted, even if they have ambiguous meaning. Their fix is (very sensibly) to be less liberal with their input.
  - burnt-resistor 1y ago
    No, it's a well-regarded, fundamental engineering principle of standard and interoperable systems.
    - quietbritishjim 1y ago
      Maybe in the early days of the internet, or in closed systems, but in the open internet it's naive and actually leads to more brittle systems. Sorry to repeat myself, but case in point is the article we're discussing! If something is accepted that is not in the specification then obviously its behaviour is unspecified. That means different implementations can easily have different behaviours, which can lead to security issues exactly like this one. Even when there's no malice involved, it can often lead to de facto extensions to the specification. If several implementations accept something outside the standard (with roughly the same behaviour), and one accidentally produces it, then soon it becomes relied upon and all implementations need to handle it. Then the standard is no longer authoritative, or has to be retrospectively updated (see for example the huge part of the HTML spec that deals with otherwise invalid HTML).
    - 1y ago

48 points by miketheman 1y ago

Thank you for the interesting article.

Now I am curious at whether these ZIP confusion attacks are mitigated at other registries that use ZIPs? Are there any such?

calebbrown 1y ago
Apart from Python Wheels, the other popular ecosystems using zip files are Java jar files, and NuGet. Of these Java is the most interesting as there a few JDKs commonly in use. But I’m also interested in various security scanners that are built in other languages that can be fooled.
- zahlman 1y ago
  Does NPM not use zip files? (Search results for `npm package format` are entirely not useful for figuring out what an NPM package actually consists of, beyond containing a `package.json` file. `pypi package format` results look wildly different; the first result I get is https://packaging.python.org/en/latest/discussions/package-formats/ https://packaging.python.org/en/latest/discussions/package-f... which is quite comprehensive about the exact information I want — disregarding for a moment the fact that I already know this stuff ;) The NPM search results, for me, start with a Geeks4Geeks tutorial on creating a package. Is there even anything analogous to the Python Packaging Authority — misunderstood and not-actually-authoritative as it is — for NPM?)
  - pornel 1y ago
    npm and Cargo use gzipped tarballs. Tar is an awful format that has multiple ways of specifying file names and file sizes, so there could be some shenanigans happening. It's also possible to make archives have different content based on case-sensitivity of the file system.
    - zahlman 1y ago
      Ah. Python source distributions are the same, so there may be additional considerations there. Though in general it doesn't seem like there's much concern in the Python ecosystem about that, considering that building them will run arbitrary code anyway....

zahlman 1y ago

> This has been done in response to the discovery that the popular installer uv has a different extraction behavior to many Python-based installers that use the ZIP parser implementation provided by the zipfile standard library module. > For maintainers of installer projects: Ensure that your ZIP implementation follows the ZIP standard and checks the Central Directory before proceeding with decompression. See the CPython zipfile module for a ZIP implementation that implements this logic. Begin checking the RECORD file against ZIP contents and erroring or warning the user that the wheel is incorrectly formatted. Good to know that I won't need to work around any issues with `zipfile` — and it would be rather absurd for any Python-based installer to use anything else to do the decompression. (Checking RECORD for consistency is straightforward, although of course it takes time.) ... but surely uv got its zip-decompression logic from a crate rather than hand-rolling it? How many other Rust projects out there might have questionable handling of zip files? > PyPI already implements ZIP and tarball compression-bomb detection as a part of upload processing. ... The implication is that `zipfile` doesn't handle this. But perhaps it can't really? Are there valid uses for zips that work that way? (Or maybe there isn't a clear rule for what counts as a "bomb", and PyPI has to choose a threshold value?)

lexicality 1y ago
> but surely uv got its zip-decompression logic from a crate rather than hand-rolling it? well... https://github.com/astral-sh/rs-async-zip https://github.com/astral-sh/rs-async-zip
- zahlman 1y ago
  Interesting. (I have neither the familarity with Rust, nor the willingness to spend time on it, to decide how much of this is the fault of the original vs the fork.)
woodruffw 1y ago
> and it would be rather absurd for any Python-based installer to use anything else to do the decompression. You'd reasonably think, but it's difficult to assert this: a lot of people use third-party tooling (uv, but also a lot of hand-rolled stuff), and Python packages aren't always processed in a straight-line-from-the-index manner. (I think a good reference example of this is security scanners: a scanner might fetch a wheel ZIP and analyze it, and use whatever ZIP implementation it pleases.) It's also worth noting that one of the differentials here concerns the Central Directory, but the other one is more pernicious: the ZIP APPNOTE[1] isn't really clear about how implementations should key from to EOCDR back to the local file entries, and implementations have (reasonably, IMO) interpreted the language differently. Python's zipfile chooses to do it in one way that I think is justifiable, but it's a "true" differential in the sense that there's no golden answer. > (Or maybe there isn't a clear rule for what counts as a "bomb", and PyPI has to choose a threshold value?) Yes, it's this. There are legitimate uses for high-ratio archives (e.g. compressed OS images), but Python package distributions are (generally) not one of them. PyPI has its own compression ratio that's intended to be a sweet spot between "that was compressed really well" and "someone is trying to ZIP-bomb the index." [1]: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT

burnt-resistor 1y ago

Related to multiple .zip formats: I've found macOS Archive Utility sometimes refuses to extract early pkzip .zips created on MS-DOS, but yet Info-ZIP handles them just fine. And, the macOS Archive Utility will complain that a proper .tar.bz2 is "corrupt" created using bzip2. In general, be liberal in input and be conservative in output. Sometimes, this means using less features or certain older formats so that all/most things work without issues.

quietbritishjim 1y ago
> In general, be liberal in input and be conservative in output. That is a dangerous maxim in a world with malicious players. In fact this PyPI problem is precisely because zip files are being too readily accepted, even if they have ambiguous meaning. Their fix is (very sensibly) to be less liberal with their input.
- burnt-resistor 1y ago
  No, it's a well-regarded, fundamental engineering principle of standard and interoperable systems.
  - quietbritishjim 1y ago
    Maybe in the early days of the internet, or in closed systems, but in the open internet it's naive and actually leads to more brittle systems. Sorry to repeat myself, but case in point is the article we're discussing! If something is accepted that is not in the specification then obviously its behaviour is unspecified. That means different implementations can easily have different behaviours, which can lead to security issues exactly like this one. Even when there's no malice involved, it can often lead to de facto extensions to the specification. If several implementations accept something outside the standard (with roughly the same behaviour), and one accidentally produces it, then soon it becomes relied upon and all implementations need to handle it. Then the standard is no longer authoritative, or has to be retrospectively updated (see for example the huge part of the HTML spec that deals with otherwise invalid HTML).
  - 1y ago