5 ms·

Wikipedia has a good explanation why the PNG magic number is 89 50 4e 47 0d 0a 1a 0a. It has some good features, such as the end-of-file character for DOS and d

Wikipedia has a good explanation why the PNG magic number is 89 50 4e 47 0d 0a 1a 0a. It has some good features, such as the end-of-file character for DOS and detection of line ending conversions. https://en.wikipedia.org/wiki/PNG#File_header https://en.wikipedia.org/wiki/PNG#File_header

CrossVR 2y ago
At first I wasn't sure why it contained a separate Unix line feed when you would already be able to detect a Unix to DOS conversion from the DOS line ending: 0D 0A 1A 0A -> 0D 0D 0A 1A 0D 0A But of course this isn't to try and detect a Unix-to-DOS conversion, it's to detect a roundtrip DOS-to-Unix-to-DOS conversion: 0D 0A 1A 0A -> 0A 1A 0A -> 0D 0A 1A 0D 0A Certainly a very well thought-out magic number.
- layer8 2y ago
  Unix2dos is idempotent on CRLF, it doesn’t change it to CRCRLF. Therefore converted singular LFs elsewhere in the file wouldn’t be recognized by the magic-number check if it only contained CRLF. This isn’t about roundtrip conversion.
- Dwedit 2y ago
  It's also detecting when a file on DOS/Windows is opened in "ASCII mode" rather than binary mode. When opened in ASCII mode, "\r\n" is automatically converted to "\n" upon reading the data.
nayuki 2y ago
The old PNG specification also explained the rationale: http://www.libpng.org/pub/png/spec/1.2/PNG-Rationale.html#R.PNG-file-signature http://www.libpng.org/pub/png/spec/1.2/PNG-Rationale.html#R.... But the new spec doesn't explain: https://www.w3.org/TR/2003/REC-PNG-20031110/ https://www.w3.org/TR/2003/REC-PNG-20031110/
- somat 2y ago
  That is unfortunate. Not enough standards have rationale or intent sections. On the one hand I sort of understand why they don't "If it is not critical and load-bearing to the standard. Why is it in there? it is just noise that will confuse the issue." On the other hand, it can provide very important clues as to the why of the standard, not just the what. While the standards authors understood why they did things the way they did, many years later when we read it often we are left with more questions than answers.
  - 0xCE0 2y ago
    Clear rationales doesn't sell well: better to obfuscate specs with hidden design decisions and build complexity moat.
IshKebab 2y ago
I can count the number of times I've had binary file corruption due to line ending conversion on zero hands. And I'm old enough to have used FTP extensively. Seems kind of unnecessary.
- jxhdbd 2y ago
  Have you tried using git with Windows clients? There are so many random line conversions going on and the detection on what is a binary file is clearly broken. I don't understand why the default would be anything but "commit the file as is"
  - kbolino 2y ago
    Up until just a few years ago, Notepad on Windows could not handle Unix-style line endings. It probably makes sense now to adopt the as-is convention, but for a while, it made more sense to convert when checking out, and then to prevent spurious diffs, convert back when committing.
    - hnlmorg 2y ago
      Line endings between windows and unix-like systems were so painful that when I started development on my shell scripting language, I wrote a bunch of code to all Linux to handle Windows files and visa versa. Though this has nothing to do with FTP. I’d already abandoned that protocol by then.
  - layer8 2y ago
    > I don't understand why the default would be anything but "commit the file as is" Because it’s not uncommon for dev tools on Windows to generate DOS line endings when modifying files (for example when adding an element to an XML configuration file, all line endings of the file may be converted when it is rewritten out from its parsed form), and if those where committed as-is, you’d get a lot of gratuitous changes in the commit and also complaints from the Unix users. For Git, the important thing is to have a .gitattributes file in the repository with “* text=auto” in it (plus more specific settings as desired). The text/binary auto-detection works mostly fine.

CrossVR 2y ago

At first I wasn't sure why it contained a separate Unix line feed when you would already be able to detect a Unix to DOS conversion from the DOS line ending: 0D 0A 1A 0A -> 0D 0D 0A 1A 0D 0A But of course this isn't to try and detect a Unix-to-DOS conversion, it's to detect a roundtrip DOS-to-Unix-to-DOS conversion: 0D 0A 1A 0A -> 0A 1A 0A -> 0D 0A 1A 0D 0A Certainly a very well thought-out magic number.

layer8 2y ago
Unix2dos is idempotent on CRLF, it doesn’t change it to CRCRLF. Therefore converted singular LFs elsewhere in the file wouldn’t be recognized by the magic-number check if it only contained CRLF. This isn’t about roundtrip conversion.
Dwedit 2y ago
It's also detecting when a file on DOS/Windows is opened in "ASCII mode" rather than binary mode. When opened in ASCII mode, "\r\n" is automatically converted to "\n" upon reading the data.

nayuki 2y ago

The old PNG specification also explained the rationale: http://www.libpng.org/pub/png/spec/1.2/PNG-Rationale.html#R.PNG-file-signature http://www.libpng.org/pub/png/spec/1.2/PNG-Rationale.html#R.... But the new spec doesn't explain: https://www.w3.org/TR/2003/REC-PNG-20031110/ https://www.w3.org/TR/2003/REC-PNG-20031110/

somat 2y ago
That is unfortunate. Not enough standards have rationale or intent sections. On the one hand I sort of understand why they don't "If it is not critical and load-bearing to the standard. Why is it in there? it is just noise that will confuse the issue." On the other hand, it can provide very important clues as to the why of the standard, not just the what. While the standards authors understood why they did things the way they did, many years later when we read it often we are left with more questions than answers.
- 0xCE0 2y ago
  Clear rationales doesn't sell well: better to obfuscate specs with hidden design decisions and build complexity moat.

IshKebab 2y ago

I can count the number of times I've had binary file corruption due to line ending conversion on zero hands. And I'm old enough to have used FTP extensively. Seems kind of unnecessary.

jxhdbd 2y ago
Have you tried using git with Windows clients? There are so many random line conversions going on and the detection on what is a binary file is clearly broken. I don't understand why the default would be anything but "commit the file as is"
- kbolino 2y ago
  Up until just a few years ago, Notepad on Windows could not handle Unix-style line endings. It probably makes sense now to adopt the as-is convention, but for a while, it made more sense to convert when checking out, and then to prevent spurious diffs, convert back when committing.
  - hnlmorg 2y ago
    Line endings between windows and unix-like systems were so painful that when I started development on my shell scripting language, I wrote a bunch of code to all Linux to handle Windows files and visa versa. Though this has nothing to do with FTP. I’d already abandoned that protocol by then.
- layer8 2y ago
  > I don't understand why the default would be anything but "commit the file as is" Because it’s not uncommon for dev tools on Windows to generate DOS line endings when modifying files (for example when adding an element to an XML configuration file, all line endings of the file may be converted when it is rewritten out from its parsed form), and if those where committed as-is, you’d get a lot of gratuitous changes in the commit and also complaints from the Unix users. For Git, the important thing is to have a .gitattributes file in the repository with “* text=auto” in it (plus more specific settings as desired). The text/binary auto-detection works mostly fine.