5 ms·

> Here’s the alphabet, in encoding order: > ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!#$%&'()+,-;=@[]^_`{}~ Yeah I definitely don't want

by abound 4d ago

> Here’s the alphabet, in encoding order: > ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!#$%&'()+,-;=@[]^_`{}~ Yeah I definitely don't want any of those last ~22 characters in file names if I need to do anything with them from a shell

raverbashing 3d ago
100% this (Also for anything that needs to work on a shell script or whatever) I roll my eyes every time some "super secure secret generator" gives me a password with &^%#$ that I need to use on a shell script. Thanks for making my life harder I guess "oh but you just need to escape them" Yes please be my guest trying to deal with character escape whack-a-mole
mmooss 3d ago
I learned, and thought everyone else learned, that the safe characters for POSIX filenames are: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_- That is, the only safe non-alphanumeric characters are dash "-" and underscore "_". Period "." is also considered 'safe' but is tricky for obvious reasons and [IMHO, for the most reliability] best avoided except where necessary. Authoritative source: POSIX.1-2024 standard, sec. 3.265 Portable Filename Character Set https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap03.html#tag_03_265 https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1...
- edoceo 3d ago
  Sometimes, I'd your file starts with a dash things can be tricky. Don't make any mistakes removing a file named '-fr'
brookst 4d ago
What you don’t want a file named `reboot`?
dgacmu 3d ago
Yeah, I'm sympathetic to their desire for compression and efficiency but base64URL is way more shell-friendly. I quite like base58 for human-presented identifiers also. I tend to use it instead of hex for having ~8-16 character IDs of things.
- jerf 3d ago
  Trying to jam in the extra characters for "efficiency" also doesn't really do much. Base84 is 6.4 bits per character, base64 is a flat 6 bits per character, base58 is 5.86 bits per character. It may intuitively feel like having 26 more characters to encode with would be hugely more efficient but for the extra risk of what all those characters get us we're only getting just over .5 more bits per character. I need to be either pretty desperate for efficiency, rare in 2026, or very, very sure that those things are only going places where those characters aren't risky or inconvenient, which is pretty difficult to guarantee since even if it is in fact true today who's to say it will be tomorrow, for this to be worth the risk of the extra characters. Even just the extra characters in base64 are often annoying and we already have multiple common base64 standards because even trying to find just two more characters to fill out a-zA-Z0-9 is hard.
  - dspillett 3d ago
    > Trying to jam in the extra characters for "efficiency" also doesn't really do much. Base84 is 6.4 bits per character, base64 is a flat 6 bits per character, base58 is 5.86 bits per character. Also on efficiency, you are trading off code complexity for a little extra storage: base64 is a nice round 6 bits per character meaning every three bytes encoded is four output. Neither base84 nor base58 align on convenient bit boundaries like that so choosing the output character is more faf. Padding could be more complicated too. Now if you are looking at a per-character limit where the characters are multi-byte (say SSMS shortcuts which are limited to 32767 UCS2 characters) then some form of base4096 (12 bits per character so three 8-bit bytes to two output characters) might be useful. Yes, I have done this: putting a long analysis proc (a replacement for sp_help & friends) into a “shortcut” I was getting close to the 32K-char limit so compressed and base64ed the code and included an unpacker. This was more than enough to deal with the problem (TBH, just stripping comments would have done!) and simplified things in some ways as I no longer needed to escape quotes and such, but I went one step further played with writing a B4096 encoder because I like playing with that sort of silliness. I went with 4096 due to aligning nicely with 4-bit boundaries, and finding 4096 useable characters (avoiding control characters, undefined codes, and other unprintables) is easy. If the limit is actually 32767 or any Unicode characters (including those not in the UCS2 or UTF16 base plane set) then you could perhaps get even more daft though I'm pretty sure it is just 16-bit characters and not full Unicode.
Dwedit 3d ago
Even just "-" in a filename is problematic, since you can name actual files with "--" at the beginning, and the name could match a switch.
- badc0ffee 3d ago
  Typically you can use the -- arg to indicate that everything to the right should not be interpreted as a switch.
  - PhilipRoman 3d ago
    and for programs that don't respect this convention ./--file
orra 3d ago
Moreover, I'd be nervous about the fact that filenames on Windows tend to be case insensitive.
- ncruces 3d ago
  I think the idea is that it being case preserving is sufficient to allow decrypting the filenames. And I guess it relies on the 31 (32?) bit blocks, to make colisions unlikely for short filenames.
- LoganDark 3d ago
  macOS has a case-insensitive filesystem by default too.
  - tobyhinloopen 3d ago
    And don’t you dare changing it unless you like random bugs
    - deleted 3d ago
      [deleted]
    - applfanboysbgon 3d ago
      Sell me on this. Why would you want to change it? What good can actually come of allowing foo and Foo to coexist in the same directory?
      - ErroneousBosh 3d ago
        They are completely different words. Why should they be used to mean the same thing?
      - retrac 3d ago
        Practically? Quite a few codebases from *nix systems rely on the case sensitivity. To do a successful git clone you need a file system that handles both Makefile and makefile, or whatever. I ran into it more than once back in the day when I used Mac OS X as my primary OS. More generally -- case sensitivity is a conceptual nightmare in the Unicode era. Should Cyrillic or Greek be case-insensitive as well? Etc. Do you really want the full complexity of Unicode string handling in your file system? I would suggest treating file names like raw bytes. On modern Linux, anything but NUL is valid.
hnlmorg 3d ago
I don’t see how they’re any worse than spaces in file names, which is an edge case you need to cater for already anyway.
- eklitzke 3d ago
  You only "need" to do this if you are handling arbitrary untrusted filenames. It's pretty common for people to write shell scripts that are only used for handling files they create that don't have unusual characters, and these shell scripts work just fine.
louky 3d ago
Same! Although I still have muscle-memory replacing spaces with _ or - when creating/saving files on any OS. Hasn't done me wrong lo these past 40 years. As for the others tailing the ASCII, no thanks.
- euroderf 3d ago
  Why do file dialogs have a button for this ?

by abound 4d ago

raverbashing 3d ago

100% this (Also for anything that needs to work on a shell script or whatever) I roll my eyes every time some "super secure secret generator" gives me a password with &^%#$ that I need to use on a shell script. Thanks for making my life harder I guess "oh but you just need to escape them" Yes please be my guest trying to deal with character escape whack-a-mole

mmooss 3d ago

I learned, and thought everyone else learned, that the safe characters for POSIX filenames are: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_- That is, the only safe non-alphanumeric characters are dash "-" and underscore "_". Period "." is also considered 'safe' but is tricky for obvious reasons and [IMHO, for the most reliability] best avoided except where necessary. Authoritative source: POSIX.1-2024 standard, sec. 3.265 Portable Filename Character Set https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap03.html#tag_03_265 https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1...

edoceo 3d ago
Sometimes, I'd your file starts with a dash things can be tricky. Don't make any mistakes removing a file named '-fr'

brookst 4d ago

What you don’t want a file named `reboot`?

dgacmu 3d ago

Yeah, I'm sympathetic to their desire for compression and efficiency but base64URL is way more shell-friendly. I quite like base58 for human-presented identifiers also. I tend to use it instead of hex for having ~8-16 character IDs of things.

jerf 3d ago
Trying to jam in the extra characters for "efficiency" also doesn't really do much. Base84 is 6.4 bits per character, base64 is a flat 6 bits per character, base58 is 5.86 bits per character. It may intuitively feel like having 26 more characters to encode with would be hugely more efficient but for the extra risk of what all those characters get us we're only getting just over .5 more bits per character. I need to be either pretty desperate for efficiency, rare in 2026, or very, very sure that those things are only going places where those characters aren't risky or inconvenient, which is pretty difficult to guarantee since even if it is in fact true today who's to say it will be tomorrow, for this to be worth the risk of the extra characters. Even just the extra characters in base64 are often annoying and we already have multiple common base64 standards because even trying to find just two more characters to fill out a-zA-Z0-9 is hard.
- dspillett 3d ago
  > Trying to jam in the extra characters for "efficiency" also doesn't really do much. Base84 is 6.4 bits per character, base64 is a flat 6 bits per character, base58 is 5.86 bits per character. Also on efficiency, you are trading off code complexity for a little extra storage: base64 is a nice round 6 bits per character meaning every three bytes encoded is four output. Neither base84 nor base58 align on convenient bit boundaries like that so choosing the output character is more faf. Padding could be more complicated too. Now if you are looking at a per-character limit where the characters are multi-byte (say SSMS shortcuts which are limited to 32767 UCS2 characters) then some form of base4096 (12 bits per character so three 8-bit bytes to two output characters) might be useful. Yes, I have done this: putting a long analysis proc (a replacement for sp_help & friends) into a “shortcut” I was getting close to the 32K-char limit so compressed and base64ed the code and included an unpacker. This was more than enough to deal with the problem (TBH, just stripping comments would have done!) and simplified things in some ways as I no longer needed to escape quotes and such, but I went one step further played with writing a B4096 encoder because I like playing with that sort of silliness. I went with 4096 due to aligning nicely with 4-bit boundaries, and finding 4096 useable characters (avoiding control characters, undefined codes, and other unprintables) is easy. If the limit is actually 32767 or any Unicode characters (including those not in the UCS2 or UTF16 base plane set) then you could perhaps get even more daft though I'm pretty sure it is just 16-bit characters and not full Unicode.

Dwedit 3d ago

Even just "-" in a filename is problematic, since you can name actual files with "--" at the beginning, and the name could match a switch.

badc0ffee 3d ago
Typically you can use the -- arg to indicate that everything to the right should not be interpreted as a switch.
- PhilipRoman 3d ago
  and for programs that don't respect this convention ./--file

orra 3d ago

Moreover, I'd be nervous about the fact that filenames on Windows tend to be case insensitive.

ncruces 3d ago
I think the idea is that it being case preserving is sufficient to allow decrypting the filenames. And I guess it relies on the 31 (32?) bit blocks, to make colisions unlikely for short filenames.
LoganDark 3d ago
macOS has a case-insensitive filesystem by default too.
- tobyhinloopen 3d ago
  And don’t you dare changing it unless you like random bugs
  - deleted 3d ago
    [deleted]
  - applfanboysbgon 3d ago
    Sell me on this. Why would you want to change it? What good can actually come of allowing foo and Foo to coexist in the same directory?
    - ErroneousBosh 3d ago
      They are completely different words. Why should they be used to mean the same thing?
    - retrac 3d ago
      Practically? Quite a few codebases from *nix systems rely on the case sensitivity. To do a successful git clone you need a file system that handles both Makefile and makefile, or whatever. I ran into it more than once back in the day when I used Mac OS X as my primary OS. More generally -- case sensitivity is a conceptual nightmare in the Unicode era. Should Cyrillic or Greek be case-insensitive as well? Etc. Do you really want the full complexity of Unicode string handling in your file system? I would suggest treating file names like raw bytes. On modern Linux, anything but NUL is valid.

hnlmorg 3d ago

I don’t see how they’re any worse than spaces in file names, which is an edge case you need to cater for already anyway.

eklitzke 3d ago
You only "need" to do this if you are handling arbitrary untrusted filenames. It's pretty common for people to write shell scripts that are only used for handling files they create that don't have unusual characters, and these shell scripts work just fine.

louky 3d ago

Same! Although I still have muscle-memory replacing spaces with _ or - when creating/saving files on any OS. Hasn't done me wrong lo these past 40 years. As for the others tailing the ASCII, no thanks.

euroderf 3d ago
Why do file dialogs have a button for this ?