9 ms·

The author missed that sshd will always execute the user's shell and pass it the command with arguments as a `-c` argument. This means that the given command st

by raimue 3y ago

The author missed that sshd will always execute the user's shell and pass it the command with arguments as a `-c` argument. This means that the given command string will always be parsed by the remote shell. This is required to restrict users to certain commands with special shells like scponly or rbash. When you keep in mind that the given command string will be parsed twice, first by your local shell and then again by the remote shell, it becomes clear why a running a remote ssh command behaves like this.

formerly_proven 3y ago
This is more or less baked into the SSH protocol because the exec request only has a single string for the command line. Also the reason echoing a MOTD from rc files or similar crap breaks tools like rsync or scp which use SSH as a neutral transport. SFTP isn’t affected because, while using the three-pipe as a transport, it’s a separate subsystem with its own SSH request type to initiate the channel (just like X11 or port forwarding). If you control client and server you can define your own subsystems and invoke them directly, which avoids this whole mess.
- raimue 3y ago
  I think the confusion comes from the documentation where ssh(1) says that the command "is executed on the remote host instead of a login shell.". Which is true from the perspective of ssh(1) in the sense of the protocol. The client has no control over what the server does with that string. However, sshd(8) clearly documents that it will always execute the login shell, even when a command has been passed. ("All commands are run under the user's login shell as specified in the system password database.") Subsystems open secondary channels to communicate separately from stdin/stdout of the remote command. I used the X11 forwarding before to run a remote command with sudo without getting the password prompt interfering with the protocol: https://raimue.blog/2016/09/25/how-to-run-rsync-on-remote-host-with-sudo/ https://raimue.blog/2016/09/25/how-to-run-rsync-on-remote-ho...
  - formerly_proven 3y ago
    An extra subtlety is that it is the user’s login shell (in the getpwnam sense) but the command is not run in a login shell (in the sh - / sh -l sense). That’s why proper motds or profile files work ok (requesting a shell runs the user’s login shell as a login shell) and don’t break applications. Also the reason why $PATH often differs between the two modes.
zokier 3y ago
yeah, this exec trace from the article is wrong, it is missing one sh -c from the chain $ ssh localhost figlet foobar bar\ baz execve("/usr/bin/ssh", ["ssh", "localhost", "figlet", "foobar", "bar baz"], … execve("/usr/bin/figlet", ["figlet", "foobar", "bar", "baz"], … in practice it looks more like this (traced with execsnoop): PCOMM PID PPID RET ARGS ssh 4255 2058 0 "/usr/bin/ssh" "localhost" "figlet" "foobar" "bar baz" sshd 4256 2147 0 "/usr/bin/sshd" "-D" "-R" bash 4259 4258 0 "/bin/bash" "-c" "figlet foobar bar baz" figlet 4259 4258 0 "/usr/bin/figlet" "foobar" "bar" "baz"
jchw 3y ago
Yep! God though, this hits me in the face so often. Trying to add `sh -c` to fix it is a trap, because obviously, you just create yet another layer of escaping. It really becomes one hell of a puzzle sometimes, especially if you're necessarily nesting another layer of escaping. It feels like you're trying to write a quine. This works: ssh host -- ls "folder\ name" This also works: ssh host -- ls \"folder name\" This works: ssh host -- sh -c \"ls \\\"folder name\\\"\" OK, so clearly, just throwing more escaping at it fixes it. But even if you figure that out, the real mental gymnastics would be figuring out which of the three shells interpreting your command line in the last case would handle shell expansion. In this case, it's the host: ssh host -- sh -c \"ls \\\"folder nam\\\"*\" In this case it's the remote: ssh host -- sh -c "\"ls \\\"folder nam\\\"*\"" Of course where you put the quotes makes no difference. All it does is prevent your shell from processing it. So this works just as well: ssh host -- "sh -c \"ls \\\"folder nam\\\"*\"" If you sit and think each layer through, it usually isn't completely impossible to understand, but the odds that you are going to get something wrong the first time is astonishingly high. It does make me wonder why ssh handles it the way it does, though. Because with the way SSH handles it, it may as well just automatically escape the spaces. Right now, not putting an SSH command in quotes doesn't make much sense unless you for some reason want local shell expansion for something.
- brazzledazzle 3y ago
  Once I run into nested escape wrangling I start to seriously question how I'm trying to accomplish something.
  - josephg 3y ago
    Once I run into nested escape wrangling I seriously question the sanity of my tools.
  - lanstin 3y ago
    If I can't fix it in like 3 tries, I switch to my local shell script having the script to run on the remote end be in a HEREDOC and just scp it it over and then ssh exec it. Inside the HEREDOC, it's a sane environment.
sureglymop 3y ago
Only tangentially related.. Is there something like rbash that is actually secure and more restrictive? Like a shell that only "sees" certain files and folders and can only execute certain commands in a non privileged manner.
- me-vs-cat 3y ago
  The shell rarely "sees" files and folders, except for expanding a glob like "*". When the shell executes "cmd folder/file", the "folder/file" is just a string as far as the shell is concerned. It is the command that uses that string with a function like unlink or open.
  - sureglymop 3y ago
    Okay... you're right. So should I say "the process" and every process forked or exec'd instead of "the shell"? But is it clear what I am looking for?
JeremyNT 3y ago
> This is required to restrict users to certain commands with special shells like scponly or rbash. I don't think this is some specific design goal of OpenSSH, I think it's just a side effect of how shell escaping works. > When you keep in mind that the given command string will be parsed twice, first by your local shell and then again by the remote shell, it becomes clear why a running a remote ssh command behaves like this. I get that this behavior may be surprising to new users, but anybody working with ssh regularly will encounter these kinds of escaping issues. SSH isn't even the only place you'll encounter this. Things like docker etc will have the same "problem". In the case of ssh you can simply write your commands to a file and send them via stdin, or copy a script to the target. The tone of this blog post rubs me wrong. Yes this is a footgun (in the same way many POSIX shell-related things are), but it's not like it's some "problem" with the design of SSH.

by raimue 3y ago

formerly_proven 3y ago

This is more or less baked into the SSH protocol because the exec request only has a single string for the command line. Also the reason echoing a MOTD from rc files or similar crap breaks tools like rsync or scp which use SSH as a neutral transport. SFTP isn’t affected because, while using the three-pipe as a transport, it’s a separate subsystem with its own SSH request type to initiate the channel (just like X11 or port forwarding). If you control client and server you can define your own subsystems and invoke them directly, which avoids this whole mess.

raimue 3y ago
I think the confusion comes from the documentation where ssh(1) says that the command "is executed on the remote host instead of a login shell.". Which is true from the perspective of ssh(1) in the sense of the protocol. The client has no control over what the server does with that string. However, sshd(8) clearly documents that it will always execute the login shell, even when a command has been passed. ("All commands are run under the user's login shell as specified in the system password database.") Subsystems open secondary channels to communicate separately from stdin/stdout of the remote command. I used the X11 forwarding before to run a remote command with sudo without getting the password prompt interfering with the protocol: https://raimue.blog/2016/09/25/how-to-run-rsync-on-remote-host-with-sudo/ https://raimue.blog/2016/09/25/how-to-run-rsync-on-remote-ho...
- formerly_proven 3y ago
  An extra subtlety is that it is the user’s login shell (in the getpwnam sense) but the command is not run in a login shell (in the sh - / sh -l sense). That’s why proper motds or profile files work ok (requesting a shell runs the user’s login shell as a login shell) and don’t break applications. Also the reason why $PATH often differs between the two modes.

zokier 3y ago

yeah, this exec trace from the article is wrong, it is missing one sh -c from the chain $ ssh localhost figlet foobar bar\ baz execve("/usr/bin/ssh", ["ssh", "localhost", "figlet", "foobar", "bar baz"], … execve("/usr/bin/figlet", ["figlet", "foobar", "bar", "baz"], … in practice it looks more like this (traced with execsnoop): PCOMM PID PPID RET ARGS ssh 4255 2058 0 "/usr/bin/ssh" "localhost" "figlet" "foobar" "bar baz" sshd 4256 2147 0 "/usr/bin/sshd" "-D" "-R" bash 4259 4258 0 "/bin/bash" "-c" "figlet foobar bar baz" figlet 4259 4258 0 "/usr/bin/figlet" "foobar" "bar" "baz"

jchw 3y ago

Yep! God though, this hits me in the face so often. Trying to add `sh -c` to fix it is a trap, because obviously, you just create yet another layer of escaping. It really becomes one hell of a puzzle sometimes, especially if you're necessarily nesting another layer of escaping. It feels like you're trying to write a quine. This works: ssh host -- ls "folder\ name" This also works: ssh host -- ls \"folder name\" This works: ssh host -- sh -c \"ls \\\"folder name\\\"\" OK, so clearly, just throwing more escaping at it fixes it. But even if you figure that out, the real mental gymnastics would be figuring out which of the three shells interpreting your command line in the last case would handle shell expansion. In this case, it's the host: ssh host -- sh -c \"ls \\\"folder nam\\\"*\" In this case it's the remote: ssh host -- sh -c "\"ls \\\"folder nam\\\"*\"" Of course where you put the quotes makes no difference. All it does is prevent your shell from processing it. So this works just as well: ssh host -- "sh -c \"ls \\\"folder nam\\\"*\"" If you sit and think each layer through, it usually isn't completely impossible to understand, but the odds that you are going to get something wrong the first time is astonishingly high. It does make me wonder why ssh handles it the way it does, though. Because with the way SSH handles it, it may as well just automatically escape the spaces. Right now, not putting an SSH command in quotes doesn't make much sense unless you for some reason want local shell expansion for something.

brazzledazzle 3y ago
Once I run into nested escape wrangling I start to seriously question how I'm trying to accomplish something.
- josephg 3y ago
  Once I run into nested escape wrangling I seriously question the sanity of my tools.
- lanstin 3y ago
  If I can't fix it in like 3 tries, I switch to my local shell script having the script to run on the remote end be in a HEREDOC and just scp it it over and then ssh exec it. Inside the HEREDOC, it's a sane environment.

sureglymop 3y ago

Only tangentially related.. Is there something like rbash that is actually secure and more restrictive? Like a shell that only "sees" certain files and folders and can only execute certain commands in a non privileged manner.

me-vs-cat 3y ago
The shell rarely "sees" files and folders, except for expanding a glob like "*". When the shell executes "cmd folder/file", the "folder/file" is just a string as far as the shell is concerned. It is the command that uses that string with a function like unlink or open.
- sureglymop 3y ago
  Okay... you're right. So should I say "the process" and every process forked or exec'd instead of "the shell"? But is it clear what I am looking for?

JeremyNT 3y ago

> This is required to restrict users to certain commands with special shells like scponly or rbash. I don't think this is some specific design goal of OpenSSH, I think it's just a side effect of how shell escaping works. > When you keep in mind that the given command string will be parsed twice, first by your local shell and then again by the remote shell, it becomes clear why a running a remote ssh command behaves like this. I get that this behavior may be surprising to new users, but anybody working with ssh regularly will encounter these kinds of escaping issues. SSH isn't even the only place you'll encounter this. Things like docker etc will have the same "problem". In the case of ssh you can simply write your commands to a file and send them via stdin, or copy a script to the target. The tone of this blog post rubs me wrong. Yes this is a footgun (in the same way many POSIX shell-related things are), but it's not like it's some "problem" with the design of SSH.