This is something that happens a lot. You have one or many hosts you want to access that are behind some kind of firewall (perhaps even behind NAT, and they're using private IP addresses). You have SSH access to the firewall, and from the firewall you can then ssh into some internal host. This works of course, but it's a bit inconvenient. It would be nice if there was some mean to say "ssh user@internalhost" or similar and it "just worked". Let's see a couple of ways to achieve this (the best one at the end!).
In this scenario, the internal hosts are on network 192.168.0.0/24, and for simplicity names relate to addresses, so 192.168.0.1 is internal1, 192.168.0.2 is internal2, etc.. The local box from where you connect is called client. I'm going to assume you have password-less SSH login to the firewall (for example using public key authentication), and possibly to the internal host as well. In other words, your public key on client also appears in the ~/.ssh/authorized_keys file on the target hosts (and public key authentication is enabled, of course). If that is not the case, all the methods described here will work just as fine, but you'll get prompted for passwords, of course.
SSH port forwarding
The first thing one might come up with is something like this:
client:~$ ssh -L1111:192.168.0.1:22 user@firewall # open another shell now... client:~$ ssh -p 1111 user@localhost Last login: Thu Jan 5 22:10:11 GMT 2010 from 10.8.0.210 on pts/12 internal1:~$
and you're in. However, what if you need access to many internal hosts? You need to open an equivalent number of sessions to the firewall, each one forwarding a different local port, and you have then to remember which is which:
client:~$ ssh -L1111:192.168.0.1:22 user@firewall client:~$ ssh -L1112:192.168.0.27:22 user@firewall client:~$ ssh -L1113:192.168.0.4:22 user@firewall ... # Which port was 192.168.0.4 again? 1112 or 1113?
This rapidly becomes inconvenient. Sure, you could use a clever mapping between forwarded ports and IP addresses or names of the internal hosts, but when the number of servers you need to log into grows, that still becomes difficult to track. Also think when using scp, you have to remember which port corresponds to which host and use the -P option to specify it.
But there is another problem with this setup: ok, you're clever and remember all the ports you've forwarded. But some day you need to connect to an entirely different firewall, to access the network behind it, and you light-heartedly reuse port 1111 for forwarding to some internal box. Then you connect to port 1111 on localhost and dang:
client:~$ ssh -L1111:172.16.0.17:22 anotheruser@anotherfirewall client:~$ ssh -p 1111 root@localhost @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. The fingerprint for the RSA key sent by the remote host is fd:b0:d4:75:99:06:d9:fc:a1:6a:0d:71:b8:c8:0b:01. Please contact your system administrator. Add correct host key in /root/.ssh/known_hosts to get rid of this message. Offending key in /root/.ssh/known_hosts:2 RSA host key for [localhost]:1111 has changed and you have requested strict checking. Host key verification failed.
Now it's again possible to work around this problem (see eg HostKeyAlias), but the point is that if you use this method the burden can only increase.
DNAT on the firewall
This is similar (though somewhat better in some respect) to the previous one, but it's the firewall that does all the work (not your local client). The idea is to set up iptables forwarding on the firewall, where connections to different ports are forwarded to different internal hosts. Something like this:
# on the firewall, the Internet interface is 10.0.0.1 firewall:~# iptables -t nat -A PREROUTING -p tcp -d 10.0.0.1 --dport 1111 -j DNAT --to-destination 192.168.0.1:22 firewall:~# iptables -t nat -A PREROUTING -p tcp -d 10.0.0.1 --dport 1112 -j DNAT --to-destination 192.168.0.2:22 firewall:~# iptables -t nat -A PREROUTING -p tcp -d 10.0.0.1 --dport 1113 -j DNAT --to-destination 192.168.0.3:22 # etc.
Then you just ssh to the firewall, say port 1112 for example, and you're transparently taken to 192.168.0.2.
This has the advantage that the mappings are permanent and don't need to be set up on the local client before connecting as you have to do with the previous method. Another advantage is the mappings are fixed, so you can define and name each connection in the configuration file (eg ~/.ssh/config):
# ~/.ssh/config Host internal1 HostName firewall Port 1111 User someuser Host internal2 HostName firewall Port 1112 User someuser ...
so you can just do for example "ssh internal1" and you're in. Also scp works transparently, since it reads the same configuration file.
But of course, for this you must be able to run command as root and/or change the configuration on the firewall, which may or may not be under your control. This method also means that you're going to open a number of ports to the Internet on the firewall, which surely means more management overhead and might not be desirable for other reasons as well.
ProxyCommand
Reading through the SSH options in the man page, there's one called ProxyCommand which looks a bit cryptic at first, but turns out to be extremely useful for our purposes. Essentially, a command is specified that SSH will use as a transport channel to connect to the target host. That is, instead of using TCP as usual, where I/O is performed through a network socket, it will write to and read from respectively the standard input and standard output of the given command. It follows that such command should eventually convert its stdin and stdout into a true network connection to some host. In other words, a command that acts like cat on one side but that also connects to a network socket and moves data back and forth between the two channels. As you guessed, this command is netcat.
Here's how we can use it in the ProxyCommand:
# ~/.ssh/config Host internal1 User root ProxyCommand ssh user@firewall 'nc 192.168.0.1 22'
So here's what happens: when you do "ssh internal1", the ProxyCommand is executed, so a connection is made to the firewall, and from there netcat to the internal host is run. On the firewall, netcat's stdin and stdout are connected to its parent sshd, which in turn is connected via TCP to the ssh command on the client. We effectively have a bidirectional connection between the local computer's ssh stdin/stdout and TCP port 22 of the host at 192.168.0.1 (internal1). We can verify this by running the command by itself:
client:~$ ssh user@firewall 'nc 192.168.0.1 22' SSH-2.0-OpenSSH_5.1p1 Debian-4
The answer here comes directly from the SSH daemon on 192.168.0.1.
This is a rough illustration of the channel set up by the ProxyCommand:
(the actual connection between sshd and netcat on the sshd side uses a UNIX socket - or a pipe in recent versions of SSH -, not shown above for simplicity; the result is what you see above. And "stdin" and "stdout" just mean fd 0 and 1 here)
The dashed red arrow indicates the logical bidirectional channel that is established. Of course this is not enough as it's just a transport channel; the SSH negotiation should be carried on and eventually completed. By using the above command as ProxyCommand, ssh then uses this new channel (instead of TCP) to carry on the usual protocol negotiation, effectively talking to the SSH daemon on the internal host:
client:~$ ssh internal1 Last login: Fri Jan 6 22:33:38 2010 from 172.24.0.254 internal1:~#
If you (like me) are curious and want to verify that ssh is really reading from/writing to the local stdin/stdout, thus effectively using the transport channel created by ProxyCommand, you can modify the ProxyCommand directive as follows:
# ~/.ssh/config Host internal1 User root ProxyCommand tee /tmp/loc2rem | ssh user@firewall 'nc 192.168.0.1 22' | tee /tmp/rem2loc
The first tee command is to capture the stdin of the ProxyCommand (ie, what goes from client to internal1), the last tee is to capture its stdout (ie, what goes from internal1 to client). Adding those extra commands does not change the semantics of the ProxyCommand, which overall still looks like a command that presents stdin/stdout on the local side, and connects to internal1 on the remote side. However, by adding those tees we are able to save the data in transit in two files, one per direction, imaginatively named loc2rem and rem2loc.
After connecting to internal1, let's see what's in the file /tmp/rem2loc:
client:~$ hexdump -C /tmp/rem2loc 00000000 53 53 48 2d 32 2e 30 2d 4f 70 65 6e 53 53 48 5f |SSH-2.0-OpenSSH_| 00000010 35 2e 31 70 31 20 44 65 62 69 61 6e 2d 34 0d 0a |5.1p1 Debian-4..| 00000020 00 00 03 0c 0a 14 fc f2 b3 d9 b1 18 02 79 3a 1c |.............y:.| 00000030 9d eb 94 d7 7b 69 00 00 00 7e 64 69 66 66 69 65 |....{i...~diffie| 00000040 2d 68 65 6c 6c 6d 61 6e 2d 67 72 6f 75 70 2d 65 |-hellman-group-e| 00000050 78 63 68 61 6e 67 65 2d 73 68 61 32 35 36 2c 64 |xchange-sha256,d| 00000060 69 66 66 69 65 2d 68 65 6c 6c 6d 61 6e 2d 67 72 |iffie-hellman-gr| 00000070 6f 75 70 2d 65 78 63 68 61 6e 67 65 2d 73 68 61 |oup-exchange-sha| 00000080 31 2c 64 69 66 66 69 65 2d 68 65 6c 6c 6d 61 6e |1,diffie-hellman| 00000090 2d 67 72 6f 75 70 31 34 2d 73 68 61 31 2c 64 69 |-group14-sha1,di| 000000a0 66 66 69 65 2d 68 65 6c 6c 6d 61 6e 2d 67 72 6f |ffie-hellman-gro| 000000b0 75 70 31 2d 73 68 61 31 00 00 00 0f 73 73 68 2d |up1-sha1....ssh-| 000000c0 72 73 61 2c 73 73 68 2d 64 73 73 00 00 00 9d 61 |rsa,ssh-dss....a| ... 00000300 00 00 00 15 6e 6f 6e 65 2c 7a 6c 69 62 40 6f 70 |....none,zlib@op| 00000310 65 6e 73 73 68 2e 63 6f 6d 00 00 00 00 00 00 00 |enssh.com.......| 00000320 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000330 00 00 00 94 08 1f 00 00 00 81 00 de 49 fc 90 69 |............I..i| 00000340 99 4c 37 9d 2b 65 63 ef d3 7e fa e6 78 5e eb 1d |.L7.+ec..~..x^..| 00000350 d0 a1 2b 09 0a ac 27 2b 22 df 8c 64 a4 a2 ab 7b |..+...'+"..d...{| 00000360 99 ce 0b 77 a9 a5 2e 08 33 d5 2d 53 b2 58 ce df |...w....3.-S.X..| 00000370 fd 17 5d c8 a3 76 6a 9b 98 07 36 26 46 dc 92 15 |..]..vj...6&F...| 00000380 62 8c 3f 4a f0 e0 8d 00 ab 60 a3 b9 e5 5b ae 47 |b.?J.....`...[.G| 00000390 e8 26 51 da 0c 15 a2 73 55 dd b0 63 65 ca e1 dd |.&Q....sU..ce...| 000003a0 de 4c 0c 97 dc 99 42 fd 65 e9 86 7f a5 0e 72 e1 |.L....B.e.....r.| 000003b0 c7 85 41 1e dd 28 de 26 d4 8c 73 00 00 00 01 02 |..A..(.&..s.....| 000003c0 00 00 00 00 00 00 00 00 00 00 02 bc 09 21 00 00 |.............!..| ...
and in /tmp/loc2rem:
client:~$ hexdump -C /tmp/loc2rem 00000000 53 53 48 2d 32 2e 30 2d 4f 70 65 6e 53 53 48 5f |SSH-2.0-OpenSSH_| 00000010 35 2e 32 0d 0a 00 00 03 14 08 14 2c 65 77 30 27 |5.2........,ew0'| 00000020 40 92 5c 0f d3 64 4d a0 de 43 7b 00 00 00 7e 64 |@.\..dM..C{...~d| 00000030 69 66 66 69 65 2d 68 65 6c 6c 6d 61 6e 2d 67 72 |iffie-hellman-gr| 00000040 6f 75 70 2d 65 78 63 68 61 6e 67 65 2d 73 68 61 |oup-exchange-sha| 00000050 32 35 36 2c 64 69 66 66 69 65 2d 68 65 6c 6c 6d |256,diffie-hellm| 00000060 61 6e 2d 67 72 6f 75 70 2d 65 78 63 68 61 6e 67 |an-group-exchang| 00000070 65 2d 73 68 61 31 2c 64 69 66 66 69 65 2d 68 65 |e-sha1,diffie-he| 00000080 6c 6c 6d 61 6e 2d 67 72 6f 75 70 31 34 2d 73 68 |llman-group14-sh| 00000090 61 31 2c 64 69 66 66 69 65 2d 68 65 6c 6c 6d 61 |a1,diffie-hellma| 000000a0 6e 2d 67 72 6f 75 70 31 2d 73 68 61 31 00 00 00 |n-group1-sha1...| 000000b0 0f 73 73 68 2d 72 73 61 2c 73 73 68 2d 64 73 73 |.ssh-rsa,ssh-dss| 000000c0 00 00 00 9d 61 65 73 31 32 38 2d 63 74 72 2c 61 |....aes128-ctr,a| ... 00000300 6e 65 2c 7a 6c 69 62 40 6f 70 65 6e 73 73 68 2e |ne,zlib@openssh.| 00000310 63 6f 6d 2c 7a 6c 69 62 00 00 00 00 00 00 00 00 |com,zlib........| 00000320 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000330 14 06 22 00 00 04 00 00 00 04 00 00 00 20 00 00 |..".......... ..| 00000340 00 00 00 00 00 00 00 00 8c 05 20 00 00 00 81 00 |.......... .....| 00000350 c8 dd 5e f5 21 c9 29 d5 da d6 f3 40 91 a2 d7 84 |..^.!.)....@....| 00000360 dd 73 62 31 48 46 c6 c2 55 f0 9d 4b aa e3 16 df |.sb1HF..U..K....| 00000370 c6 4a 30 39 b2 f8 79 34 19 ef d0 ba 30 dd 69 96 |.J09..y4....0.i.| 00000380 b0 50 88 87 37 f6 8b 78 cb 26 9a 95 b3 f3 2e 6e |.P..7..x.&.....n| 00000390 0a f3 d1 e2 e9 51 0d de df 58 cc 68 5d 28 6f 87 |.....Q...X.h](o.| 000003a0 dc 56 25 96 4e f5 20 9c 6d e3 1f 8e c2 76 ea fd |.V%.N. .m....v..| 000003b0 aa 92 f0 31 76 90 8c e9 c1 2e 13 04 6a c7 b6 47 |...1v.......j..G| 000003c0 f6 5b 31 9f cc 00 1c f6 ee c6 a8 49 6f cc 43 db |.[1........Io.C.| ...
It seems it does what it says on the tin. The contents of those two files do not differ from what we would see if we captured the payload part of the same negotiation occurring over TCP (for example with Wireshark), except the network capture would contain data flowing in both directions properly interleaved instead of two separate files as we have to do when capturing stdin/stdout.
If the DNS on the firewall works correctly, you can even save some keystrokes by using the %h and %p variables that ssh makes available in a ProxyCommand; they are replaced by the supplied hostname and port respectively (the port defaults to 22 if not provided), so you can do
# ~/.ssh/config Host internal1 User root ProxyCommand ssh user@firewall 'nc %h %p'
This, together with the ability to use wildcards in hostnames, makes it possible for example to use a single entry in the config file for all the internal* hosts:
# ~/.ssh/config Host internal* User root ProxyCommand ssh user@firewall 'nc %h %p'
Of course, do NOT do this (for obvious reasons):
# ~/.ssh/config Host * User root ProxyCommand ssh user@firewall 'nc %h %p'
If it's not clear why, try it yourself on a non-critical client host (you have been warned).
You can also use the builtin -W parameters in openssh:
-W host:port
Requests that standard input and output on the client be for‐
warded to host on port over the secure channel. Implies -N, -T,
ExitOnForwardFailure and ClearAllForwardings and works with Pro‐
tocol version 2 only.
Thanks, that should be a relatively recent addition. Looking at the changelog it seems it has been introduced in version 5.4:
* Added a 'netcat mode' to ssh(1): "ssh -W host:port ..." This connects
stdio on the client to a single port forward on the server. This
allows, for example, using ssh as a ProxyCommand to route connections
via intermediate servers. bz#1618
So I guess it could be used as something like (to keep the example in the article)
Host internal1
User root
ProxyCommand ssh -W 192.168.0.1:22 user@firewall
And indeed it works (with VisualHostKey ASCII art to make it more evident):
Thanks!
found a solution. thanks anyway
Hey there,
i have a simliar problem then the other(s). I want to rsync between two server which i connected with Proxycommand. The Problem is the following. I have to sync from root to user httpd but my rsync can just log into user@privatserver.
If i just want to ssh to the server and switch the user "ssh -t privatserver 'sudo su - httpd' works fine but i dont know who to teach this to either rsync or Proxycommand. Do you have an solution ?
Thanks for the awesome tutorial
greetings luccino
PS: i hope you can understand what i mean, not sure about my english skills :P
Thanks very much. It's really useful! :)
Hi again, thanks for your answer. I am trying to scp some root-owned files on the server. That would be possible with a Proxycommand that switches the user after login (since I can't login as root)
so I am looking for a solution where I can
scp proxycommand:/some/root/owned/file .
Is that any cleare? I admit I am not overly familar with the linux command line and it could very well be that I am overlooking the obvious.
Thanks again
Ok, I see. No, ProxyCommand can't do that. Its job is not to give you a shell on the private system; it only establishes a channel to its port 22.
What you can probably do is to use normal ssh (not scp) and then run commands as root on the private server, for example (with ProxyCommand in place)
ssh user@privateserver 'sudo "tar -czvf - /path/to/files"' > localcopy.tar.gz
In the end, that should get the result you want.
Hey,
thanks for the article. I have a somewhat different problem. I regularily need to download files from a server that is not directly reachable and into which I can't ssh as root (PermitRootLogin no). So I would need a possibilty to do something like:
ProxyCommand ssh user@firewall 'nc %h %p su root'
This obviously does not work, do you know a way to get it working? (My netcat is the openbsd version that does not sport the -e)
Thanks a lot
Not sure I understand, but why can't you use a normal ProxyCommand just to set up the channel to the private server and then do something along the lines of
ssh -t user@privateserver 'su -'
Or you may run whatever command you need to run directly using sudo, depending on the configuration.
Remember that when ProxyCommand is in place, you're then talking directly to the private server.