Skip to content
 

“Jump in” with ssh and netcat

This is something that happens a lot. You have one or many hosts you want to access that are behind some kind of firewall (perhaps even behind NAT, and they're using private IP addresses). You have SSH access to the firewall, and from the firewall you can then ssh into some internal host. This works of course, but it's a bit inconvenient. It would be nice if there was some mean to say "ssh user@internalhost" or similar and it "just worked". Let's see a couple of ways to achieve this (the best one at the end!).

In this scenario, the internal hosts are on network 192.168.0.0/24, and for simplicity names relate to addresses, so 192.168.0.1 is internal1, 192.168.0.2 is internal2, etc.. The local box from where you connect is called client. I'm going to assume you have password-less SSH login to the firewall (for example using public key authentication), and possibly to the internal host as well. In other words, your public key on client also appears in the ~/.ssh/authorized_keys file on the target hosts (and public key authentication is enabled, of course). If that is not the case, all the methods described here will work just as fine, but you'll get prompted for passwords, of course.

SSH port forwarding

The first thing one might come up with is something like this:

client:~$ ssh -L1111:192.168.0.1:22 user@firewall   # open another shell now...
client:~$ ssh -p 1111 user@localhost
Last login: Thu Jan  5 22:10:11 GMT 2010 from 10.8.0.210 on pts/12
internal1:~$

and you're in. However, what if you need access to many internal hosts? You need to open an equivalent number of sessions to the firewall, each one forwarding a different local port, and you have then to remember which is which:

client:~$ ssh -L1111:192.168.0.1:22 user@firewall
client:~$ ssh -L1112:192.168.0.27:22 user@firewall
client:~$ ssh -L1113:192.168.0.4:22 user@firewall
...
# Which port was 192.168.0.4 again? 1112 or 1113?

This rapidly becomes inconvenient. Sure, you could use a clever mapping between forwarded ports and IP addresses or names of the internal hosts, but when the number of servers you need to log into grows, that still becomes difficult to track. Also think when using scp, you have to remember which port corresponds to which host and use the -P option to specify it.
But there is another problem with this setup: ok, you're clever and remember all the ports you've forwarded. But some day you need to connect to an entirely different firewall, to access the network behind it, and you light-heartedly reuse port 1111 for forwarding to some internal box. Then you connect to port 1111 on localhost and dang:

client:~$ ssh -L1111:172.16.0.17:22 anotheruser@anotherfirewall
client:~$ ssh -p 1111 root@localhost
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
fd:b0:d4:75:99:06:d9:fc:a1:6a:0d:71:b8:c8:0b:01.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending key in /root/.ssh/known_hosts:2
RSA host key for [localhost]:1111 has changed and you have requested strict checking.
Host key verification failed.

Now it's again possible to work around this problem (see eg HostKeyAlias), but the point is that if you use this method the burden can only increase.

DNAT on the firewall

This is similar (though somewhat better in some respect) to the previous one, but it's the firewall that does all the work (not your local client). The idea is to set up iptables forwarding on the firewall, where connections to different ports are forwarded to different internal hosts. Something like this:

# on the firewall, the Internet interface is 10.0.0.1
firewall:~# iptables -t nat -A PREROUTING -p tcp -d 10.0.0.1 --dport 1111 -j DNAT --to-destination 192.168.0.1:22
firewall:~# iptables -t nat -A PREROUTING -p tcp -d 10.0.0.1 --dport 1112 -j DNAT --to-destination 192.168.0.2:22
firewall:~# iptables -t nat -A PREROUTING -p tcp -d 10.0.0.1 --dport 1113 -j DNAT --to-destination 192.168.0.3:22
# etc.

Then you just ssh to the firewall, say port 1112 for example, and you're transparently taken to 192.168.0.2.
This has the advantage that the mappings are permanent and don't need to be set up on the local client before connecting as you have to do with the previous method. Another advantage is the mappings are fixed, so you can define and name each connection in the configuration file (eg ~/.ssh/config):

# ~/.ssh/config
Host internal1
HostName firewall
Port 1111
User someuser

Host internal2
HostName firewall
Port 1112
User someuser
...

so you can just do for example "ssh internal1" and you're in. Also scp works transparently, since it reads the same configuration file.
But of course, for this you must be able to run command as root and/or change the configuration on the firewall, which may or may not be under your control. This method also means that you're going to open a number of ports to the Internet on the firewall, which surely means more management overhead and might not be desirable for other reasons as well.

ProxyCommand

Reading through the SSH options in the man page, there's one called ProxyCommand which looks a bit cryptic at first, but turns out to be extremely useful for our purposes. Essentially, a command is specified that SSH will use as a transport channel to connect to the target host. That is, instead of using TCP as usual, where I/O is performed through a network socket, it will write to and read from respectively the standard input and standard output of the given command. It follows that such command should eventually convert its stdin and stdout into a true network connection to some host. In other words, a command that acts like cat on one side but that also connects to a network socket and moves data back and forth between the two channels. As you guessed, this command is netcat.

Here's how we can use it in the ProxyCommand:

# ~/.ssh/config
Host internal1
User root
ProxyCommand ssh user@firewall 'nc 192.168.0.1 22'

So here's what happens: when you do "ssh internal1", the ProxyCommand is executed, so a connection is made to the firewall, and from there netcat to the internal host is run. On the firewall, netcat's stdin and stdout are connected to its parent sshd, which in turn is connected via TCP to the ssh command on the client. We effectively have a bidirectional connection between the local computer's ssh stdin/stdout and TCP port 22 of the host at 192.168.0.1 (internal1). We can verify this by running the command by itself:

client:~$ ssh user@firewall 'nc 192.168.0.1 22'
SSH-2.0-OpenSSH_5.1p1 Debian-4

The answer here comes directly from the SSH daemon on 192.168.0.1.

This is a rough illustration of the channel set up by the ProxyCommand:

(the actual connection between sshd and netcat on the sshd side uses a UNIX socket - or a pipe in recent versions of SSH -, not shown above for simplicity; the result is what you see above. And "stdin" and "stdout" just mean fd 0 and 1 here)
The dashed red arrow indicates the logical bidirectional channel that is established. Of course this is not enough as it's just a transport channel; the SSH negotiation should be carried on and eventually completed. By using the above command as ProxyCommand, ssh then uses this new channel (instead of TCP) to carry on the usual protocol negotiation, effectively talking to the SSH daemon on the internal host:

client:~$ ssh internal1
Last login: Fri Jan  6 22:33:38 2010 from 172.24.0.254
internal1:~#

If you (like me) are curious and want to verify that ssh is really reading from/writing to the local stdin/stdout, thus effectively using the transport channel created by ProxyCommand, you can modify the ProxyCommand directive as follows:

# ~/.ssh/config
Host internal1
User root
ProxyCommand tee /tmp/loc2rem | ssh user@firewall 'nc 192.168.0.1 22' | tee /tmp/rem2loc

The first tee command is to capture the stdin of the ProxyCommand (ie, what goes from client to internal1), the last tee is to capture its stdout (ie, what goes from internal1 to client). Adding those extra commands does not change the semantics of the ProxyCommand, which overall still looks like a command that presents stdin/stdout on the local side, and connects to internal1 on the remote side. However, by adding those tees we are able to save the data in transit in two files, one per direction, imaginatively named loc2rem and rem2loc.

After connecting to internal1, let's see what's in the file /tmp/rem2loc:

client:~$ hexdump -C /tmp/rem2loc
00000000  53 53 48 2d 32 2e 30 2d  4f 70 65 6e 53 53 48 5f  |SSH-2.0-OpenSSH_|
00000010  35 2e 31 70 31 20 44 65  62 69 61 6e 2d 34 0d 0a  |5.1p1 Debian-4..|
00000020  00 00 03 0c 0a 14 fc f2  b3 d9 b1 18 02 79 3a 1c  |.............y:.|
00000030  9d eb 94 d7 7b 69 00 00  00 7e 64 69 66 66 69 65  |....{i...~diffie|
00000040  2d 68 65 6c 6c 6d 61 6e  2d 67 72 6f 75 70 2d 65  |-hellman-group-e|
00000050  78 63 68 61 6e 67 65 2d  73 68 61 32 35 36 2c 64  |xchange-sha256,d|
00000060  69 66 66 69 65 2d 68 65  6c 6c 6d 61 6e 2d 67 72  |iffie-hellman-gr|
00000070  6f 75 70 2d 65 78 63 68  61 6e 67 65 2d 73 68 61  |oup-exchange-sha|
00000080  31 2c 64 69 66 66 69 65  2d 68 65 6c 6c 6d 61 6e  |1,diffie-hellman|
00000090  2d 67 72 6f 75 70 31 34  2d 73 68 61 31 2c 64 69  |-group14-sha1,di|
000000a0  66 66 69 65 2d 68 65 6c  6c 6d 61 6e 2d 67 72 6f  |ffie-hellman-gro|
000000b0  75 70 31 2d 73 68 61 31  00 00 00 0f 73 73 68 2d  |up1-sha1....ssh-|
000000c0  72 73 61 2c 73 73 68 2d  64 73 73 00 00 00 9d 61  |rsa,ssh-dss....a|
...
00000300  00 00 00 15 6e 6f 6e 65  2c 7a 6c 69 62 40 6f 70  |....none,zlib@op|
00000310  65 6e 73 73 68 2e 63 6f  6d 00 00 00 00 00 00 00  |enssh.com.......|
00000320  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000330  00 00 00 94 08 1f 00 00  00 81 00 de 49 fc 90 69  |............I..i|
00000340  99 4c 37 9d 2b 65 63 ef  d3 7e fa e6 78 5e eb 1d  |.L7.+ec..~..x^..|
00000350  d0 a1 2b 09 0a ac 27 2b  22 df 8c 64 a4 a2 ab 7b  |..+...'+"..d...{|
00000360  99 ce 0b 77 a9 a5 2e 08  33 d5 2d 53 b2 58 ce df  |...w....3.-S.X..|
00000370  fd 17 5d c8 a3 76 6a 9b  98 07 36 26 46 dc 92 15  |..]..vj...6&F...|
00000380  62 8c 3f 4a f0 e0 8d 00  ab 60 a3 b9 e5 5b ae 47  |b.?J.....`...[.G|
00000390  e8 26 51 da 0c 15 a2 73  55 dd b0 63 65 ca e1 dd  |.&Q....sU..ce...|
000003a0  de 4c 0c 97 dc 99 42 fd  65 e9 86 7f a5 0e 72 e1  |.L....B.e.....r.|
000003b0  c7 85 41 1e dd 28 de 26  d4 8c 73 00 00 00 01 02  |..A..(.&..s.....|
000003c0  00 00 00 00 00 00 00 00  00 00 02 bc 09 21 00 00  |.............!..|
...

and in /tmp/loc2rem:

client:~$ hexdump -C /tmp/loc2rem
00000000  53 53 48 2d 32 2e 30 2d  4f 70 65 6e 53 53 48 5f  |SSH-2.0-OpenSSH_|                                                
00000010  35 2e 32 0d 0a 00 00 03  14 08 14 2c 65 77 30 27  |5.2........,ew0'|                                                           
00000020  40 92 5c 0f d3 64 4d a0  de 43 7b 00 00 00 7e 64  |@.\..dM..C{...~d|                                                           
00000030  69 66 66 69 65 2d 68 65  6c 6c 6d 61 6e 2d 67 72  |iffie-hellman-gr|                                                           
00000040  6f 75 70 2d 65 78 63 68  61 6e 67 65 2d 73 68 61  |oup-exchange-sha|
00000050  32 35 36 2c 64 69 66 66  69 65 2d 68 65 6c 6c 6d  |256,diffie-hellm|
00000060  61 6e 2d 67 72 6f 75 70  2d 65 78 63 68 61 6e 67  |an-group-exchang|                                                           
00000070  65 2d 73 68 61 31 2c 64  69 66 66 69 65 2d 68 65  |e-sha1,diffie-he|                                                           
00000080  6c 6c 6d 61 6e 2d 67 72  6f 75 70 31 34 2d 73 68  |llman-group14-sh|                                                           
00000090  61 31 2c 64 69 66 66 69  65 2d 68 65 6c 6c 6d 61  |a1,diffie-hellma|                                                           
000000a0  6e 2d 67 72 6f 75 70 31  2d 73 68 61 31 00 00 00  |n-group1-sha1...|                                                           
000000b0  0f 73 73 68 2d 72 73 61  2c 73 73 68 2d 64 73 73  |.ssh-rsa,ssh-dss|                                                           
000000c0  00 00 00 9d 61 65 73 31  32 38 2d 63 74 72 2c 61  |....aes128-ctr,a|
...
00000300  6e 65 2c 7a 6c 69 62 40  6f 70 65 6e 73 73 68 2e  |ne,zlib@openssh.|                                                           
00000310  63 6f 6d 2c 7a 6c 69 62  00 00 00 00 00 00 00 00  |com,zlib........|                                                           
00000320  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|                                                           
00000330  14 06 22 00 00 04 00 00  00 04 00 00 00 20 00 00  |..".......... ..|                                                           
00000340  00 00 00 00 00 00 00 00  8c 05 20 00 00 00 81 00  |.......... .....|                                                           
00000350  c8 dd 5e f5 21 c9 29 d5  da d6 f3 40 91 a2 d7 84  |..^.!.)....@....|                                                           
00000360  dd 73 62 31 48 46 c6 c2  55 f0 9d 4b aa e3 16 df  |.sb1HF..U..K....|                                                           
00000370  c6 4a 30 39 b2 f8 79 34  19 ef d0 ba 30 dd 69 96  |.J09..y4....0.i.|                                                           
00000380  b0 50 88 87 37 f6 8b 78  cb 26 9a 95 b3 f3 2e 6e  |.P..7..x.&.....n|                                                           
00000390  0a f3 d1 e2 e9 51 0d de  df 58 cc 68 5d 28 6f 87  |.....Q...X.h](o.|                                                           
000003a0  dc 56 25 96 4e f5 20 9c  6d e3 1f 8e c2 76 ea fd  |.V%.N. .m....v..|                                                           
000003b0  aa 92 f0 31 76 90 8c e9  c1 2e 13 04 6a c7 b6 47  |...1v.......j..G|                                                           
000003c0  f6 5b 31 9f cc 00 1c f6  ee c6 a8 49 6f cc 43 db  |.[1........Io.C.|    
...

It seems it does what it says on the tin. The contents of those two files do not differ from what we would see if we captured the payload part of the same negotiation occurring over TCP (for example with Wireshark), except the network capture would contain data flowing in both directions properly interleaved instead of two separate files as we have to do when capturing stdin/stdout.

If the DNS on the firewall works correctly, you can even save some keystrokes by using the %h and %p variables that ssh makes available in a ProxyCommand; they are replaced by the supplied hostname and port respectively (the port defaults to 22 if not provided), so you can do

# ~/.ssh/config
Host internal1
User root
ProxyCommand ssh user@firewall 'nc %h %p'

This, together with the ability to use wildcards in hostnames, makes it possible for example to use a single entry in the config file for all the internal* hosts:

# ~/.ssh/config
Host internal*
User root
ProxyCommand ssh user@firewall 'nc %h %p'

Of course, do NOT do this (for obvious reasons):

# ~/.ssh/config
Host *
User root
ProxyCommand ssh user@firewall 'nc %h %p'

If it's not clear why, try it yourself on a non-critical client host (you have been warned).

9 Comments

  1. Nico says:

    You can also use the builtin -W parameters in openssh:
    -W host:port
    Requests that standard input and output on the client be for‐
    warded to host on port over the secure channel. Implies -N, -T,
    ExitOnForwardFailure and ClearAllForwardings and works with Pro‐
    tocol version 2 only.

    • waldner says:

      Thanks, that should be a relatively recent addition. Looking at the changelog it seems it has been introduced in version 5.4:

      * Added a 'netcat mode' to ssh(1): "ssh -W host:port ..." This connects
      stdio on the client to a single port forward on the server. This
      allows, for example, using ssh as a ProxyCommand to route connections
      via intermediate servers. bz#1618

      So I guess it could be used as something like (to keep the example in the article)

      Host internal1
      User root
      ProxyCommand ssh -W 192.168.0.1:22 user@firewall

      And indeed it works (with VisualHostKey ASCII art to make it more evident):

      $ ssh internal1
      Host key fingerprint is 01:b0:7a:87:ff:13:36:26:bd:e4:1c:ae:78:1f:72:34
      +--[ RSA 1024]----+
      |    ...          |
      |     . .         |
      |    .   .        |
      |   +     .       |
      | E+ .   S        |
      |. .o             |
      |....+            |
      |oo.**o.          |
      |o.o==+..         |
      +-----------------+
      
      Host key fingerprint is 90:47:e7:f8:a6:53:fa:14:4c:ff:3a:b3:9f:b7:ea:c0
      +--[ECDSA  256]---+
      |         .o .    |
      |         o +     |
      |        o o .    |
      |         + o     |
      |        S o =    |
      |       .   * .   |
      |        E =   .  |
      |         +oo..   |
      |         .*B+.   |
      +-----------------+
      
      Last login: Wed Oct  5 16:48:36 2011 from 10.18.0.233
      internal1 #

      Thanks!

  2. luccino says:

    found a solution. thanks anyway

  3. luccino says:

    Hey there,

    i have a simliar problem then the other(s). I want to rsync between two server which i connected with Proxycommand. The Problem is the following. I have to sync from root to user httpd but my rsync can just log into user@privatserver.

    If i just want to ssh to the server and switch the user "ssh -t privatserver 'sudo su - httpd' works fine but i dont know who to teach this to either rsync or Proxycommand. Do you have an solution ?

    Thanks for the awesome tutorial

    greetings luccino

    PS: i hope you can understand what i mean, not sure about my english skills :P

  4. Jianing Yang says:

    Thanks very much. It's really useful! :)

  5. Conrad says:

    Hi again, thanks for your answer. I am trying to scp some root-owned files on the server. That would be possible with a Proxycommand that switches the user after login (since I can't login as root)

    so I am looking for a solution where I can

    scp proxycommand:/some/root/owned/file .

    Is that any cleare? I admit I am not overly familar with the linux command line and it could very well be that I am overlooking the obvious.

    Thanks again

    • waldner says:

      Ok, I see. No, ProxyCommand can't do that. Its job is not to give you a shell on the private system; it only establishes a channel to its port 22.
      What you can probably do is to use normal ssh (not scp) and then run commands as root on the private server, for example (with ProxyCommand in place)

      ssh user@privateserver 'sudo "tar -czvf - /path/to/files"' > localcopy.tar.gz

      In the end, that should get the result you want.

  6. Conrad says:

    Hey,

    thanks for the article. I have a somewhat different problem. I regularily need to download files from a server that is not directly reachable and into which I can't ssh as root (PermitRootLogin no). So I would need a possibilty to do something like:

    ProxyCommand ssh user@firewall 'nc %h %p su root'

    This obviously does not work, do you know a way to get it working? (My netcat is the openbsd version that does not sport the -e)

    Thanks a lot

    • waldner says:

      Not sure I understand, but why can't you use a normal ProxyCommand just to set up the channel to the private server and then do something along the lines of

      ssh -t user@privateserver 'su -'

      Or you may run whatever command you need to run directly using sudo, depending on the configuration.
      Remember that when ProxyCommand is in place, you're then talking directly to the private server.