Volume Number: 23 (2007)
Issue Number: 10
Column Tag: Mac in the Shell

ssh Basics and More

Is there anything ssh can't do?

by Edward Marczak

If it weren't for OS X, I'd probably be toting around a FreeBSD laptop now. While OS 9 had its charms, when I first fired up Terminal.app under OS X 10.0 and used ping to test a network route, I knew it was a keeper (and it didn't even look that good back then!). The command shell was an environment I was comfortable in, and had the tools I needed to get work done. However, I often need to open up a shell on a remote machine. More than any other command, I probably use ssh the most. At the very least, it's the one I'd most miss. Why? What's the draw and advantage here? Read on and find out.

In the beginning, there was telnet. Telnet allowed one access to a command environment on a remote machine, typically Unix-based. It was a fine way of doing so for a very long time - until the Internet-with-a-capital-I came along. Since telnet passed along all information in clear text, anyone between you and your remote session could happily sniff and read everything you typed, or that the host sent back to you. That's just not cool, man.

ssh was created by Tatu Ylonen in 1995 and was originally released under an open-source license. This initial release formed version 1 of the ssh protocol. After forming SSH Communications, versions of ssh became commercial, with free for non-commercial use versions. Both versions went closed-source. In 1997, however, there was a movement to make the ssh protocol and Internet standard under the IETF. This led to version 2 of the protocol, which cleaned up some security issues, bugs, and made other reliability enhancements.

Meanwhile, in 1999, the OpenBSD team, took the early open source version of Tatu Ylonen's code and began modifying and rewriting, to bring it up to current standards. This is what became OpenSSH, and was released in December 1999. Markus Friedl added support for version 2 of the ssh protocol in early 2000.

Fast forward a bit, and under OS X "Tiger", we're blessed with version 4.5p1 of this venerable application. While we can use ssh as a simple remote login replacement of telnet (and rlogin...and rsh), it has quite a bit more functionality. Let's dig in.

The basics of ssh are pretty straightforward. A machine that you want to access, called the server, must be running sshd, the ssh server daemon, which will listen on some port (22 by default). Under OS X, this is accomplished via the "Sharing" pref pane. Simply check the box next to "Remote Login", as in Figure 1.

Figure 1 - Enabling ssh in the Sharing Pref Pane

We're even given instructions: "To log in to this computer remotely...". Of course, these instructions may not be entirely correct, depending on certain circumstances. In figure 1, the pref pane is picking up an IP address from one of my Parallels interfaces, so, its advice is completely bogus. Be sure to figure out your own appropriate IP address - an external one if necessary - before blindly trusting what the pref pane tells you.

As of OS X 10.4, sshd doesn't run continuously, but is run by launchd on demand. launchd listens on the port specified in the sshd.plist file, and when a connection is made, launchd fires up sshd and passes along the connection. It's due to this that you won't see sshd in a process listing, even though you've enabled it (as you will in 10.3...or FreeBSD and Linux, for that matter).

Now that you have a machine acting as a server, what can you do with it? For starters, you can access a shell on the remote machine from any ssh client. This includes clients running on Linux/BSD, OS X, Windows or even a smart phone that has ssh capabilities (like the Nokia N800). That's cool enough on its own. We can't have Apple Remote Desktop from our iPhone, but I can access my OS X Server using a non-Apple smart phone. Brilliant!

Since we're living in OS X-land, let's pretend our client lives there. Pop open terminal.app, and simply:

ssh user@host

You'll be asked for a password, and then presented with a remote shell. Couldn't be easier. If you have the muscle memory built up from the telnet days, you can also supply the user name with the "-l" switch, like I tend to do:

ssh -l admin django.radiotope.com

You can also supply the port, if the server is running on something besides the well defined ssh port of 22:

ssh -l admin -p 8022 django.radiotope.com

Finally, if you want a little more detail about what's going on, use one or more verbose switches (-v):

ssh -v -l admin -p 8022 django.radiotope.com

Once you have your remote shell, you can start typing, just as if you were sitting at a console connected directly to that machine. One last tip: if you're using the defaults, and your current user name is the same as what you'll pass off to the remote, simply give the host name:

ssh django.radiotope.com

Easy, right?

Of course, the first 's' in ssh stands for "secure". You should have noticed that the first time you connect to a server, you're asked to verify and accept the host's key. The exchange looks something like this:

$ ssh marczak@django.radiotope.com
The authenticity of host ' django.radiotope.com (10.10.7.8)' can't be established.
RSA key fingerprint is cd:42:72:31:47:15:80:e6:31:36:66:f1:56:ef:db.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'django.radiotope.com' (RSA) to the list of known hosts.
marczak@ django.radiotope.com's password:
Last login: Wed Aug 29 10:56:25 2007

Also note the part about "permanently added...to the list of known hosts". This list is kept in ~/.ssh, in a file named "known_hosts". Don't forget that the "." In front of the file name means that it's hidden in terminal unless you use the "-a" switch with ls. (OK - or if you're root, or have performed some other trickery with environment variables, etc. If you've done that, then you probably know what you're doing...move along, nothing to see here). ssh adds both the host's key, dns name and IP address to the known_hosts file.

If the host key ever changes, you'll be notified when you connect:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
55:78:c6:24:2b:44:3c:55:87:c5:e9:01:17:ef:83:6d.
Please contact your system administrator.
Add correct host key in /Users/marczak/.ssh/known_hosts to get rid of this message.
Offending key in /Users/marczak/.ssh/known_hosts:70
RSA host key for gryphon.radiotope.com has changed and you have requested strict checking.
Host key verification failed.

...and you'll be left back at a prompt, rather than connecting. If the host's key and IP address change, you will see a different message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The RSA host key for gryphon.radiotope.com has changed,
and the key for the according IP address 192.168.88.6
has a different value. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
Offending key for IP in /Users/marczak/.ssh/known_hosts:59

All of this helps keep the first 's' in ssh! In the next section, I'll talk a little bit about configuration of ssh, but do know that you can turn this feature off. Doing so will warn you, but still allow you to connect:

$ ssh -l root django.radiotope.com Warning: the RSA host key for django.radiotope.com' differs from the key for the IP address '192.168.88.3' Offending key for IP in /Users/marczak/.ssh/known_hosts:27 Matching host key in /Users/marczak/.ssh/known_hosts:30 Are you sure you want to continue connecting (yes/no)? yes Password: Last login: Wed Aug 29 10:56:25 2007

These are excellent things to be mindful of as you use ssh.

Now that you have the basics down, let's look at some configuration options. On an OS X machine, you'll find the file /etc/sshd_config that configures the sshd server. Since this is an introductory and cookbook type article, I won't exhaustively detail every option, but only ones that impact the options I'll refer to in this article itself.

If you open up /etc/sshd_config, you'll notice that much of it is commented out with "#" symbols. This shows the default values of these parameters. Uncommenting them will allow you to change the values. Unlike other varieties of Unix, and thanks to launchd, just make the change, and it will go into effect upon next connection. Since sshd isn't really running all the time, there's no need to HUP or restart it.

Any ssh server exposed to the Internet - directly or through a firewall via port forwarding or PAT - should immediately uncomment the "PermitRootLogin" line, and change it to "no".

Now, that was a bit simplistic, and on OS X Server, we need to take one other thing into account: if we're running Open Directory. An unadorned ssh server is needed when creating an Open Directory replica. The replica uses ssh to connect to the OD master and transfer the LDAP database. So, you can either a) not expose an OD master to the Internet - use another machine for remote ssh access - or b) remember to toggle this back when adding a replica.

On the client side, you can edit /etc/ssh_config. I typically add the following two lines to any client I touch immediately:

TCPKeepAlive yes
ServerAliveInterval 30

Just tack these onto the end of the stock ssh_config file. These will 'ping' the server using a handshake protocol that stops your from getting dropped if you're idle.

As referenced earlier, if a server's key changes, you'll be stopped from connecting in the default state. This is controlled by the "StrictHostChecking" option. It's set to "yes". While possibly unfriendly, a host's key should basically never change, so, I recommend leaving it set this way. Other possible values are "no", which just blindly adds the new host key to your known_hosts file, and "ask", which warns you and asks if you would still like to connect. If you answer "yes", the new key will be added to your known_hosts file at that point.

Nicely, you can specify options as parameters, so, if you use DHCP to hand out client addresses, many times the IP address will not match the key as recorded - properly so. If you are connecting to a host that you know will not match, you can leave your /etc/ssh_config file alone, and for this time, use this:

ssh -o stricthostkeychecking=ask django.radiotope.com

There are nice man pages for both ssh_config and sshd_config that explain each option.

So, now you have an ssh server, or servers to use, as every OS X machine in your midst is potentially an ssh server. "Remote Login" is enabled by default on OS X Server, and I enable it on OS X in my standard operating environment (SOE) images. What can we do from here?

One thing that makes life with ssh very sweet is password-less login. While this is possible in a single-sign on environment, you may be dealing with machines that aren't part of this setup. Password-less login uses the keys that were referenced above. To do so, you need to generate a public key for your account, and transport that key to any server that you'd like to login on without using a password. Here's a short tutorial/how-to.

First, generate your keys:

$ ssh-keygen -t rsa -b 2048

You'll receive the following output, during which, you can press return to accept the defaults for each question:

Generating public/private rsa key pair.
Enter file in which to save the key (/Users/edacct/.ssh/id_rsa): 
Created directory '/Users/edacct/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /Users/edacct/.ssh/id_rsa.
Your public key has been saved in /Users/edacct/.ssh/id_rsa.pub.
The key fingerprint is:
47:51:7b:ec:de:61:b4:e0:23:7d:fc:bd:1f:65:8c:11 edacct@Jack-Kerouac.local

Again, simply press return for each question - including the passphrase questions.

Second, notice the line that states, "Your public key has been saved in /Users/edacct/.ssh/id_rsa.pub". This is the file that we need to transport to the server, and add its contents to the target account's authorized_keys file. You can do this several different ways, and I'll cover one simple one here.

Simply `cat /Users/edacct/.ssh/id_rsa.pub | pbcopy` (substituting the appropriate account name), which will place the contents of your key onto the clipboard, or "paste buffer", ready to be pasted (Apple-v or Edit->Paste). Now, ssh into the target machine. For example:

ssh edacct@django.radiotope.com

You'll still need to supply your password, as we haven't altered the server yet. Once in, edit the ~/.ssh/authorized_keys file, and tack the paste buffer onto the end:

$ vi .ssh/authorized_keys

ssh-rsa CAAB5NzaC1yc2ECBEBIwAAAQEAvyDnmn09A6z2dHJrh8RXq5BVoJOIW4OZPZ4ex++
Z0ixF6BBtEzOYkF04LaTTJaOykySidz2R6PeqZyFXSahZhUvLJLNmqNlGQZP//
QDUbrJm0f0jX0tBc3PUbJHfhfw4S1HTTeXJreJ/L4ue4AGZziqR/
mYNkuyaVyfe07NInQPHebr9asvrEa6F9zxmqB3Ngn2baw== edacct@Jack-Kerouac.local

This may be the only entry in the file, which is just fine. If there are already entries, navigate to the end and place this on its own line.

Finally, logout of the target machine, and then ssh right back in again. This time, you should find yourself logged in, no password asked of you. While this may be a convenience when you're working interactively, the real bonus comes in when using ssh or scp (a utility that copies data over an ssh connection) in a script - no human present. Also, you'll see further on that ssh can do much, much more than simply present you with a remote interactive shell.

Let's get to the good stuff. You've seen the basics, but what else can ssh do to make your tech life easier and better?

Port Forwarding

Port forwarding may be one of the more popular 'alternative' ways to use ssh. Why the heck would you want to do that? Well, the simple reason is that you can protect clear-text protocols in this manner. Take POP mail retrieval for example. Typically, not only are the username and password sent in the clear, but the body of each message is transferred in plain text. If you're at a conference, or sitting in your local WiFi-equipped coffee shop, you're exposing all of this information to anyone that wants to sniff it. And that's just not cool. So, we tunnel it through ssh.

Here's an example, and then we'll dissect it:

ssh -N -L 8110:mail.radiotope.com:110 edacct@mail.radiotope.com

This tells ssh:

-N no interactive shell, thankyouverymuch.

-L On this local machine, take port 8110, and forward all requests to mail.radiotope.com's port 110.

edacct@mail.radiotope.com is the id that we're authenticating with, and the host that we're forwarding through, which, in this case, is that same as the remote destination itself.

You're still connecting to a host, but then the "-L" switch tells that host what to do with packets arriving on the port you specified. Interestingly, you can forward to another machine that the remote machine has access to. Let's say that your office has a single ssh bastion host, but you need to tunnel to another machine on that LAN. No prob. Just forward to that machine:

ssh -N -L 8110:192.168.55.6:110 edacct@gw.radiotope.com

What's going on here? Look at figure 2, which may help.

Figure 2 - ssh tunneling pass-through

In Figure 2's example, the firewall at gw.radiotope.com has port 22 port forwarded to server.radiotope.com. Telling ssh to connect to gw really connects us to server. server.radiotope.com handles the ssh connect, authentication and session, but then, as instructed by the "-L" switch, passes the traffic through to 192.168.55.6's port 110.

Once a port forward is in place, it's your local machine that now has a port open for listening. So, in the example of POP that we've been using, you'd need to tell your mail program that the server is at localhost, or 127.0.0.1, running on port 8110. Of course, this then gets shuttled over the ssh connection to the designated remote machine on the designated port.

Remote Port Forwarding

Remote port forwarding allows the remote machine to define the tunnel. For instance, take this example:

ssh -N -R 8022:127.0.0.1:22 admin@loud.radiotope.com

This tells the ssh server - the machine that you're ssh'ing into - to take traffic on port 8022, and pass it back along to the client machine at 127.0.0.1 - itself, on port 22. In this case, the server is loud.radiotope.com, and the client is whatever machine you're on. Then, from loud.radiotope.com, we can simply:

ssh -l marczak -p 8022 127.0.0.1

...to log into the remote host. What's happening here? Let visualize it again:

Figure 3 - Remote port forward

You, at 10.10.10.7 have a very restrictive firewall in place that doesn't allow any incoming connections. However, it does allow you to get an ssh connection out. To allow someone on loud.radiotope.com to ssh into your machine, you create a reverse port forward. We ssh into loud with the "-R 8022:127.0.0.1:22" parameter. This tells loud.radiotope.com to listen on port 8022, and forward it along to the client. Then, someone on loud can connect to the local port of 8022, which will forward along to us over ssh.

You can even connect two separate machines this way, however, some configuration changes must be made. On the server (loud.radiotope.com), the /etc/sshd_config file must be modified. Change the line:

#GatewayPorts no

to

GatewayPorts yes

Additionally, you must also specify "-g" on the ssh command line, which tells the server to allow remote connections to use the forwarded port.

Again, let's visualize:

Figure 4 - Remote port forward with gateway mode enabled

On server.radiotope.com, remote port forward to public.example.com, using this command:

ssh -N -g -R 8022:127.0.0.1:22 admin@public.example.com

Now, public.example.com is listening on port 8022 on all interfaces, and passing it through to server.radiotope.com on port 22. On the laptop in figure 4, you can access server.radiotope.com - even though it's not allowed by the firewall - simply by connecting to pulic.example.com:

ssh -p 8022 account@public.example.com

Note that the account used is an account on server.radiotope.com!

SOCKS Proxy

Certainly one of the lesser-known features of ssh is its ability to act as a SOCKS proxy. Rather than forward a specific port, or ports, SOCKS is a dynamic proxy, forwarding any ports requested. This doesn't solve every problem, though, as applications must be written with SOCKS proxy support. Most web browsers support SOCKS, as do many ftp clients.

To test this, open a web browser, I'm going to target Safari here, and go to http://www.whatismyip.com. Make a note of your ip address as it shows to an external web site. In a local shell, run this:

ssh -N -D 9999 account@public.example.com

...where public.example.com is a machine on a remote network that you have ssh access to. Once complete, open Safari's preferences and choose the advanced tab, or, go right for the Network Preference pane. If you're using a different web browser, adjust the proxy setting accordingly.

Figure 5 - SOCKS proxy configuration

Apply the changes and now, reload the whatismyip page. See? All requests look like they're coming from the remote machine. Note, though, that if you're counting on this for security or privacy, don't. For web browsing, cookies, Google analytics and others will foil you. Also, your DNS requests are not re-routed, so, those requests will still show up in your local or ISPs DNS server.

Passing Commands

If given a command, ssh will run that command on the remote host and return the output to you:

$ ssh loud.radiotope.com last
Password:
ladmin    ttyp9                     Thu Aug 16 07:39   still logged in
ladmin    ttyp9                     Thu Aug 16 07:39 - 07:39  (00:00)
ladmin    console  loud.radiotope.c Thu Aug 16 07:39 - 07:44  (00:04)
marczak   console  loud.radiotope.c Tue Aug 14 17:30 - 07:39 (1+14:09)

This is, naturally, incredibly handy for scripts, and a great way to collect remote data about hosts.

You can also run commands that require interaction, however, try it, and you may get some strange results, and a message like this:

ssh 127.0.0.1 vi .ssh/known-hosts
Password:
Vim: Warning: Output is not to a terminal
Vim: Warning: Input is not from a terminal

What's happening? Since ssh is just expecting to run the command and bail out, there's no real terminal allocated for the environment. Never fear, as you can tell ssh that you'd like a pseudo tty allocated for this very reason. Just use the "-t" flag. This is handy when using an editor remotely, using an IRC client from a remote machine, or finding that server with nethack on it!

One great trick is to gather your server list somewhere, and to run the same command on each. While this could be a full script, here's a one liner to see who's logged into 3 different servers using the w command:

servers="loud.radiotope.com 192.168.55.9 public.example.com"; for host in $servers; do ssh user@$host "w"; done

Of course, setting this up with password-less keys will really make this script useful.

Moving Data

With an ssh server running, there are several options for transferring data to and from that system. Two methods are distinct binaries that install alongside the ssh app itself.

The first is scp. The 'secure copy' application tunnels the copy across an ssh link. Its invocation resembles the standard cp command:

scp local_file account@public.example.com:path/remote_name

This copies 'local_file' to "account's" home directory, in the folder "path", naming the result "remote_name". To copy from a remote server, just reverse the source and destination:

scp account@public.example.com:some_file ./$1

This copies "some_file" from "account's" home directory on public.example.com into the current working directory, giving it the same name. Like cp, you can use the "-r" switch to recursively copy entire directories.

When installed with the sftp subsystem, an ssh server also offers an ftp-like interactive interface. Just use 'sftp' in place of 'ftp':

$ sftp 127.0.0.1
Connecting to 127.0.0.1...
Password:
sftp> ls
Applications     Desktop    Desktop DB    Desktop DF   Documents
Library    Movies    Music    Pictures    Public    Sites
memberd_dump.log    stop_vm_log.sh    test.jpg   test2.jpg 
sftp> get test2.jpg
Fetching /Users/marczak/test2.jpg to test2.jpg
/Users/marczak/test2.jpg      100% 38KB 38.0KB/s   00:00
sftp> quit

Anyone familiar with an ftp shell should be immediately comfortable with sftp.

scp and sftp are fine, fine utilities. However, we're OS X people, and we have certain needs. We like our data fork and resource fork to get transferred. We like our metadata on the complete side. Neither of which scp and sftp will do. Let's take advantage of pipes and redirection.

If we're local on an OS X box and want to copy data, ditto is the preferred method. Written specifically for OS X, we get to have our data fork and resource fork, too. We can extend this by taking ditto's output and sending it not to a disk-based file, but to standard out. Pipe this into ssh, which will happily read it and spit it out on the remote end. Let's start with a simple example:

cat data.txt | ssh -l user loud.radiotope.com '(cat > data2.txt)'

First, we send data.txt to standard out using cat. This is piped to ssh, which connects to the remote host loud.radiotope.com, which in turn takes the piped data, passes it over the encrypted connection, and passes it in to the subshell on standard in. We run commands that take standard in and write it to the file data2.txt. Let's extend on this:

ditto -c MyFolder -|ssh loud.radiotope.com ditto -x - ./MyFolder

The first part, before the pipe, is a standard ditto command. The "-c" switch tells ditto to create a cpio archive of the data it's receiving. The trailing hyphen ("-") tells ditto to send its output on standard out. From there, the data is piped to ssh, connecting to loud.radiotope.com, where the data is piped into ditto running on the remote computer. The "-x" switch on the remote tells ditto to expect cpio archive formatted data, followed by a single hyphen, forcing data to be read from standard in. Finally, we tell ditto on the remote what folder to put all of this data into.

The examples in this section are extremely powerful, and allow you to move massive amounts of data in a safe fashion. Combine these with some of the port forwarding techniques listed earlier, and you can reach hosts potentially not possible for you before.

Finally, it would be foolish of me not to mention rsync. While I covered ditto first due to its OS X-specific nature, rsync is a much more efficient application all around if you need data forks only. This turns out to cover the vast majority of data. Of course, I bring this up as rsync can transport over ssh without any fancy shell piping tricks. Just use the "-e" switch:

rsync -ave ssh --delete /path/to/data user@loud.radiotope.com:/path/to/destination/

Appropriate for anything not containing a resource fork or Mac-specific metadata that you wish to keep intact.

Other Apps

Finally, note that many GUI apps that offer a remote connection, such as Dreamweaver, TextWrangler or your favorite ftp application will offer to do so over ssh or sftp (ssh protected ftp), protecting your content, id and password.

ssh was designed for security - we can now see that. Like anything, though, your implementation must be well thought out. Here are some things to watch out for.

ssh can't protect you against weak passwords! The few times I've heard of an OS X machine getting hacked, it was because the owner had an account named "test" with a password of "test" enabled on the machine. Another popular one is "game" and "game". DON'T MAKE THIS MISTAKE! Choose strong passwords, don't allow passwords at all, or restrict the users that can access the machine via ssh.

The password issue also burns people setting up OS X Server. OS X Server enables root by default. It also enables sshd ("Remote Login"). Where does root's password come from? It comes from the initial admin account that is set up when you first install the server. So, if your initial admin password is weak, and decide to change it later, do not forget to change root's password as well! Past that, they're out of sync.

Also, for you network administrators: ssh should pretty muck make you sick. If you didn't get the dry-heaves reading the section on remote port forwarding, go read it again and think of the implications. Like anything, security is a balance. If you run any networks where guests are allowed access, think hard about allowing them outbound ssh.

There are options in sshd_config that will appeal to the security conscious. Check out the man page and implement appropriately. In particular, look at the following options: ignorerhosts, rhostsauthentication, rhostsrsaauthentication, rsaauthentication, passwordauthentication, permitemptypasswords and uselogin.

So, to answer the original question: is there anything that ssh can't do? Well, it won't make you julienne fries, however, you could certainly ssh into a fry maker and control it remotely! I had hoped to also cover a shell that makes ssh control over multiple machines a breeze, but this info seemed to be more than enough. The other info will have to wait for a future column. In the meantime, get some practice with the techniques described here. I'm sure you'll come to depend on them in short order!

Media of the month: Liam Finn's debut album, "I'll Be Lightning". I'm always keeping my ears open for something new, unique and fresh. Liam delivers. If you're an iTunes person, it's currently only available on the Australian iTunes store. CD hits America this January.

Speaking of January, Macworld will be upon us again. Despite the seeming feeling that we're always making plans for some Mac show...I hope you're making your plans for Macworld, and to see you there.

Ed Marczak owns and operates Radiotope, a technology consulting company that focuses on networking, server automation and business process management. He believes that small businesses can use the same tools afforded to enterprise-class customers. How can this help? http://www.radiotope.com