TweetFollow Us on Twitter

mac in the Shell: More ftp

Volume Number: 23 (2007)
Issue Number: 09
Column Tag: Mac in the shell

More ftp

I've got a fever, and the only prescription is more ftp

by Edward Marczak

Introduction

ftp – whether we're referring to the actual protocol, or just file transfer in general – is something we all need on some basis. When I started toying with computers, I saw punch cards, but never really had to deal with them. But that was only one method of file (or more generally, data) transfer. Then we moved up to tape and floppy disk. Now, very few computers are not connected to a network of some type, and the primary reason is to transfer data in the form of files. Our cover story this month touches on several GUI-based clients, but when you read this column, those utilities get the "Mac In The Shell" treatment. We need to be able to transfer files easily from a shell!

Why?

I often create automated solutions that run on a server without a GUI. There are also plenty of times when a simple, repeated file transfer shouldn't pop up anything visually on a client machine, either. It should 'just happen' simply and reliably with no pomp and circumstance. Enter curl, ftp and wget.

Of the three, "ftp" is the oldest and most simple. wget brings further power, and curl is a veritable Swiss Army Knife of transfer agents. If one of these options can't do what you want, it's most likely not possible (or, consider a different tactic!).

ftp

ftp, the application, implements a client side version of the ftp protocol (which is detailed by Mary Norbury in this month's cover story, "FTP Clients for Mac OS X"). In simple use, you can use it interactively:

$ ftp ftp.example.com
Trying ftp.example.com...
Connected to ftp.example.com.
220 example.com FTP server ready.
Name (ftp.example.com:marczak): 
331 Password required for marczak.
Password:
230 User marczak logged in.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> cd Public
250 CWD command successful.
ftp> ls
229 Entering Extended Passive Mode (|||50077|)
150 Opening ASCII mode data connection for '/bin/ls'.
total 1
-rw-r--r--  1 marczak  marczak      0 Nov 16  2006 .localized
drwx-wx-wx  3 marczak  marczak    102 Nov 16  2006 Drop Box
-rw-r--r--  1 marczak  marczak  43796 Jul 20 07:14 test.jpg
226 Transfer complete.
ftp> bin
200 Type set to I.
ftp> get test.jpg
local: test.jpg remote: test.jpg
229 Entering Extended Passive Mode (|||50079|)
150 Opening BINARY mode data connection for 'test.jpg' (43796 bytes).
100% |********************| 43796     121.76 MB/s    00:00    
226 Transfer complete.
43796 bytes received in 00:00 (21.79 MB/s)
ftp> quit

Any techy person over the age of 25 should recognize this immediately. They should also remember in the days before big-business-on-the-Internet that it was polite to wait until "after hours" before using ftp against a University server! In the annals of tech-history, though, one needed to be familiar with transferring files this way.

The shell-based ftp application has a good lexicon in its interpreter. It's one that has grown substantially since its inception. However, for purposes of automation, that can get clumsy. You could script it with expect. Some versions of ftp allow creating a script and having ftp simply run through the motions that the script indicates. However, the version of ftp that ships with OS X (at least in Tiger) omits this option. It does keep the macro definition option in, though.

Never fear! The parameters available to you are greatly expanded, including passing a user name and password along. This is ideal for scripting within your own scripting environment. So, if I know in advance the names of the files I need to transfer, I could script this in bash thusly:

ftp -V ftp://user:password@server.example.com/directory/file.txt

...which will download file.txt and name the local file "file.txt". Note the -V switch, which is the opposite of -v – keeping the output quiet. I can also go the other way using:

ftp -V -u ftp://user:password@server.example.com/directory/ file.txt file2.txt 

...which will upload the specified file(s) – in this case, file.txt and file2.txt – to the given directory. Don't forget the trailing slash on the target directory!

Don't miss the fact that any valid URL syntax will work, so, you can 'ftp' a file from an HTTP server, too:

$ ftp http://www.example.com/directory/cars.jpg
Requesting http://www.example.com/directory/cars.jpg
 17746      29.87 KB/s

So, good 'ol ftp provides us with some quick and easy ways to move files around. Not ideal ways, perhaps, as not only is our password sent in the clear as part of the ftp protocol (which may not be an issue for you), but also displays our password in a process listing. That's not really cool.

ftp does offer many more options, so, check the man page if you need to get in deeper.

wget

wget bills itself as the "non-interactive network downloader." So, unlike ftp, there is no interactive mode with which you can use to generally poke around. However, we're here to talk about automated use, so, we don't need no stinkin interactive mode! If you were desperate, though, you could use one of wget's more interesting features: when an ftp directory is requested, it will automatically convert the output into an html listing. That might be a little too esoteric...even for me!

Disapointingly, wget is not installed by default under OS X Tiger. However, it's simple to install one way or another. You can grab the source from the GNU page at http://ftp.gnu.org/pub/gnu/wget/ and compile it yourself. Quentin Stafford-Fraser has a pre-compiled binary here: http://www.statusq.org/images/wget.zip. Finally, you can install wget using fink or MacPorts.

To get right down to it, like ftp, you can use any rational URL to specify your target:

$ wget ftp://emarczak:sekretpass@ftp.example.com/path/Big_File.zip
--07:23:08--  ftp://emarczak:*password*@ftp.example.com/path/Big_File.zip
           => `Big_File.zip'
Resolving ftp.example.com... 192.168.77.201
Connecting to ftp.example.com|192.168.77.201|:21... connected.
Logging in as emarczak ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /path ... done.
==> PASV ... done.    ==> RETR Big_File.zip ... done.
Length: 10,871,922 (10M) (unauthoritative)
100%[=======================>] 10,871,922   638.53K/s    ETA 00:00
07:23:27 (601.74 KB/s) - `Big_File.zip' saved [10871922]

Also like ftp, wget will display your password in a process listing, so, use this with care! Here's where the roads diverge, though, and wget has a few more tricks up its sleeve. You can recursively download an entire ftp or http directory with the "-r" switch:

wget -r -t 5 ftp://emarczak:sekretpass@ftp.example.com/path/

I also threw in the "-t" switch, which will allows for multiple retries if some part of a file download fails. "-t" also allows for a value of "inf" causing infinite retries. Also useful here is the "-l" (ell) switch, which limits the depth of the traversal. So, to grab just the items from the top layer of the directory you specify, use "-l1".

An absolute life saver is the "-c" switch: it tells wget to continue a partial download. So, if your download bombs, or, perhaps you're on a laptop and need to run before the transfer is complete, retry the operation with the "-c" switch and pick up right from where you left off. Nice.

Wget will read a list of URLs from a file, using the "-i" switch. This is handy for scripting, of course. However, it's also a very nice way to keep your password out of a process listing. With your username and password embedded in a file, you're not using it on the command line. This list also comes in handy as a way to store your favorite sites and then recursively them locally using the "-r" switch mentioned above. In fact, toss in the "-A" switch, which will only accept certain files, and you can download only files of a certain type from a site. Next time you want all of the mov files from a given site, try this:

wget -r -l4 http://www.example.com/movies -A.mov -np

This will mirror the given site, and only transfer files ending in ".mov" on the given pages up to 4 levels deep. We also ensure that we don't follow links back up to the parent directory ("-np", or, "--no-parent").

Again, wget has many, many tricks up its sleeve. Too many to list here, but the brief introduction should convince you of its utility above the standard "ftp" application. Check out the man page for much more. (Specifically, check out the "-k" switch!).

curl

Like the other utilities mentioned here, curl will accept any valid URL as its file description. Unlike wget, curl is installed as a part of OS X. One very cool curl trick is that it dumps files to standard out unless it's told where to write them. Why is that cool?

Sometimes, you just want to view a remote document, be it an actual file – like a README or index.html file – or a directory listing. So, you could easily:

curl http://server2.example.com/instructions/how_to_do_it.txt | less

...which will get the file from the server and pipe it into less. When you quit less, there will be no file remaining to clutter up your disk. I sometimes use that with http://www.whatismyip.com, and then pipe the output to a script that simply reports back the machine's external IP address. This is also a cool way to run a remote script:

curl ftp://server.example.local/script.sh | bash

If you are interested in downloading a file, use the "-O" switch (capital O), which names the local file the same as the remote:

curl -O ftp://ftp.example.com/path/to/file/some_file.zip

This will anonymously download some_file.zip, and store it in the current working directory as some_file.zip. I also particularly like the "-L" switch (capital ell) when used with http servers as this will make curl follow http redirects.

Of course, curl will upload files, too. The "-T" switch will take care of this for you:

curl -T "pix[1-100].jpg" ftp://ftp.example.com/pictures/

I also threw in the fact that curl will respect globbing and regular expressions. So, the previous example will upload pix1.jpg, pix2.jpg...up through pix100.jpg. Clearly very handy.

Both upload and download can be resumed using the "-C -" switch (capital C followed by a hyphen). The hyphen tells curl to figure out where to resume from automatically. This does require server-side support in the form of telling the server at which byte to start appending at (the SIZE command for upload) or which byte to start the transfer from (ftp resume or HTTP 1.1 for downloads).

If you're an all-OS X shop, you'll be happy to hear that curl supports Kerberos. You can get your initial ticket the usual way (kinit), and then tell curl to use that authentication via the "--krb" switch.

curl --krb private ftp://krb4site.com -u username:boguspw

If this uses Kerberos, why did I still supply a name and password? This is a bit of a hack, but with no password, curl will still want to prompt you for one. However, if you supply one, but use Kerberos, it'll just ignore the password you supply – so use a bogus password as this will appear in process listings.

Conclusion

Being able to script data transfer is an important part of every system administrator's toolkit. While good 'old ftp will do the job in many cases, wget and curl give you much more flexibility. Both utilities have overlap in functionality, but curl goes deeper in many cases. Case in point: when I said that curl accepts any valid URL syntax, try TELNET://, dict:// and even LDAP:// (although, you'll currently need to build your own curl for LDAP support as the Apple supplied version isn't linked correctly with the LDAP framework).

Of course, there are other file transfer options available to you, including scp, sftp, ditto and rsync, to name a few. However, I focused mainly on ftp options here, as ftp is alive and well, but sometimes overlooked. While perhaps a deceiving name, sftp is not true ftp, but file transfer over ssh, requiring no ftp server at all. Of course, over "hostile" networks, you should use no less that an encrypted solution. However, with the right internal setup, and in certain other cases, ftp can be the perfect solution.

Media of the month: the ftp RFC: http://www.faqs.org/rfcs/rfc959.html. If you want to get deeper into ftp and understand why it behaves the way it does, this RFC is the way to go.

Please practice this in a test environment and then press it into real-world use where appropriate. Until next month, I think you'll find this a great tool in your automation arsenal.


Ed Marczak owns and operates Radiotope, a technology consulting company that guides companies to use what they have as efficiently as possible. He is also the Executive Editor of MacTech Magazine, a husband and father of two. His spare time is spent editing MacTech Magazine and enjoying his family. He finds keeping it all running smoothly good practice. Improve your practice at http://www.radiotope.com.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

FotoMagico 5.6.12 - Powerful slideshow c...
FotoMagico lets you create professional slideshows from your photos and music with just a few, simple mouse clicks. It sports a very clean and intuitive yet powerful user interface. High image... Read more
OmniGraffle Pro 7.12.1 - Create diagrams...
OmniGraffle Pro helps you draw beautiful diagrams, family trees, flow charts, org charts, layouts, and (mathematically speaking) any other directed or non-directed graphs. We've had people use... Read more
beaTunes 5.2.1 - Organize your music col...
beaTunes is a full-featured music player and organizational tool for music collections. How well organized is your music library? Are your artists always spelled the same way? Any R.E.M. vs REM?... Read more
HandBrake 1.3.0 - Versatile video encode...
HandBrake is a tool for converting video from nearly any format to a selection of modern, widely supported codecs. Features Supported Sources VIDEO_TS folder, DVD image or real DVD (unencrypted... Read more
Macs Fan Control 1.5.1.6 - Monitor and c...
Macs Fan Control allows you to monitor and control almost any aspect of your computer's fans, with support for controlling fan speed, temperature sensors pane, menu-bar icon, and autostart with... Read more
TunnelBear 3.9.3 - Subscription-based pr...
TunnelBear is a subscription-based virtual private network (VPN) service and companion app, enabling you to browse the internet privately and securely. Features Browse privately - Secure your data... Read more
calibre 4.3.0 - Complete e-book library...
Calibre is a complete e-book library manager. Organize your collection, convert your books to multiple formats, and sync with all of your devices. Let Calibre be your multi-tasking digital librarian... Read more
Lyn 1.13 - Lightweight image browser and...
Lyn is a fast, lightweight image browser and viewer designed for photographers, graphic artists, and Web designers. Featuring an extremely versatile and aesthetically pleasing interface, it delivers... Read more
Visual Studio Code 1.40.0 - Cross-platfo...
Visual Studio Code provides developers with a new choice of developer tool that combines the simplicity and streamlined experience of a code editor with the best of what developers need for their... Read more
OmniGraffle 7.12.1 - Create diagrams, fl...
OmniGraffle helps you draw beautiful diagrams, family trees, flow charts, org charts, layouts, and (mathematically speaking) any other directed or non-directed graphs. We've had people use Graffle to... Read more

Latest Forum Discussions

See All

The House of Da Vinci 2 gets a new gamep...
The House of Da Vinci launched all the way back in 2017. Now, developer Blue Brain Games is gearing up to deliver a second dose of The Room-inspired puzzling. Some fresh details have now emerged, alongside the game's first official trailer. [Read... | Read more »
Shoot 'em up action awaits in Battl...
BattleBrew Productions has just introduced another entry into its award winning, barrelpunk inspired, BattleSky Brigade series. Whilst its previous title BattleSky Brigade TapTap provided fans with idle town building gameplay, this time the... | Read more »
Arcade classic R-Type Dimensions EX blas...
If you're a long time fan of shmups and have been looking for something to play lately, Tozai Games may have just released an ideal game for you on iOS. R-Type Dimensions EX brings the first R-Type and its sequel to iOS devices. [Read more] | Read more »
Intense VR first-person shooter Colonicl...
Our latest VR obsession is Colonicle, an intense VR FPS, recently released on Oculus and Google Play, courtesy of From Fake Eyes and Goboogie Games. It's a pulse-pounding multiplayer shooter which should appeal to genre fanatics and newcomers alike... | Read more »
PUBG Mobile's incoming update bring...
PUGB Mobile's newest Royale Pass season they're calling Fury of the Wasteland arrives tomorrow and with it comes a fair chunk of new content to the game. We'll be seeing a new map, weapon and even a companion system. [Read more] | Read more »
PSA: Download Bastion for free, but wait...
There hasn’t been much news from Supergiant Games on mobile lately regarding new games, but there’s something going on with their first game. Bastion released on the App Store in 2012, and back then it was published by Warner Bros. This Warner... | Read more »
Apple Arcade: Ranked - 51+ [Updated 11.5...
This is Part 2 of our Apple Arcade Ranking list. To see part 1, go here. 51. Patterned [Read more] | Read more »
NABOKI is a blissful puzzler from acclai...
Acclaimed developer Rainbow Train's latest game, NABOKI, is set to launch for iOS, Android, and Steam on November 13th. It's a blissful puzzler all about taking levels apart in interesting, inventive ways. [Read more] | Read more »
A Case of Distrust is a narrative-driven...
A Case of Distrust a narrative-focused mystery game that's set in the roaring 20s. In it, you play as a detective with one of the most private eye sounding names ever – Phyllis Cadence Malone. You'll follow her journey in San Francisco as she... | Read more »
Brown Dust’s October update offers playe...
October is turning out to be a productive month for the Neowiz team, and a fantastic month to be a Brown Dust player. First, there was a crossover event with the popular manga That Time I Got Reincarnated as a Slime. Then, there was the addition of... | Read more »

Price Scanner via MacPrices.net

Score a 37% discount on Apple Smart Keyboards...
Amazon has Apple Smart Keyboards for current-generation 10″ iPad Airs and previous-generation 10″ iPad Pros on sale today for $99.99 shipped. That’s a 37% discount over Apple’s regular MSRP of $159... Read more
Apple has refurbished 2019 13″ 1.4GHz MacBook...
Apple has a full line of Certified Refurbished 2019 13″ 1.4GHz 4-Core Touch Bar MacBook Pros available starting at $1099 and up to $230 off MSRP. Apple’s one-year warranty is included, shipping is... Read more
2019 13″ 1.4GHz 4-Core MacBook Pros on sale f...
Amazon has new 2019 13″ 1.4GHz 4-Core Touch Bar MacBook Pros on sale for $150-$200 off Apple’s MSRP. These are the same MacBook Pros sold by Apple in its retail and online stores: – 2019 13″ 1.4GHz/... Read more
11″ 64GB Gray WiFi iPad Pro on sale for $674,...
Amazon has the 11″ 64GB Gray WiFi iPad Pro on sale today for $674 shipped. Their price is $125 off MSRP for this iPad, and it’s the lowest price available for the 64GB model from any Apple reseller. Read more
2019 15″ MacBook Pros available for up to $42...
Apple has a full line of 2019 15″ 6-Core and 8-Core Touch Bar MacBook Pros, Certified Refurbished, available for up to $420 off the cost of new models. Each model features a new outer case, shipping... Read more
2019 15″ MacBook Pros on sale this week for $...
Apple resellers B&H Photo and Amazon are offering the new 2019 15″ MacBook Pros for up to $300 off Apple’s MSRP including free shipping. These are the same MacBook Pros sold by Apple in its... Read more
Sunday Sale: AirPods with Wireless Charging C...
B&H Photo has Apple AirPods with Wireless Charging Case on sale for $159.99 through 11:59pm ET on November 11th. Their price is $40 off Apple’s MSRP, and it’s the lowest price available for these... Read more
Details of Sams Club November 9th one day App...
Through midnight Saturday night (November 9th), Sams Club online has several Apple products on sale as part of their One Day sales event. Choose free shipping or free local store pickup (if available... Read more
Sprint is offering the 64GB Apple iPhone 11 f...
Sprint has the new 64GB iPhone 11 available for $15 per month for new lines. That’s about 50% off their standard monthly lease of $29.17. Over is valid until November 24, 2019. The fine print: “Lease... Read more
New Sprint November iPhone deal: Lease one iP...
Switch to Sprint and purchase an Apple iPhone 11, 11 Pro, or 11 Pro Max, and get a second 64GB iPhone 11 for free. Requires 2 new lines or 1 upgrade-eligible line and 1 new line. Offer is valid from... Read more

Jobs Board

*Apple* Mobility Pro - Best Buy (United Stat...
**746087BR** **Job Title:** Apple Mobility Pro **Job Category:** Store Associates **Store NUmber or Department:** 000319-Harlem & Irving-Store **Job Description:** Read more
Best Buy *Apple* Computing Master - Best Bu...
**743392BR** **Job Title:** Best Buy Apple Computing Master **Job Category:** Store Associates **Store NUmber or Department:** 001171-Southglenn-Store **Job Read more
Best Buy *Apple* Computing Master - Best Bu...
**746015BR** **Job Title:** Best Buy Apple Computing Master **Job Category:** Sales **Store NUmber or Department:** 000372-Federal Way-Store **Job Description:** Read more
*Apple* Mobility Pro - Best Buy (United Stat...
**744658BR** **Job Title:** Apple Mobility Pro **Job Category:** Store Associates **Store NUmber or Department:** 000586-South Hills-Store **Job Description:** At Read more
Best Buy *Apple* Computing Master - Best Bu...
**741552BR** **Job Title:** Best Buy Apple Computing Master **Job Category:** Sales **Store NUmber or Department:** 000277-Metcalf-Store **Job Description:** **What Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.