TweetFollow Us on Twitter

Mac In The Shell: More from the Unknown

Volume Number: 23 (2007)
Issue Number: 07
Column Tag: Mac In The Shell

More from the Unknown

More shell tools that typically go unnoticed

by Edward Marczak

Introduction

OS X has an incredible breadth and depth available in the tools accessible from a shell. So much so, that it's hard to know each and every tool available. Some are almost hidden: tucked away in places one does not typically look. Others are hidden in plain sight: exactly where you'd expect them, but buried among the volume. This month, I'm going to dig out and introduce some more useful shell tools that you may not know about.

text manipulation

It's long been the domain of scripting languages like perl, and perhaps now to a lesser extent PHP, to slice, dice and otherwise manipulate text. However, there's a framework built right into OS X, so why not take advantage of it? The Cocoa text system is exposed to the shell via the "textutil" application. "textutil" can convert and manipulate many different formats of text documents. In fact, it works with a surprising number formats:

   txt
   html
   Microsoft Word "doc"
   Microsoft Word XML
   rtf
   webarchive

It's also incredibly easy to get started. Using the "-convert" switch, you can accomplish most conversions that you'd ever want:

textutil -convert html resume.doc

This will convert the Microsoft Word-based "resume.doc" into a separate HTML document named "resume.html". I'll immediately point out that textutil writes much, much better HTML than Word.

Note that the input format is determined 'automagically' by a number of methods. If textutil misidentifies the format of the source file, you tell it with the "-format" switch:

textutil -format html -convert rtf file1

This will tell textutil that "file1" is an HTML file, and we want to convert it into a rich text format file named "file1.rtf".

There's also a very handy "-info" switch that returns info on a given file:

$ textutil -info 2007-01-On\ Logs.doc
File:  2007-01-Marczak On Logs.doc
Type:  Word format
Size:  25600 bytes
Length:  3746 characters
Title:  MacTech Template
Author:  Classics Department
Last Editor:  Edward Marczak
Company:  Radiotope
Subject:  
Keywords:  
Created:  2006-11-02 17:56:00 -0500
Last Modified:  2006-11-06 09:47:00 -0500

As you can see, documents sometimes have incorrect metadata. Well, textutil can help you handle that as well! There are a number of metadata related switches:

strip: Remove metadata in target during conversion.

title: Set the title metadata attribute for output files.

author: Set the author metadata attribute for output files.

subject: Specify the subject metadata attribute for output files.

keywords: a shell-quoted list of 'tags' ("keyword1,keyword2, ...")

comment: Set the comment attribute for the output files.

editor: val Set the editor attribute for output files.

company: val Set the company attribute for output files.

creationtime: Set the creation time attribute for output files (in "yyyy-mm-ddThh:mm:ssZ" format).

modificationtime: Set the modification time attribute for output files (in "yyyy-mm-ddThh:mm:ssZ" format).

So, we could create an HTML document from a plain text file with the title and keywords set by running textutil like this:

textutil -convert html -title "Anti-gravity Thesis" -keywords "(anti,gravity,'atom power',physics)" -author "Bruce Banner" ag-file.txt

You'll find a well-formed HTML file with a little bit of CSS. Of course, there are many times when you need finer control over the tags that will be used in that HTML. Have no fear! The "-excludedelements" switch can help you out. This switch allows you to specify which HTML elements should not be used in the generated HTML. Like the "-keywords" switch, "-excludedelements" expects a single argument, so you can use a shell-quoted list of tags to skip.

Keep it clean

Speaking of well-formed HTML, manipulating tags and their parameters is a special case of text manipulation. Fortunately, tidy, the standard Unix utility just for this purpose, ships with OS X. So, if you've used textutil to generate HTML based output, tidy can take you even further.

The simplest case lets tidy modify your HTML document in place. To do so, use the "-m" switch:

$ tidy -im ag-file.html
Info: Doctype given is "-//W3C//DTD HTML 4.01//EN"
Info: Document content looks like HTML 4.01 Strict
No warnings or errors were found.

Take look at your file before and after running the tidy command. I also like to include the "-i" switch, which ensures tag levels are indented appropriately. Also, by default, tidy will lowercase all tags (which, by me, is "correct"). This can be overridden with the "-u" switch.

One of my absolute favorite uses of tidy is to get rid of "illegal" characters. This is easily accomplished with the "-b" (bare) switch. This strips the document of smart quotes, and dashes, and other characters that cause problems in HTML documents.

There's much, much more that tidy can do. Check the very well written man page to look at more options. Don't forget, though, that you can combine textutil and tidy into one neat shell statement:

textutil -stdout ag-file.txt -convert html | tidy -i -output ag-file.html

This command uses textutil to convert a text file to HTML, and sends the output to stdout. Then, that output is piped to tidy, which then requires us to use the "-output" switch to write the file to disk.

Finally, realize that both textutil and tidy can be used with a wildcard character to process an entire group of files. If all files are in a single directory, simply change to that directory and:

textutil -convert html *.doc

This will convert all Word documents in the current directory into HTML files. To process the current folder and all subfolders, use the find utility:

find . -name \*txt -print0 | xargs -0 textutil -convert html

I opted for xargs over find's exec command to ensure that large directories can be processed. You can scrub Word metadata in-place using this technique:

textutil -strip -convert doc *.doc

Painless, right?

Image Manipulation

While slinging text around may be a common, and very specific operation, doing the same to images requires a different set of tools. OS X's scriptable image processor server or, "sips" is just the ticket.

Like textutil, sips leans on frameworks already built into the core of OS X. This, of course, lets you do all of the great things that you can with a GUI right in a shell! Why is this exciting? Automation, naturally.

In the past, I've put together both nightly reports and 'one-click-builds' of documents using both user submitted and random images. The trick is not so much the content, but the dimensions of the graphic: you need consistent dimensions so you can automate your layout. Resizing graphic files is one of the things that sips does best:

sips -Z 100x100 IMG_1312.JPG --out image1-sized.jpg

The "-Z" switch is really nice: it keeps the proportions of the original image, but makes sure neither the width or height exceed the dimensions specified. The "--out" switch writes the result to a new file. The image you just made doesn't have a nice preview icon in the Finder? Add one:

sips -i image1-sized.jpg

Oh, you didn't want a JPEG format file? Convert it at the same time using the "-s" (setProperty) switch (let's do it all in one shot):

sips -Z 100x100 IMG_1312.JPG -i -s format png --out image-sized.png

This one command resizes the graphic, converts the output to png and creates the Finder preview icon. Nice.

For automation, though, you often need to find out information about the source before you blindly process it. Again, like textutil, sips has functions for this. You can dump all of an image's data with the "-g all" switch and parse the output yourself:

$ sips -g all IMG_1312.JPG 
/Users/erm/Pictures/Parade/IMG_1312.JPG
  pixelWidth: 2592
  pixelHeight: 1944
  typeIdentifier: public.jpeg
  format: jpeg
  formatOptions: default
  dpiWidth: 180.000
  dpiHeight: 180.000
  samplesPerPixel: 3
  bitsPerSample: 8
  hasAlpha: no
  space: RGB
  profile: Camera RGB Profile
  creation: 2007:04:04 17:23:31
  make: Canon
  model: Canon PowerShot S2 IS

...or, you can request one or more of these parameters directly:

$ sips -g pixelWidth -g pixelHeight IMG_1312.JPG
/Users/marczak/Pictures/Parade/IMG_1312.JPG
  pixelWidth: 2592
  pixelHeight: 1944

Like textutil, feel free to throw a wildcard at sips, or, use the find/xargs trick.

Why?

So, why go through this trouble, when you can fire up Photoshop and make your changes (and even script it!), or use a Word processor to mold your words? Again, automation comes to mind. Specifically, automation with low overhead. Using the shell tools, you can run scripts on a server without having a GUI or installing an application like Photoshop at all. Combine this with some of the techniques I presented in April, and you can achieve some incredibly complex workflows.

Even cooler: more and more GUI utilities have some way to run shell jobs. Textmate, for example, can run a selection or entire document through a shell tool and plop the results right back in the live document. Even apps that don't have this functionality built in can usually be faked thanks to AppleScript. Take Mail.app, for instance. It lets you create signatures, but not dynamic signatures that rely on a running process. What if you wanted to include a snapshot from your iSight in your sig? Script a capture from isightcapture (http://www.intergalactic.de/hacks.html), run it through sips and AppleScript the copy and pasting of into Mail.

Or, consider a simpler alternative: you've installed fortune and want a random fortune in your sig. A simple bash script could be as follows:

#!/bin/bash
# Tell Mail.app to select all and copy to clipboard
osascript <<MailCopy
tell application "System Events"
        tell application "Mail" to activate
        keystroke "a" using {command down}
        keystroke "c" using {command down}
end tell
MailCopy
# Write clipboard to file
pbpaste > /tmp/mailpaste.tmp
fortune -s >> /tmp/mailpaste.tmp
textutil -stdout -convert rtf /tmp/mailpaste.tmp | pbcopy
# Tell Mail.app to activate and paste in the contents of the clipboard
osascript <<MailPaste
tell application "System Events"
        tell application "Mail" to activate
        keystroke "a" using {command down}
        keystroke "v" using {command down}
end tell
MailPaste

This is admittedly a quick hack with absolutely no error checking or other niceties. However, if you open up Mail.app, create a new message, put your cursor anywhere in the body and run this, you'll get a fortune tagged onto the end of your message (it helps if you already have a sig created). Pretty it up a bit and you could run it via the GUI via a ".command" file, or, wrap it in AppleScript and run it from your script menulet. Or, if you're a Quicksilver user, don't ignore the qs shell tool.

Of course, the real power lies in integrating other data: from the web, from Excel, from a database....wherever!

Conclusion

When venerable shell tools are integrated with the power of OS X, the workflow possibilities truly are endless. Be creative in your application of scripting tools, and you'll be able to solve any problem for your client/employer/self.

Media of the month: Brendan Benson's new album, "Alternative to Love". It's just good, fun pop music, no strings attached.

Until next month, keep thinking of ways to keep the work flowing.

References:

"Developer Release Note": http://developer.apple.com/releasenotes/Cocoa/AppKit.html


Ed Marczak owns and operates Radiotope, a technology consultancy that just tries to make life easier through the technology that already exists. He's also the Executive Editor of MacTech Magazine, and CTO of WheresSpot. His copious free time is spent with his wife and two daughters, currently enjoying the weather in the North East U.S.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

WiFiSpoof 3.5.4 - Change your WiFi MAC a...
WiFiSpoof quickly and easily allows you to change your WiFi MAC address via hot-key or the system menu bar. Version 3.5.4: Dutch localization Download Now]]> Read more
Sparkle Pro 3.0.7 - Visual website creat...
Sparkle Pro will change your mind if you thought building websites wasn't for you. Sparkle is the intuitive site builder that lets you create sites for your online portfolio, team or band pages, or... Read more
MySQL 8.0.23 - Industry-leading open-sou...
MySQL, the industry-leading open-source SQL database, is an accessible, easy-to-use relational database management system (RDBMS). As an alternative to Oracle and SQL server, MySQL offers features... Read more
OmniGraffle 7.18.2 - Create diagrams, fl...
OmniGraffle helps you draw beautiful diagrams, family trees, flow charts, org charts, layouts, and (mathematically speaking) any other directed or non-directed graphs. We've had people use Graffle to... Read more
OmniGraffle Pro 7.18.2 - Create diagrams...
OmniGraffle Pro helps you draw beautiful diagrams, family trees, flow charts, org charts, layouts, and (mathematically speaking) any other directed or non-directed graphs. We've had people use... Read more
Malwarebytes 4.7.9.3978 - Adware removal...
Malwarebytes (was AdwareMedic) helps you get your Mac experience back. Malwarebytes scans for and removes code that degrades system performance or attacks your system. Making your Mac once again your... Read more
OmniFocus 3.11.2 - GTD task manager with...
OmniFocus is an organizer app. It uses projects to organize tasks naturally, and then add tags to organize across projects. Easily enter tasks when you’re on the go, and process them when you have... Read more
Dropbox 113.4.507 - Cloud backup and syn...
Dropbox for Mac is a file hosting service that provides cloud storage, file synchronization, personal cloud, and client software. It is a modern workspace that allows you to get to all of your files... Read more
Duet 2.3.3.0 - Use your iPad as an exter...
Duet is the first app that allows you to use your iDevice as an extra display for your Mac using the Lightning or 30-pin cable. Note: This app requires a $9.99 iOS companion app. Version 2.3.3.0:... Read more
f.lux 41.1 - Adjusts the color of your d...
f.lux makes the color of your computer's display adapt to the time of day, warm at night and like sunlight during the day. Ever notice how people texting at night have that eerie blue glow? Or wake... Read more

Latest Forum Discussions

See All

Cookie Run: Kingdom is a base-building R...
Today, developer Devsisters has launched the latest title in their Cookie Run series. It's called Cookie Run: Kingdom and is a base-building RPG where you'll do battle against other players once you've created your delicious realms. [Read more] | Read more »
Here's what SpongeBob SquarePants:...
SpongeBob SquarePants: Battle for Bikini Bottom - Rehydrated came out this week on iOS, and it seems like a pretty solid platformer with a Banjo-Kazooie-style collection-based progression system. You play as SpongeBob, obviously, as he tries to... | Read more »
Pokemon Go will celebrate the arrival of...
Niantic has recently announced that Mega Ampharos, the more fabulous looking version of the Light Pokemon, is now available in the game. To commemorate the occasion, an Incense Day event featuring a host of different Electric and Dragon-Type... | Read more »
Nuts is a squirrel adventure game coming...
Noodlecake’s Nuts is a lo-fi photography game about surveying and documenting the secret lives of squirrels. It’s coming to Apple Arcade this Friday, 22nd January, with a further release coming later on PC and Nintendo Switch. [Read more] | Read more »
Super JetPak DX’s physical preorders sti...
The physical cartridge edition of Pocket Pixel Design’s Super JetPak DX is still available to preorder and will be until 31st January, with the game launching later this year on the Nintendo Game Boy. [Read more] | Read more »
Honkai Impact 3rd has revealed more deta...
Honkai Impact 3rd has now announced more details about its upcoming crossover with Neon Genesis Evangelion. This is set to arrive next week on January 22nd and will introduce an exclusive story, battlesuit and weapon to the popular action-RPG. [... | Read more »
Space Frog Intern, James Bolton's a...
James Bolton's arcade shooter Space Frog Intern is now available for Android devices following its release on iOS last year. It sees players taking control of the titular amphibian and blasting their way through various space beasties. [Read more... | Read more »
The 5 Best Mobile Ports
Ports or coversions of games from one platform to another can be quite tricky. This is especially true when porting games to phones and tablets, as these titles have to be designed with workable touch controls and account for smaller screen sizes... | Read more »
Erica, the PS4-exclusive FMV thriller, c...
London-based studio Flavourworks is bringing its FMV game Erica to iOS on Friday as a free trial, with a full release presumably following later. This is the second platform the game has launched on after its original release on PlayStation 4 two... | Read more »
Cute simulation game Dinosaur Park: Prim...
Dinosaur Park: Primeval Zoo is a cute zoo simulation game from upjers, and it sounds perfect for anyone who’s a dinosaur fan. It’s coming to Android in February, and is available right now for pre-registration. [Read more] | Read more »

Price Scanner via MacPrices.net

Apple restocks $599 4-Core Mac minis, Certifi...
Apple has restocked Certified Refurbished Intel-based 4-Core Mac minis for only $599 or $200 off original MSRP. Each mini comes with a new outer case plus a standard Apple one-year warranty. Shipping... Read more
Apple Magic Keyboard for 11″ iPad Pro/10.9″ i...
Amazon has the Apple Magic Keyboard for 11″ iPad Pro (2nd gen)/10.9″ iPad Air (4th gen) on sale today for only $199 including free shipping. Their price is $100 off Apple’s MSRP for this Magic... Read more
Verizon offers $400 off on iPhone 11 Pro mode...
Verizon is offering $400 off on iPhone 11 Pro models through January 24, 2021 when purchased with a new line of Unlimited service. Price start at $999.99 before the discount. Price is prorated... Read more
Flash sale! Apple’s 21″ 2.3GHz Dual-Core iMac...
Adorama has the 21″ 2.3GHz Dual-Core iMac on sale for a very limited time — today and early tomorrow — for $988 shipped. That’s $111 off Apple’s MSRP, and it’s the cheapest price available for a new... Read more
Boost Mobile continues to offer no contract A...
Boost Mobile is offering 2020 Apple iPhone SE models for $150 off MSRP with service. Their discount reduces the cost of an iPhone SE/64GB to $249.99. 128GB models are offered at a $50 discount: $399.... Read more
B&H launches Apple iMac sale with 21″ and...
B&H Photo has Apple 21″ and 27″ iMacs on sale today for up to $150 off MSRP with prices starting at $1049. Shipping is free. These are the same iMacs sold by Apple in their retail and online... Read more
Select 11″ WiFi iPad Pros on sale today for $...
B&H Photo has select 2020 Apple 11″ WiFi iPad Pros on sale today for $50 off Apple’s MSRP, each including free expedited shipping to most addresses in the US: – 11″ 256GB WiFi iPad Pro: $849 $50... Read more
Clearance 8-core iMac Pro available again for...
Apple has Certified Refurbished, clearance, 27″ 3.2GHz 8-Core iMac Pros available $3819 including free shipping. Their price is $1180 off the original MSRP of new models. A standard Apple one-year... Read more
B&H launches 16″ MacBook Pro sale with mo...
B&H has Apple’s 16″ MacBook Pros on sale for $200-$250 off Apple’s MSRP with prices starting at $2199. Expedited shipping is free to many addresses in the US. Their prices are among the lowest... Read more
At only $809, these are the cheapest MacBooks...
Apple has restocked at full line of Certified Refurbished 2020 13″ Intel-based MacBook Airs available starting at only $809 and up to $280 off original MSRP. Each MacBook features a new outer case,... Read more

Jobs Board

Technical Support - VIVA USA Inc (United Stat...
…when necessary to other client personnel. Knowledge: Systems administration knowledge of Microsoft, Apple , and Linux OS to include but not limited to Windows 7, 8, Read more
Store Associate - Shortage Control - Burlingt...
**LOCATION** 19131 Bear Valley Road Apple Valley CA US 92308 **Overview** If you want an exciting job with one of the largest off-price retail stores in the nation, Read more
Lead Software Engineer - 72673BR - CVS Health...
…by Aetna app, a first-of-its-kind health experience designed in collaboration with Apple . It helps drive healthy behavior by combining your health history with Read more
IOS Developer - Deloitte (United States)
…applications, including knowledge of Swift, Xcode, UIKit, AppleWatch and other core Apple frameworks. + Knowledge of Apple 's HealthKit, ResearchKit, and CareKit Read more
OBGYN Nurse Practitioner - Provider Solutions...
…clinicians in serving a medically diverse, high-acuity, underserved patient population in Apple Valley, California. You will have the opportunity to grow your Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.