TweetFollow Us on Twitter

Mac In The Shell: More from the Unknown

Volume Number: 23 (2007)
Issue Number: 07
Column Tag: Mac In The Shell

More from the Unknown

More shell tools that typically go unnoticed

by Edward Marczak

Introduction

OS X has an incredible breadth and depth available in the tools accessible from a shell. So much so, that it's hard to know each and every tool available. Some are almost hidden: tucked away in places one does not typically look. Others are hidden in plain sight: exactly where you'd expect them, but buried among the volume. This month, I'm going to dig out and introduce some more useful shell tools that you may not know about.

text manipulation

It's long been the domain of scripting languages like perl, and perhaps now to a lesser extent PHP, to slice, dice and otherwise manipulate text. However, there's a framework built right into OS X, so why not take advantage of it? The Cocoa text system is exposed to the shell via the "textutil" application. "textutil" can convert and manipulate many different formats of text documents. In fact, it works with a surprising number formats:

   txt
   html
   Microsoft Word "doc"
   Microsoft Word XML
   rtf
   webarchive

It's also incredibly easy to get started. Using the "-convert" switch, you can accomplish most conversions that you'd ever want:

textutil -convert html resume.doc

This will convert the Microsoft Word-based "resume.doc" into a separate HTML document named "resume.html". I'll immediately point out that textutil writes much, much better HTML than Word.

Note that the input format is determined 'automagically' by a number of methods. If textutil misidentifies the format of the source file, you tell it with the "-format" switch:

textutil -format html -convert rtf file1

This will tell textutil that "file1" is an HTML file, and we want to convert it into a rich text format file named "file1.rtf".

There's also a very handy "-info" switch that returns info on a given file:

$ textutil -info 2007-01-On\ Logs.doc
File:  2007-01-Marczak On Logs.doc
Type:  Word format
Size:  25600 bytes
Length:  3746 characters
Title:  MacTech Template
Author:  Classics Department
Last Editor:  Edward Marczak
Company:  Radiotope
Subject:  
Keywords:  
Created:  2006-11-02 17:56:00 -0500
Last Modified:  2006-11-06 09:47:00 -0500

As you can see, documents sometimes have incorrect metadata. Well, textutil can help you handle that as well! There are a number of metadata related switches:

strip: Remove metadata in target during conversion.

title: Set the title metadata attribute for output files.

author: Set the author metadata attribute for output files.

subject: Specify the subject metadata attribute for output files.

keywords: a shell-quoted list of 'tags' ("keyword1,keyword2, ...")

comment: Set the comment attribute for the output files.

editor: val Set the editor attribute for output files.

company: val Set the company attribute for output files.

creationtime: Set the creation time attribute for output files (in "yyyy-mm-ddThh:mm:ssZ" format).

modificationtime: Set the modification time attribute for output files (in "yyyy-mm-ddThh:mm:ssZ" format).

So, we could create an HTML document from a plain text file with the title and keywords set by running textutil like this:

textutil -convert html -title "Anti-gravity Thesis" -keywords "(anti,gravity,'atom power',physics)" -author "Bruce Banner" ag-file.txt

You'll find a well-formed HTML file with a little bit of CSS. Of course, there are many times when you need finer control over the tags that will be used in that HTML. Have no fear! The "-excludedelements" switch can help you out. This switch allows you to specify which HTML elements should not be used in the generated HTML. Like the "-keywords" switch, "-excludedelements" expects a single argument, so you can use a shell-quoted list of tags to skip.

Keep it clean

Speaking of well-formed HTML, manipulating tags and their parameters is a special case of text manipulation. Fortunately, tidy, the standard Unix utility just for this purpose, ships with OS X. So, if you've used textutil to generate HTML based output, tidy can take you even further.

The simplest case lets tidy modify your HTML document in place. To do so, use the "-m" switch:

$ tidy -im ag-file.html
Info: Doctype given is "-//W3C//DTD HTML 4.01//EN"
Info: Document content looks like HTML 4.01 Strict
No warnings or errors were found.

Take look at your file before and after running the tidy command. I also like to include the "-i" switch, which ensures tag levels are indented appropriately. Also, by default, tidy will lowercase all tags (which, by me, is "correct"). This can be overridden with the "-u" switch.

One of my absolute favorite uses of tidy is to get rid of "illegal" characters. This is easily accomplished with the "-b" (bare) switch. This strips the document of smart quotes, and dashes, and other characters that cause problems in HTML documents.

There's much, much more that tidy can do. Check the very well written man page to look at more options. Don't forget, though, that you can combine textutil and tidy into one neat shell statement:

textutil -stdout ag-file.txt -convert html | tidy -i -output ag-file.html

This command uses textutil to convert a text file to HTML, and sends the output to stdout. Then, that output is piped to tidy, which then requires us to use the "-output" switch to write the file to disk.

Finally, realize that both textutil and tidy can be used with a wildcard character to process an entire group of files. If all files are in a single directory, simply change to that directory and:

textutil -convert html *.doc

This will convert all Word documents in the current directory into HTML files. To process the current folder and all subfolders, use the find utility:

find . -name \*txt -print0 | xargs -0 textutil -convert html

I opted for xargs over find's exec command to ensure that large directories can be processed. You can scrub Word metadata in-place using this technique:

textutil -strip -convert doc *.doc

Painless, right?

Image Manipulation

While slinging text around may be a common, and very specific operation, doing the same to images requires a different set of tools. OS X's scriptable image processor server or, "sips" is just the ticket.

Like textutil, sips leans on frameworks already built into the core of OS X. This, of course, lets you do all of the great things that you can with a GUI right in a shell! Why is this exciting? Automation, naturally.

In the past, I've put together both nightly reports and 'one-click-builds' of documents using both user submitted and random images. The trick is not so much the content, but the dimensions of the graphic: you need consistent dimensions so you can automate your layout. Resizing graphic files is one of the things that sips does best:

sips -Z 100x100 IMG_1312.JPG --out image1-sized.jpg

The "-Z" switch is really nice: it keeps the proportions of the original image, but makes sure neither the width or height exceed the dimensions specified. The "--out" switch writes the result to a new file. The image you just made doesn't have a nice preview icon in the Finder? Add one:

sips -i image1-sized.jpg

Oh, you didn't want a JPEG format file? Convert it at the same time using the "-s" (setProperty) switch (let's do it all in one shot):

sips -Z 100x100 IMG_1312.JPG -i -s format png --out image-sized.png

This one command resizes the graphic, converts the output to png and creates the Finder preview icon. Nice.

For automation, though, you often need to find out information about the source before you blindly process it. Again, like textutil, sips has functions for this. You can dump all of an image's data with the "-g all" switch and parse the output yourself:

$ sips -g all IMG_1312.JPG 
/Users/erm/Pictures/Parade/IMG_1312.JPG
  pixelWidth: 2592
  pixelHeight: 1944
  typeIdentifier: public.jpeg
  format: jpeg
  formatOptions: default
  dpiWidth: 180.000
  dpiHeight: 180.000
  samplesPerPixel: 3
  bitsPerSample: 8
  hasAlpha: no
  space: RGB
  profile: Camera RGB Profile
  creation: 2007:04:04 17:23:31
  make: Canon
  model: Canon PowerShot S2 IS

...or, you can request one or more of these parameters directly:

$ sips -g pixelWidth -g pixelHeight IMG_1312.JPG
/Users/marczak/Pictures/Parade/IMG_1312.JPG
  pixelWidth: 2592
  pixelHeight: 1944

Like textutil, feel free to throw a wildcard at sips, or, use the find/xargs trick.

Why?

So, why go through this trouble, when you can fire up Photoshop and make your changes (and even script it!), or use a Word processor to mold your words? Again, automation comes to mind. Specifically, automation with low overhead. Using the shell tools, you can run scripts on a server without having a GUI or installing an application like Photoshop at all. Combine this with some of the techniques I presented in April, and you can achieve some incredibly complex workflows.

Even cooler: more and more GUI utilities have some way to run shell jobs. Textmate, for example, can run a selection or entire document through a shell tool and plop the results right back in the live document. Even apps that don't have this functionality built in can usually be faked thanks to AppleScript. Take Mail.app, for instance. It lets you create signatures, but not dynamic signatures that rely on a running process. What if you wanted to include a snapshot from your iSight in your sig? Script a capture from isightcapture (http://www.intergalactic.de/hacks.html), run it through sips and AppleScript the copy and pasting of into Mail.

Or, consider a simpler alternative: you've installed fortune and want a random fortune in your sig. A simple bash script could be as follows:

#!/bin/bash
# Tell Mail.app to select all and copy to clipboard
osascript <<MailCopy
tell application "System Events"
        tell application "Mail" to activate
        keystroke "a" using {command down}
        keystroke "c" using {command down}
end tell
MailCopy
# Write clipboard to file
pbpaste > /tmp/mailpaste.tmp
fortune -s >> /tmp/mailpaste.tmp
textutil -stdout -convert rtf /tmp/mailpaste.tmp | pbcopy
# Tell Mail.app to activate and paste in the contents of the clipboard
osascript <<MailPaste
tell application "System Events"
        tell application "Mail" to activate
        keystroke "a" using {command down}
        keystroke "v" using {command down}
end tell
MailPaste

This is admittedly a quick hack with absolutely no error checking or other niceties. However, if you open up Mail.app, create a new message, put your cursor anywhere in the body and run this, you'll get a fortune tagged onto the end of your message (it helps if you already have a sig created). Pretty it up a bit and you could run it via the GUI via a ".command" file, or, wrap it in AppleScript and run it from your script menulet. Or, if you're a Quicksilver user, don't ignore the qs shell tool.

Of course, the real power lies in integrating other data: from the web, from Excel, from a database....wherever!

Conclusion

When venerable shell tools are integrated with the power of OS X, the workflow possibilities truly are endless. Be creative in your application of scripting tools, and you'll be able to solve any problem for your client/employer/self.

Media of the month: Brendan Benson's new album, "Alternative to Love". It's just good, fun pop music, no strings attached.

Until next month, keep thinking of ways to keep the work flowing.

References:

"Developer Release Note": http://developer.apple.com/releasenotes/Cocoa/AppKit.html


Ed Marczak owns and operates Radiotope, a technology consultancy that just tries to make life easier through the technology that already exists. He's also the Executive Editor of MacTech Magazine, and CTO of WheresSpot. His copious free time is spent with his wife and two daughters, currently enjoying the weather in the North East U.S.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Iridient Developer 3.3.12 - Powerful ima...
Iridient Developer (was RAW Developer) is a powerful image-conversion application designed specifically for OS X. Iridient Developer gives advanced photographers total control over every aspect of... Read more
DxO PhotoLab 3.3.0.54 - Image enhancemen...
DxO PhotoLab (was DxO Optics Pro) provides a complete set of smart assisted corrections that you can manually fine-tune at any time. Take control on every aspect of your photos: effectively remove... Read more
Airfoil 5.9.1 - Send audio from any app...
Airfoil allows you to send any audio to AirPort Express units, Apple TVs, and even other Macs and PCs, all in sync! It's your audio - everywhere. With Airfoil you can take audio from any... Read more
TunnelBear 3.9.8 - Subscription-based pr...
TunnelBear is a subscription-based virtual private network (VPN) service and companion app, enabling you to browse the internet privately and securely. Features Browse privately - Secure your data... Read more
Things 3.12.4 - Elegant personal task ma...
Things is a task management solution that helps to organize your tasks in an elegant and intuitive way. Things combines powerful features with simplicity through the use of tags and its intelligent... Read more
Vitamin-R 3.17 - Personal productivity t...
Vitamin-R creates the optimal conditions for your brain to work at its best by structuring your work into short bursts of distraction-free, highly focused activity alternating with opportunities for... Read more
iMazing 2.11.6 - Complete iOS device man...
iMazing (was DiskAid) is the ultimate iOS device manager with capabilities far beyond what iTunes offers. With iMazing and your iOS device (iPhone, iPad, or iPod), you can: Copy music to and from... Read more
BBEdit 13.1 - Powerful text and HTML edi...
BBEdit is the leading professional HTML and text editor for the Mac. Specifically crafted in response to the needs of Web authors and software developers, this award-winning product provides a... Read more
Apple Security Update 2020-003 - For mac...
Apple Security Update is recommended for all users and improves the security of OS X. For detailed information about the security content of this update, please visit lists security updates for... Read more
BetterTouchTool 3.379 - Customize multi-...
BetterTouchTool adds many new, fully customizable gestures to the Magic Mouse, Multi-Touch MacBook trackpad, and Magic Trackpad. These gestures are customizable: Magic Mouse: Pinch in / out (zoom)... Read more

Latest Forum Discussions

See All

The 5 Best Mobile Real Time Strategy Gam...
Real-time strategy games feel like they’d be a perfect fit for mobile, but they’re trickier to pull off that you might think. The traditional mold of base-building and micro management can work on touch screens, but needs to be carefully honed so... | Read more »
Using your phone in a protest
I can't write about games today. There is a struggle happening in the streets right now and it needs everyone's attention. Here's some good info on how you can use your iOS device safely amidst a protest. | Read more »
Dungonian is a card-based dungeon crawle...
Dungonian is a card-based dungeon crawler from developer SandFish Games that only recently launched as a free-to-play title. It offers an extensive roster of playable heroes to collect and enemies to take down, and it's available right now for iOS... | Read more »
Steam Link Spotlight - Signs of the Sojo...
Steam Link Spotlight is a feature where we look at PC games that play exceptionally well using the Steam Link app. Our last entry was XCOM: Chimera Squad. Read about how it plays using Steam Link's new mouse and keyboard support over here. | Read more »
Steampunk Tower 2, DreamGate's sequ...
Steampunk Tower 2 is a DreamGate's follow up to their previous tower defence game. It's available now for both iOS and Android as a free-to-play title and will see players defending their lone base by kitting it out with a variety of turrets. [... | Read more »
Clash Royale: The Road to Legendary Aren...
Supercell recently celebrated its 10th anniversary and their best title, Clash Royale, is as good as it's ever been. Even for lapsed players, returning to the game is as easy as can be. If you want to join us in picking the game back up, we've put... | Read more »
Pokemon Go Fest 2020 will be a virtual e...
Niantic has announced that Pokemon Go Fest will still take place this year although understandably it won't be a physical event. Instead, it will become a virtual celebration and is set to be held on 25th and 26th July. [Read more] | Read more »
Marvel Future Fight's major May upd...
Marvel Future Fight's latest update has now landed, and it sounds like a big one. The focus this time around is on Marvel's Guardians of the Galaxy, and it introduces all-new characters, quests, and uniforms for players to collect. [Read more] | Read more »
SINoALICE, Yoko Taro and Pokelabo's...
Yoko Taro and developer Pokelabo's SINoALICE has now opened for pre-registration over on the App Store. It's already amassed 1.5 million Android pre-registrations, and it's currently slated to launch on July 1st. [Read more] | Read more »
Masketeers: Idle Has Fallen's lates...
Masketeers: Idle Has Fallen is the latest endeavour from Appxplore, the folks behind Crab War, Thor: War of Tapnarok and Light A Way. It's an idle RPG that's currently available for Android in Early Access and will head to iOS at a later date. [... | Read more »

Price Scanner via MacPrices.net

Apple’s new 2020 13″ MacBook Pros on sale for...
Apple reseller Abt Electronics has new 2020 13″ MacBook Pros on sale today for up to $140 off MSRP, starting at $1208. Shipping is free, and most configurations are in stock today. Note that Abt’s... Read more
Apple CEO Reacts To Nationwide Protests Over...
NEWS: 06.03.20 – With the recent death of a black man in the custody of a white police officer igniting outrage among Americans from all walks of life, which resulted in protests and civil unrest... Read more
At up to $420 off MSRP, these Certified Refur...
Apple has Certified Refurbished 2019 16″ MacBook Pros available for up to $420 off the cost of new models, starting at $2039. Each model features a new outer case, shipping is free, and an Apple 1-... Read more
Apple restocks refurbished 3rd generation 12....
Apple restocked select 3rd generation 12.9″ WiFi iPad Pros starting at only $699 and up to $330 off original MSRP. Each iPad comes with a standard Apple one-year warranty, outer cases are new, and... Read more
These wireless carriers are offering Apple’s...
Looking for a deal on Apple’s new iPhone SE? Apple itself offers the unlocked 64GB iPhone SE for $399 or $16.62/month. If you’re willing to try a new carrier, two of Apple’s wireless carriers are... Read more
Save $80 on the 64GB 10.5″ iPad Air with this...
Apple has 10.5″ 64GB WiFi iPad Airs models available for $80 off MSRP, Certified Refurbished. Each iPad comes with Apple’s standard one-year warranty and includes a new outer case. Shipping is free... Read more
Apple offers wide range of refurbished custom...
Save up to $7740 on a custom-configured Mac Pro with these Certified Refurbished models available at Apple. Each Mac Pro features a new outer case, free shipping, and includes Apple’s standard 1-year... Read more
Sale! $200 off on select 2020 13″ MacBook Pro...
Amazon has select 2020 13″ MacBook Pro configurations on sale today for $200 off Apple’s MSRP. Shipping is free. Be sure to purchase the MacBook Pro from Amazon, rather than a third-party seller, and... Read more
June 1 only: $100 off Apple’s iPhones at Boos...
Boost Mobile is offering Apple iPhone 11, 11 Pro, and iPhone 11 Pro Max models for $100 off MSRP with service. Their discount reduces the cost of an iPhone 11/64GB to $599, iPhone 11 Pro to $899 for... Read more
Sams Club Sales Event: $100 off every Apple W...
Sams Club is discounting all Apple Watch Series 5 models by $100 off Apple’s MSRP through June 3, 2020. Choose free shipping or free local store pickup (if available). Sale prices for online orders... Read more

Jobs Board

Security Officer ($23.00/Hourly) - *Apple*...
**Security Officer \($23\.00/Hourly\) \- Apple Store** **Description** About NMS Built on a culture of safety and integrity, NMSdelivers award\-winning, integrated Read more
*Apple* Service Desk Analyst - Zones LLC (Un...
Apple Service Desk Analyst Professional Services Albany, New York Apply Description Position OverviewAs a Service Desk Analyst, you will provide customer-friendly Read more
*Apple* Mac Desktop Support - Global Dimensi...
…Operate and support an Active Directory (AD) server-client environment for all Apple devices operating on the BUMED network + Leverage necessary industry enterprise Read more
*Apple* Mac Desktop Support - Global Dimensi...
…Operate and support an Active Directory (AD) server-client environment for all Apple devices operating on the BUMED network + Leverage necessary industry enterprise Read more
Surgical Technologist III, *Apple* Hill Sur...
Surgical Technologist III, Apple Hill Surgical Center - Full Time Tracking Code D5.29.2020 Job Description Surgical Technologist III Apple Hill Surgical Center Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.