TweetFollow Us on Twitter

Not CurSED, BlesSED!

Volume Number: 22 (2006)
Issue Number: 2
Column Tag: Programming

Mac In The Shell

Not CurSED, BlesSED!

by Edward Marczak

But sometimes, the keys to sed can be disguised.

This month, we're going to follow up on sed, the powerhouse non-interactive editor, introduced in part 1 in December's issue. Hopefully the holidays of that month didn't take you away from really digging in and putting the lessons into practice. If you did take it all in, you've probably found several uses for it already. This month, we'll visit more sed mnemonics and add a few new tricks.

Bob's Yer Uncle

A few reminders about sed regarding redirection and piping. In part 1, we worked on a single file, viewing the results on screen. There are some things I need to clear up before continuing.

Naturally, you can redirect the output from sed to a file. What I didn't mention last month is a tendency that many first-time sed users need to stay away from. Namely, redirecting to the same file that you're working on. Sure, you've tested your script and you're 100% confident that it's going to Do The Right Thing. So, you do this:

sed -f myscript.sed long_text_file.txt > long_text_file.txt

Seems perfectly natural, right? We'll, sorry, you can't do that. You can't redirect to the same file you're altering. This is not an issue with sed, but just the way the shell works. The shell will truncate the destination file before it executes your command. So, while it seems like it might work, it doesn't. You need to write to a temporary file first, and then overwrite the original if you desire to do so. The desired effect is achieved like this:

$ sed -f myscript.sed long_text_file.txt > tmp_file.txt
$ mv -f tmp_file.txt long_text_file.txt

Also, it may not have been clear from the introductory article that sed is happy to work 'on-the-fly' by accepting and pumping out data via a pipe. To get a count of directories in a certain folder, for example, you could do this:

ls -lRF /Users/marczak/Documents | grep "^/" | sed s/:/'\/'/ | sed s/\ /'\\ '/g | wc -l

Pipe-to-pipe-to-pipe-to-pipe! However, a sed master will never use two sed statements where one would do. This can also be written as:

ls -lRF /Users/marczak/Documents | grep "^/" | sed -e s/:/'\/'/ -e s/\ /'\\ '/g | wc -l

In short, pipe away, redirect carefully, and Bob's yer uncle!

Do it Better

While the information in part 1 of this article is more than enough for very powerful manipulations, you still may run into some limitations. That's why part 1 was the introduction: there's still more!

First, I mentioned that I would give the solution the 'swapping Bill and Michael' problem. The answer, of course, is, "it depends." You really have to be familiar with your source material. For now, I'll show the easy, yet most brute-force way of handling our scenario. Remember, the source text is this:

    Bill and Michael went to the store. Bill needed to buy some butter, eggs and flour. He and Michael were in a hurry to bake a cake for their parent's Anniversary. Once they got home, Bill and Michael realized that they forgot cake icing.

Just like writing any code, you can swap using a temporary variable (sorry - sed doesn't quite have anything like a bitwise swap!). So, here you go:

sed -e 's/Bill/David/g' -e 's/Michael/Bill/g' -e 's/David/Michael/g' short_story.txt

In our case, we probably really only want the fully qualified "Bill and Michael", so we can actually do just that:

sed 's/Bill and Michael/Michael and Bill/g' short_story.txt

However, you may realize, that this does not take into account line endings. What if our phrase crosses a newline boundary? Ah! That's where multi-line commands come in.

Multi-line commands give sed the ability to look at more than one line in the pattern space. This gives the sed script-crafter the ability to inject a little logic into the flow. Since our story does not split "Bill and Michael" anywhere, let's look at something that does: "buy some...". If we wanted to change all occurrences of "buy some" to "purchase some", even if it spans lines, we need to coax sed into doing so. Again, the brute-force way is simply this:

/buy/ {
N
s/buy *\n*some/purchase some/
}

Here, we look for the address "buy" and when found, run a multi-line next ("N") command. This command reads the next line of input and appends it to the current pattern space - still 'separated' by a new line. Like I said, brute-force, and doesn't really scale well. Also, this has the potential of outputting some really long lines. Of course, elegance is just around the corner. A script that gives us use in more general cases could look like this:

/buy/ {
N
s/ *\n/ /
s/buy some */purchase milk,\
/
}

First, we look for the address "buy", and if we find it, we pull the next line into the pattern space with "N". Then, we can ditch the new line character and replace it with a space. From there we can try to match our patterns. However, even this is a little problematic - just a drop. The example just given works just fine, but let's alter our story and script. Adding a line to the story to make it this:

    Bill and Michael went to the store. Bill needed to buy some butter, eggs and flour. He and Michael were in a hurry to bake a cake for their parent's Anniversary. Once they got home, Bill and Michael realized that they forgot cake icing. It is important that they had not forgotten anything for the special day.

And changing the script to this:

/got/ {
N
s/ *\n/ /
s/got home */returned to their abode\
/
}

...will lead to problems. The goal of this script is to change all occurrences of "got home" into "returned to their abode". Run it and see what happens. See the problem? "got" matches "forgotten" on the last line, but makes no substitutions, so the script quits without outputting that line. It's just MIA! What to do? Exempt the last line of the script. Change the "N" to "$!N" - sed recognizes "$" as the last line (not EOL like regexp).

The reality is that you'll find many, many, many examples like this. Depending on your source(s), you may only be able to make a script just so general. This goes back to the rule: test, test, test! You can't test your script enough.

Loose Fit

In addition to the main pattern space that sed matches and manipulates, there is also a hold buffer. The commands are pretty self explanatory: 'x' will eXchange the pattern space with the hold buffer. 'h' will copy the current pattern space into the hold buffer, overwriting what was being held previously. 'H' will do the same, but append to the current hold buffer. 'g' gets the contents of the hold buffer and replaces the pattern space. 'G' gets the contents of the hold buffer and appends its contents to the current pattern space.

Before I get into these commands, please remember that sed certainly is a descendent of the phrase TIMTOWTDI - There is more than one way to do it. Many times, there will be multiple solutions to the particular problem you're trying to overcome. Build up one piece at a time, test, and for goodness sake, document your solution! (Did I mention that sed scripts recognize "#" as a comment?)

So, let's say we want to print the line before and the line after our match, so we can see it in context - like grep's 'A', 'B', and 'C' flags. Here's one way to approach this:

sed -n '
'/Anniversary/' !{
# lines that do not match what we're looking for - save
x
# clear the current pattern buffer with delete
d
}
'/Anniversary/' {
# lines that match
# get the previous line from the hold buffer
x
# print it with p
p
# get the current line back from the hold buffer
x
# print that
p
# get the next line
n
# print it
p
# finally, drop this line into the hold buffer
x
}' short_story.txt

Going back to our short story, this will look for a line containing 'Anniversary', print the line before it, the line itself, and the line following. Note the use of the "-n" switch passed into sed. This switch tells sed to not print all output by default. Otherwise, you'll still see all of your input as it filters through sed. Of course, to make all of this more useful, you could drop this right into a shell script, and use $1 for the pattern - this would give you a generic script that will always perform the equivalent of "grep -C 1 pattern file.txt". Just remember: the way I broke this up over several lines is very bash specific. csh users must use the backslash to tell the interpreter that the line continues on.

Step On

Earlier, I alluded to sed as a programming language by mentioning the classic temp-variable-swap. Well, sed tends to be more full featured than most people realize. You can even implement flow-control! sed features two commands that let you control the logic of your script. 'b' branches to a label. (Reminds me of my favorite Motorola Assembly mnemonic - BRA - BRanch Always). One example, and you'll get it. This script is an alternate to the script just presented that emulates 'grep -C 1':

# find our pattern? jump!
/Anniversary/ b printit
# else hold it
h
# jump to end of script
b
:printit
# get previous line from hold buf
x
# print it
p
# get current line back
x
# print it
p
# get next line
n
# print that
p

First to note is the label. A label starts a line with a colon, and should contain seven characters or less. While most modern implementations allow a label to be any length - and is actually the POSIX spec - there are still some versions of sed that restrict a label to 7 characters max. With 'b', we just branch - make the jump. Another flow-control command is 't', or, test. Test allows us to branch conditionally. The jump only happens if the previous substitution was successful. Another example is in order. Imagine a file that lists a userclass by number, and it should be by name. Additionally, you must process the file differently for each userclass. Here's a mock script that could handle this:

/userclass/{
s/2000/executive/
t excpath
s/1500/management/
t manpath
s/1000/staff/
t stpath
# default action
=
b
:excpath
...
b
:manpath
...
b
:stpath
...
# end of script

Those examples should get you going with flow control in sed. Remember, you can certainly jump to a label ahead of your test and even get into (basic) recursion!

Harmony

There are a bunch of miscellaneous things that I'd like to point out before we wrap up our conversation about sed. Some of these fall into the "you have to know the rules before you can break them" category, so I waited to present them.

All of the examples I've given, and most that you'll see, use a forward slash ("/") as the delimiter. Amazingly, sed lets you use whatever you like. The delimiter is great if you have a very simple replacement, such as sed s/one/two/g. However, if you're replacing file paths, that could get messy. So, use an underscore, if you like: sed 's_/usr/sbin_/usr/bin_'. Easier to read, right?

The "=" (equal sign) mnemonic prints out the current line number. So, you can simply line number any text file:

sed '=' textfile.txt

Or, you can use to just find the lines that the pattern you're looking for lie in:

sed -n '/Bill/ =' story.txt

You've probably picked these two up by now, but I should make them clear. Quoting: you really only need to quote if you have metacharacters, but it's a good habit to get into. This: sed s/Bill/Mike/g is the same as this: sed 's/Bill/ Mike/g'. However, try this: sed s/.*address*\ (astring\)\(\1)$1)/ and sed is just going to cough up electrons. The second thing is the use of the "-e" switch. If you only have one command, you can forgo the "-e". If you have multiple commands, you need to add each of them to the list of editing commands with the "-e" switch.

Lastly, a note about newlines, or, EOL. Somewhat confusingly, you match a newline with the standard regexp '\n'. However, to output a newline, you use a literal newline:

sed 's/Bill/Bill\
/g' story.txt

This will drop a newline after every occurrence of 'Bill' in our sample text.

Cut 'Em Loose Bruce

...and I thought I was going to get to awk this month! Hopefully, this gives you some ideas about sed and its power. I also hope with practice, that you use this power. You really have to see sed as an editor. It just happens to be one that you don't use interactively like vi, emacs or pico. Go forth and edit non-interactively! sed is an indispensable tool for any system administrator's toolchest, and there are plenty of repetitive tasks waiting for you to automate.


Ed Marczak owns and operates Radiotope, a technology consulting company. Despite being around the technology block once or twice, he's thankful that there's always something new to look forward to. Something new at http://www.radiotope.com

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Art Text 4.0.1 - $29.99
Art Text is graphic design software specifically tuned for lettering, typography, text mockups and various artistic text effects. Supplied with a great variety of ready to use styles and materials,... Read more
Adobe Dreamweaver CC 2020 20.2 - Build w...
Dreamweaver CC 2020 is available as part of Adobe Creative Cloud for as little as $20.99/month (or $9.99/month if you're a previous Dreamweaver customer). Adobe Dreamweaver CC 2020 allows you to... Read more
Adobe Acrobat DC 20.009.20074 - Powerful...
Acrobat DC is available only as a part of Adobe Creative Cloud, and can only be installed and/or updated through Adobe's Creative Cloud app. Adobe Acrobat DC with Adobe Document Cloud services is... Read more
beaTunes 5.2.10 - Organize your music co...
beaTunes is a full-featured music player and organizational tool for music collections. How well organized is your music library? Are your artists always spelled the same way? Any R.E.M. vs REM?... Read more
DiskCatalogMaker 8.1.5 - Catalog your di...
DiskCatalogMaker is a simple disk management tool which catalogs disks. Simple, light-weight, and fast Finder-like intuitive look and feel Super-fast search algorithm Can compress catalog data for... Read more
Meteorologist 3.4.1 - Popular weather ap...
Meteorologist is a simple interface to weather provided by weather.com. It provides the ability to show the weather in the main menu bar, displaying more detail in a pop-up menu, whose contents are... Read more
NeoFinder 7.6 - Catalog your external me...
NeoFinder (formerly CDFinder) rapidly organizes your data, either on external or internal disks, or any other volumes. It catalogs and manages all your data, so you stay in control of your data... Read more
GarageSale 8.1.1 - Create outstanding eB...
GarageSale is a slick, full-featured client application for the eBay online auction system. Create and manage your auctions with ease. With GarageSale, you can create, edit, track, and manage... Read more
Firetask Pro 4.2.2 - Innovative task man...
Firetask Pro uniquely combines the advantages of classical priority-and-due-date-based task management with GTD. Stay focused and on top of your commitments - Firetask Pro's "Today" view shows all... Read more
Bookends 13.4.3 - Reference management a...
Bookends is a full-featured bibliography/reference and information-management system for students and professionals. Bookends uses the cloud to sync reference libraries on all the Macs you use.... Read more

Latest Forum Discussions

See All

Distract Yourself With These Great Mobil...
There’s a lot going on right now, and I don’t really feel like trying to write some kind of pithy intro for it. All I’ll say is lots of people have been coming together and helping each other in small ways, and I’m choosing to focus on that as I... | Read more »
Pokemon Go's July Community Day wil...
Pokemon Go developers have announced the details concerning the upcoming Gastly Community Day. This particular event was selected by the players of the game after the Gas Pokemon came in second place after a poll that decided which Pokemon would... | Read more »
Clash Royale: The Road to Legendary Aren...
Supercell recently celebrated its 10th anniversary and their best title, Clash Royale, is as good as it's ever been. Even for lapsed players, returning to the game is as easy as can be. If you want to join us in picking the game back up, we've put... | Read more »
Detective Di is a point-and-click murder...
Detective Di is a point-and-click murder mystery set in Tang Dynasty-era China. You'll take on the role of China's best-known investigator, Di Renjie, as he solves a series of grisly murders that will ultimately lead him on a collision course with... | Read more »
Dissidia Final Fantasy Opera Omnia is se...
Dissidia Final Fantasy Opera Omnia, one of Square Enix's many popular mobile RPGs, has announced a plethora of in-game events that are set to take place over the summer. This will include several rewards, Free Multi Draws and more. [Read more] | Read more »
Sphaze is a neat-looking puzzler where y...
Sphaze is a neat-looking puzzler where you'll work to guide robots through increasingly elaborate mazes. It's set in a visually distinct world that's equal parts fantasy and sci-fi, and it's finally launched today for iOS and Android devices. [... | Read more »
Apple Arcade is in trouble
Yesterday, Bloomberg reported that Apple is disappointed in the performance of Apple Arcade and will be shifting their approach to the service by focusing on games that can retain subscribers and canceling other upcoming releases that don't fit... | Read more »
Pixel Petz, an inventive platform for de...
Pixel Petz has built up a sizeable player base thanks to its layered, easy-to-understand creative tools and friendly social experience. It revolves around designing, trading, and playing with a unique collection of pixel art pets, and it's out now... | Read more »
The King of Fighters Allstar's late...
The King of Fighters ALLSTAR, Netmarble's popular action RPG, has once again been updated with a plethora of new content. This includes battle cards, events and 21 new fighters, which increases the already sizeable roster even more. [Read more] | Read more »
Romancing SaGa Re;univerSe, the mobile s...
Square Enix latest mobile spin-off Romancing SaGa Re;univerSe is available now globally for both iOS and Android. It initially launched in Japan back in 2018 where it's proven to be incredibly popular, so now folks in the West can finally see what... | Read more »

Price Scanner via MacPrices.net

Apple’s $779 13″ MacBook Air deal returns to...
Apple has clearance, Certified Refurbished, 2019 13″ MacBook Airs available again starting at $779. Each MacBook features a new outer case, comes with a standard Apple one-year warranty, and is... Read more
$200 13″ MacBook Pro discounts are back at Am...
Amazon has 2020 13″ 2.0GHz MacBook Pros on sale again today for $150-$200 off Apple’s MSRP. Shipping is free. Be sure to purchase the MacBook Pro from Amazon, rather than a third-party seller, and... Read more
Deal Alert! Apple AirPods with Wireless Charg...
Sams Club has Apple AirPods with Wireless Charging Case on sale on their online store for only $149.98 from July 6, 2020 to July 9, 2020. Their price is $50 off Apple’s MSRP, and it’s the lowest... Read more
Xfinity Mobile promo: Apple iPhone XS models...
Take $300 off the purchase of any Apple iPhone XS model at Xfinity Mobile while supplies last. Service plan required: – 64GB iPhone XS: $599.99 save $300 – 256GB iPhone XS: $749.99 save $300 – 512GB... Read more
New July 2020 promo at US Cellular: Switch an...
US Cellular has introduced a new July 2020 deal offering free 64GB Apple iPhone 11 smartphones to customers opening a new line of service. No trade-in required, and discounts are applied via monthly... Read more
Apple offers up to $400 Education discount on...
Apple has launched their Back to School promotion for 2020. They will include one free pair Apple AirPods (with charging case) with the purchase of a MacBook Air, MacBook Pro, iMac, or iMac Pro (Mac... Read more
July 4th Sale: Woot offers wide range of Macs...
Amazon-owned Woot is blowing out a wide range of Apple Macs and iPads for July 4th staring at $279 and ranging up to just over $1000. Models vary from older iPads and 11″ MacBook Airs to some newer... Read more
Apple Pro Display XDR with Nano-Texture Glass...
Abt Electronics has Apple’s new 32″ Pro Display XDR model with the nano-texture glass in stock and on sale today for up to $144 off MSRP. Shipping is free: – Pro Display XDR (nano-texture glass): $... Read more
New 2020 Mac mini on sale for up to $100 off...
Amazon has Apple’s new 2020 Mac minis on sale today for $40-$100 off MSRP with prices starting at $759. Shipping is free: – 2020 4-Core Mac mini: $759 $40 off MSRP – 2020 6-Core Mac mini: $998.99 $... Read more
July 4th Sale: $100 off every 2020 13″ MacBoo...
Apple resellers have new 2020 13″ MacBook Airs on sale for $100 off Apple’s MSRP as part of their July 4th sales. Starting at $899, these are the cheapest new 2020 MacBooks for sale anywhere: (1) B... Read more

Jobs Board

Operating Room Assistant, *Apple* Hill Surg...
Operating Room Assistant, Apple Hill Surgical Center - Full Time, Day Shift, Monday - Saturday availability required Tracking Code 62363 Job Description Operating Read more
Perioperative RN - ( *Apple* Hill Surgical C...
Perioperative RN - ( Apple Hill Surgical Center) Tracking Code 60593 Job Description Monday - Friday - Full Time Days Possible Saturdays General Summary: Under the Read more
Product Manager, *Apple* Commercial Sales -...
Product Manager, Apple Commercial Sales Austin, TX, US Requisition Number:77652 As an Apple Product Manager for the Commercial Sales team at Insight, you Read more
*Apple* Mac Product Engineer - Barclays (Uni...
Apple Mac EngineerWhippany, NJ Support the development and delivery of solutions, products, and capabilities into the Barclays environment working across technical Read more
Blue *Apple* Cafe Student Worker - Pennsylv...
…enhance your work experience. Student positions are available at the Blue Apple Cafe. Employee meal discount during working hours. Duties include food preparation, Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.