TweetFollow Us on Twitter

Mac in the Shell: Python Text Parsing

Volume Number: 25
Issue Number: 07
Column Tag: Mac in the Shell

Mac in the Shell: Python Text Parsing

Automating entries through keyword searching

by Edward Marczak

Introduction

We've been covering Python basics over the last several columns. This month, we'll hit something with a little practicality: text processing. While computers are really good with numbers, people are really good with words. More often than not, input from people comes as text. Turns out that Python is pretty good at dealing with text processing and manipulation. Let's have a closer look, shall we?

More To The Story

OK, there's a little bit more to the story. I've dealt with e-mail systems and e-mail processing for a very long time. (Let's just say that I started with sendmail before it used m4, mmmmkay?). Oftentimes, though, we want a program dealing with incoming mail. This may be for the purposes of a mailing list, for auto-response or to parse the e-mail and then put relevant bits into a database.

E-mail is either really complex or really simple, depending on how you look at it. It's complex because it's got headers and encoding and parts. But it's simple, because it's all text. No matter what all of the pieces are, they're all just human readable text. Fortunately, there are many pre-built libraries that help deal with the complexity, allowing you, the script writer, to focus on the task at hand: processing the parts of the message body that you're interested in. Python's "batteries included" philosophy ensures that a good mail processing library ships as part of the core package.

How is any of this Mac-specific? Well, it isn't. Not directly. However, I just mentioned that Python has, by default-no extra installation required-a good e-mail processing library. Python ships standard with OS X. That's part of the equation solved. Then, there's the issue of receiving the mail in the first place.

Just about every contemporary mail system has a method of taking incoming mail and feeding it to a script. Postfix, which ships with OS X, is no exception. By default, an SMTP (Simple Mail Transfer Protocol) server wants to receive mail, decide if the mail is for a valid user on it's system, and to then drop that mail in the user's mailbox. That's it. But what about a list server? Well, you take the same SMTP server, but instead of delivering any mail to an end user's mailbox, you hand all mail off to the list server program. The list server program will determine who to deliver this mail to.

This is also similar to server-side anti-spam. All incoming mail is handed off to an anti-spam program. The mail is analyzed, potentially acted upon (read: dropped), and mail is then fed back into the SMTP server for final delivery.

We're not going to do anything so grand here today, but after finishing up, you'll have the groundwork. If you have an OS X machine acting as a mail relay and really want to test/use this, you're going to need to modify some postfix config files directly.

In /etc/postfix/transport, you'll need to first define a transport. Let's say your main mail server is called mail.example.com. If you want to divert mail to a script, have the mail sent to mproc.example.com, and add the following like to /etc/postfix/transport:

mproc.example.com mproc:

This says, "all mail that arrives for mproc.example.com send it to the transport named mproc." Once a transport is defined, we also need to tell postfix how to connect the dots between the transport named mproc and our script. That happens in /etc/postfix/master.cf. Add the following line to the end of the file:

mproc  unix  -       n       n       -       -       pipe
  flags=DRhu user=mproc argv=/usr/bin/mproc.py

This tells postfix that any mail arriving on the mproc transport should be piped to the mproc.py script. This is, of course, assuming that we store our script in /usr/bin as "mproc.py". Adjust as needed.

Of course, we're going to keep it simple: since the text will be piped into the script, it's easy to simulate. The pipe simply delivers the entire message on stdin.

A Text Processing Script

Again, we said that we're really focusing on processing e-mail as it arrives, so, we're going to look for input via stdin (which the pipe above does for us). Other text processing scripts may want to deal with text already in a file or elsewhere. I'll make sure to cover that in a future column, but that's not the goal of today's exercise. Despite 'keeping it simple,' we'll be covering a few new-to-us concepts.

Here's the assignment: currently, stock information arrives via e-mail where a dedicated person reads the mail and inputs the entries into a database. This person could clearly be doing better things, as this can be automated without changing the backend system that is sending the e-mail message (whether that's a person or a machine is immaterial for this article). These messages will have a strict format: category and value, separated by a colon. The body of a message would look like this:

Company: Cartier

Product: Watch

Model: Original Tank

Number: 12324A332

Price: $4,500

Available: Yes

However, there's a problem when parsing an e-mail message: it's never just the body that you receive. It's headers. And MIME parts. Oy. Fortunately, Python's email library has functions to deal with this.

I say, let's dive right in. Here's the code I'm using, which will be followed by an explanation of the program.

Listing 1: e-mail parsing program, epp.py

#!/usr/bin/env python
import email
import re
import sys
from email.Parser import Parser
# The keywords we're looking for
keys = ['Company', 'Product', 'Model', 'Number', 'Price', 'Available']
# Compile each keyword into a regular expression
keysre = {}
for i in keys:
  keysre[i] = re.compile(i)
# Read stdin into a single string
mystdin = sys.stdin.read()
# Create a parser object and parse the input
p = Parser()
ps = p.parsestr(mystdin)
# Examine each message part for an appripriate plain body
for i in ps.walk():
  if i.get_content_subtype() != "plain":
    continue
  plainbody = i.as_string()
# Break message into lines, based on newline char
plainbody = plainbody.splitlines()
for i in plainbody:
  # Look at each key for a match.
  for k in keys:
    if keysre[k].match(i):
      print i
sys.exit(0)

First thing to notice about the code is the relative brevity-37 lines in total. As usual, the first few lines simply get us set up: she-bang line and relevant imports, including the Python-supplied email module.

#!/usr/bin/env python
import email
import re
import sys
from email.Parser import Parser

There have been a few times in this column that I've mentioned the importance of regular expressions (RE). Python has good support for RE from the re module:

# The keywords we're looking for
keys = ['Company', 'Product', 'Model', 'Number', 'Price', 'Available']
# Compile each keyword into a regular expression
keysre = {}
for i in keys:
  keysre[i] = re.compile(i)

What is happening here is that we define a list of the keywords we're going to be looking for in the message body. Python regular expressions need to be compiled into an object, which is why we define the keysre dictionary. Of course, we could define these objects one at a time, but that's really inelegant and doesn't scale. In the loop, the dictionary is filled with keys that correspond to the words we're going to match, with a value of the compiled RE object.

# Read stdin into a single string
mystdin = sys.stdin.read()
# Create a parser object and parse the input
p = Parser()
ps = p.parsestr(mystdin)

The first part of this section is pretty simple: assign all of stdin to the variable mystdin. Part of the email library is the email parser object. This object allows an e-mail message, headers, MIME parts and all to be parsed, iterated over and picked apart. We're defining a new parser object and then loading the variable ps with a parsed version of the message that's arriving on stdin.

# Examine each message part for an appropriate plain body
for i in ps.walk():
  if i.get_content_subtype() != "plain":
    continue
  plainbody = i.as_string()

This section of the code hands us back the plain part of the message. MIME types are described in two parts, such as "text/html". We're only interested in the plain portion of the message if there are additional parts in the message. The conditional tests if the subpart is not plain. If it is not, we continue and go back to the top of the loop. If it is plain, we fall though and assign the entire subpart, as a string, to the variable plainbody.

# Break message into lines, based on newline char
plainbody = plainbody.splitlines()

The splitlines() string method returns a list, each element a line in the string, split by a separator-by default, the newline character. Now, we can examine each line in turn:

for i in plainbody:
  # Look at each key for a match.
  for k in keys:
    if keysre[k].match(i):
      print i

As we examine each line, an if statement tests for a match of our regular expressions by looping through the keysre dirctionary. If there's a match, we print it out. Naturally, we can take other action here besides printing it out, such as storing it internally, comparing it to some known value or even inserting it into a database. One thing you will likely want to do is to split the matching lines into key/value pairs. The string's split method does this very nicely. For example:

key, value = i.split(':')

The argument to split is the separator to split on. In our case, we know the lines are split by the colon character and that we're expecting back two values. The split method will happily split as many times as needed. In the case where you don't know how many values to expect, you may just want to assign to a list, like so:

values = i.split(':')

From there you can work out how many values were split and returned to you, and what to do with them.

Finally, we exit the program with a 'clean' exit code:

sys.exit(0)

Running the Program

If you don't happen to have any test e-mail sitting around, I've placed one on the MacTech ftp site, under this month's directory (ftp.mactech.com/src/mactech/volume25_2009/25.07.sit ). If you run your own mail server, you can actually just go grab a raw message from the mail spool-your own mail, mind you!

Since the instructions I gave in the first part allow postfix to send incoming mail through a pipe and to the application, we need a more convenient way to test. The command line makes this easy: just pipe it yourself. Don't forget to mark the program as executable:

chmod 770 epp.py

and then pipe away:

cat /path/to/mits_test_mail | ./epp.py

(or, substitute the ./ with the full path to the program, if needed). If you're using the test mail from the MacTech ftp site, you should see the output you expect: the values that we're matching on, with no headers, MIME clutter, etc. Take a look at the original test mail file to see just how much cruft is being left out.

Conclusion

This was a bit of a whirlwind tour of several concepts. I'd encourage you to bulk up an application like this by checking for error conditions and then taking appropriate action. Outside of that, though, it's pretty impressive at how few dedicated commands are needed to process a well-formed e-mail message. The rest are really just 'nuts and bolts' features of the language.

Media of the month: I'd like to think that everyone has some kind of music that they like. Something that reached them, or that reminds them of some period of time. Well, growing up in New York certainly left a musical stamp on me. I just finished "No Wave" by Marc Masters, and I just loved every second of it. I remember the NY scene around that time, but was certainly too young to fully appreciate it. I don't expect everyone to fully enjoy or 'get' No Wave. But sometimes, the best way to enjoy music is by reading about it. So think of the music that inspires you and find the reading material that points out its inspiration. Thanks to Bruce Gerson for inspiring the topic this month.

Next month, we'll expand on some of the concepts covered here and dig deeper into the well that Python has to offer.


Ed Marczak is the Executive Editor of MacTech Magazine. He has written for MacTech since 2004.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Sibelius 2020.6 - Music notation solutio...
Sibelius is the world's best-selling music notation software for Mac. It is as intuitive to use as a pen, yet so powerful that it does most things in less than the blink of an eye. The demo includes... Read more
Bookends 13.4.2 - Reference management a...
Bookends is a full-featured bibliography/reference and information-management system for students and professionals. Bookends uses the cloud to sync reference libraries on all the Macs you use.... Read more
OmniGraffle Pro 7.16 - Create diagrams,...
OmniGraffle Pro helps you draw beautiful diagrams, family trees, flow charts, org charts, layouts, and (mathematically speaking) any other directed or non-directed graphs. We've had people use... Read more
Drive Genius 6.1.0 - $79.00
Drive Genius features a comprehensive Malware Scan. Automate your malware protection. Protect your investment from any threat. The Malware Scan is part of the automated DrivePulse utility. DrivePulse... Read more
Tor Browser 9.5 - Anonymize Web browsing...
The Tor Browser Bundle is an easy-to-use portable package of Tor, Vidalia, Torbutton, and a Firefox fork preconfigured to work together out of the box. It contains a modified copy of Firefox that... Read more
VueScan 9.7.28 - Scanner software with a...
VueScan is a scanning program that works with most high-quality flatbed and film scanners to produce scans that have excellent color fidelity and color balance. VueScan is easy to use, and has... Read more
OmniGraffle 7.16 - Create diagrams, flow...
OmniGraffle helps you draw beautiful diagrams, family trees, flow charts, org charts, layouts, and (mathematically speaking) any other directed or non-directed graphs. We've had people use Graffle to... Read more
WALTR 2 2.6.26 - $39.95
WALTR 2 helps you wirelessly drag-and-drop any music, ringtones, videos, PDF, and ePub files onto your iPhone, iPad, or iPod without iTunes. It is the second major version of Softorino's critically-... Read more
Airmail 4.1 - Powerful, minimal email cl...
Airmail is an mail client with fast performance and intuitive interaction. Support for iCloud, MS Exchange, Gmail, Google Apps, IMAP, POP3, Yahoo!, AOL, Outlook.com, Live.com. Airmail was designed... Read more
Iridient Developer 3.3.12 - Powerful ima...
Iridient Developer (was RAW Developer) is a powerful image-conversion application designed specifically for OS X. Iridient Developer gives advanced photographers total control over every aspect of... Read more

Latest Forum Discussions

See All

Willy Jetman: Astromonkey's Revenge...
Barcelona-based developer Last Chicken Games are set to bring their game Willy Jetman: Astromonkey's Revenge to both iOS and Android on 9th July. The Metroidvania is already available on the likes of PS4, Switch and PC but now mobile folk will be... | Read more »
The 5 Best Mobile Real Time Strategy Gam...
Real-time strategy games feel like they’d be a perfect fit for mobile, but they’re trickier to pull off that you might think. The traditional mold of base-building and micro management can work on touch screens, but needs to be carefully honed so... | Read more »
Using your phone in a protest
I can't write about games today. There is a struggle happening in the streets right now and it needs everyone's attention. Here's some good info on how you can use your iOS device safely amidst a protest. | Read more »
Dungonian is a card-based dungeon crawle...
Dungonian is a card-based dungeon crawler from developer SandFish Games that only recently launched as a free-to-play title. It offers an extensive roster of playable heroes to collect and enemies to take down, and it's available right now for iOS... | Read more »
Steam Link Spotlight - Signs of the Sojo...
Steam Link Spotlight is a feature where we look at PC games that play exceptionally well using the Steam Link app. Our last entry was XCOM: Chimera Squad. Read about how it plays using Steam Link's new mouse and keyboard support over here. | Read more »
Steampunk Tower 2, DreamGate's sequ...
Steampunk Tower 2 is a DreamGate's follow up to their previous tower defence game. It's available now for both iOS and Android as a free-to-play title and will see players defending their lone base by kitting it out with a variety of turrets. [... | Read more »
Clash Royale: The Road to Legendary Aren...
Supercell recently celebrated its 10th anniversary and their best title, Clash Royale, is as good as it's ever been. Even for lapsed players, returning to the game is as easy as can be. If you want to join us in picking the game back up, we've put... | Read more »
Pokemon Go Fest 2020 will be a virtual e...
Niantic has announced that Pokemon Go Fest will still take place this year although understandably it won't be a physical event. Instead, it will become a virtual celebration and is set to be held on 25th and 26th July. [Read more] | Read more »
Marvel Future Fight's major May upd...
Marvel Future Fight's latest update has now landed, and it sounds like a big one. The focus this time around is on Marvel's Guardians of the Galaxy, and it introduces all-new characters, quests, and uniforms for players to collect. [Read more] | Read more »
SINoALICE, Yoko Taro and Pokelabo's...
Yoko Taro and developer Pokelabo's SINoALICE has now opened for pre-registration over on the App Store. It's already amassed 1.5 million Android pre-registrations, and it's currently slated to launch on July 1st. [Read more] | Read more »

Price Scanner via MacPrices.net

Apple restocks 27″ iMacs, Certified Refurbish...
Apple has restocked Certified Refurbished 2019 27″ iMacs starting at $1529 and up to $350 off the cost of new models. Apple’s one-year warranty is standard, shipping is free, and each iMac features a... Read more
Apple’s new 2020 13″ 4-Core MacBook Airs on s...
B&H Photo has Apple’s new 2020 13″ 4-Core MacBook Airs on sale today for $100 off Apple’s MSRP, only $1199. Expedited shipping is free to many addresses in the US. The MacBook Airs are the same... Read more
New Verizon promo: $150 off any Apple Watch w...
Verizon is offering $150 off any Apple Watch when purchased alongside an iPhone through June 10, 2020. They’re also offering up to $100 on any Apple Watch trade-in. Here are the details: “Get $150... Read more
Last year’s 13″ 2.4GHz MacBook Pros are avail...
Apple has Certified Refurbished 2019 13″ 2.4GHz/256GB 4-Core Touch Bar MacBook Pros available for $1359, $440 off original MSRP. Apple’s one-year warranty is included, shipping is free, and each... Read more
Apple’s new 2020 13″ MacBook Pros on sale for...
Apple reseller Abt Electronics has new 2020 13″ MacBook Pros on sale today for up to $140 off MSRP, starting at $1208. Shipping is free, and most configurations are in stock today. Note that Abt’s... Read more
Apple CEO Reacts To Nationwide Protests Over...
NEWS: 06.03.20 – With the recent death of a black man in the custody of a white police officer igniting outrage among Americans from all walks of life, which resulted in protests and civil unrest... Read more
At up to $420 off MSRP, these Certified Refur...
Apple has Certified Refurbished 2019 16″ MacBook Pros available for up to $420 off the cost of new models, starting at $2039. Each model features a new outer case, shipping is free, and an Apple 1-... Read more
Apple restocks refurbished 3rd generation 12....
Apple restocked select 3rd generation 12.9″ WiFi iPad Pros starting at only $699 and up to $330 off original MSRP. Each iPad comes with a standard Apple one-year warranty, outer cases are new, and... Read more
These wireless carriers are offering Apple’s...
Looking for a deal on Apple’s new iPhone SE? Apple itself offers the unlocked 64GB iPhone SE for $399 or $16.62/month. If you’re willing to try a new carrier, two of Apple’s wireless carriers are... Read more
Save $80 on the 64GB 10.5″ iPad Air with this...
Apple has 10.5″ 64GB WiFi iPad Airs models available for $80 off MSRP, Certified Refurbished. Each iPad comes with Apple’s standard one-year warranty and includes a new outer case. Shipping is free... Read more

Jobs Board

*Apple* Architect - SAIC (United States)
**Description** We are currently seeking a motivated, career and customer oriented Apple Architect to join our team in Washington, DC to begin an exciting and Read more
*Apple* Support Engineer - SAIC (United Stat...
**Description** We are currently seeking a motivated, career and customer oriented Apple Support Engineer to join our team in Washington, DC to begin an exciting and Read more
Perioperative RN - ( *Apple* Hill Surgical C...
Perioperative RN - ( Apple Hill Surgical Center) Tracking Code 60593 Job Description Monday - Friday - Full Time Days Possible Saturdays General Summary: Under the Read more
Senior Practice Manager - *Apple* Hill Eye...
Senior Practice Manager - Apple Hill Eye Center Tracking Code 61713 Job Description Apple Hill Medical Center General Summary: Under general supervision, manages Read more
*Apple* Mac Desktop Support - Global Dimensi...
…Operate and support an Active Directory (AD) server-client environment for all Apple devices operating on the BUMED network + Leverage necessary industry enterprise Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.