TweetFollow Us on Twitter

Mac in the Shell: Python Text Parsing

Volume Number: 25
Issue Number: 07
Column Tag: Mac in the Shell

Mac in the Shell: Python Text Parsing

Automating entries through keyword searching

by Edward Marczak

Introduction

We've been covering Python basics over the last several columns. This month, we'll hit something with a little practicality: text processing. While computers are really good with numbers, people are really good with words. More often than not, input from people comes as text. Turns out that Python is pretty good at dealing with text processing and manipulation. Let's have a closer look, shall we?

More To The Story

OK, there's a little bit more to the story. I've dealt with e-mail systems and e-mail processing for a very long time. (Let's just say that I started with sendmail before it used m4, mmmmkay?). Oftentimes, though, we want a program dealing with incoming mail. This may be for the purposes of a mailing list, for auto-response or to parse the e-mail and then put relevant bits into a database.

E-mail is either really complex or really simple, depending on how you look at it. It's complex because it's got headers and encoding and parts. But it's simple, because it's all text. No matter what all of the pieces are, they're all just human readable text. Fortunately, there are many pre-built libraries that help deal with the complexity, allowing you, the script writer, to focus on the task at hand: processing the parts of the message body that you're interested in. Python's "batteries included" philosophy ensures that a good mail processing library ships as part of the core package.

How is any of this Mac-specific? Well, it isn't. Not directly. However, I just mentioned that Python has, by default-no extra installation required-a good e-mail processing library. Python ships standard with OS X. That's part of the equation solved. Then, there's the issue of receiving the mail in the first place.

Just about every contemporary mail system has a method of taking incoming mail and feeding it to a script. Postfix, which ships with OS X, is no exception. By default, an SMTP (Simple Mail Transfer Protocol) server wants to receive mail, decide if the mail is for a valid user on it's system, and to then drop that mail in the user's mailbox. That's it. But what about a list server? Well, you take the same SMTP server, but instead of delivering any mail to an end user's mailbox, you hand all mail off to the list server program. The list server program will determine who to deliver this mail to.

This is also similar to server-side anti-spam. All incoming mail is handed off to an anti-spam program. The mail is analyzed, potentially acted upon (read: dropped), and mail is then fed back into the SMTP server for final delivery.

We're not going to do anything so grand here today, but after finishing up, you'll have the groundwork. If you have an OS X machine acting as a mail relay and really want to test/use this, you're going to need to modify some postfix config files directly.

In /etc/postfix/transport, you'll need to first define a transport. Let's say your main mail server is called mail.example.com. If you want to divert mail to a script, have the mail sent to mproc.example.com, and add the following like to /etc/postfix/transport:

mproc.example.com mproc:

This says, "all mail that arrives for mproc.example.com send it to the transport named mproc." Once a transport is defined, we also need to tell postfix how to connect the dots between the transport named mproc and our script. That happens in /etc/postfix/master.cf. Add the following line to the end of the file:

mproc  unix  -       n       n       -       -       pipe
  flags=DRhu user=mproc argv=/usr/bin/mproc.py

This tells postfix that any mail arriving on the mproc transport should be piped to the mproc.py script. This is, of course, assuming that we store our script in /usr/bin as "mproc.py". Adjust as needed.

Of course, we're going to keep it simple: since the text will be piped into the script, it's easy to simulate. The pipe simply delivers the entire message on stdin.

A Text Processing Script

Again, we said that we're really focusing on processing e-mail as it arrives, so, we're going to look for input via stdin (which the pipe above does for us). Other text processing scripts may want to deal with text already in a file or elsewhere. I'll make sure to cover that in a future column, but that's not the goal of today's exercise. Despite 'keeping it simple,' we'll be covering a few new-to-us concepts.

Here's the assignment: currently, stock information arrives via e-mail where a dedicated person reads the mail and inputs the entries into a database. This person could clearly be doing better things, as this can be automated without changing the backend system that is sending the e-mail message (whether that's a person or a machine is immaterial for this article). These messages will have a strict format: category and value, separated by a colon. The body of a message would look like this:

Company: Cartier

Product: Watch

Model: Original Tank

Number: 12324A332

Price: $4,500

Available: Yes

However, there's a problem when parsing an e-mail message: it's never just the body that you receive. It's headers. And MIME parts. Oy. Fortunately, Python's email library has functions to deal with this.

I say, let's dive right in. Here's the code I'm using, which will be followed by an explanation of the program.

Listing 1: e-mail parsing program, epp.py

#!/usr/bin/env python
import email
import re
import sys
from email.Parser import Parser
# The keywords we're looking for
keys = ['Company', 'Product', 'Model', 'Number', 'Price', 'Available']
# Compile each keyword into a regular expression
keysre = {}
for i in keys:
  keysre[i] = re.compile(i)
# Read stdin into a single string
mystdin = sys.stdin.read()
# Create a parser object and parse the input
p = Parser()
ps = p.parsestr(mystdin)
# Examine each message part for an appripriate plain body
for i in ps.walk():
  if i.get_content_subtype() != "plain":
    continue
  plainbody = i.as_string()
# Break message into lines, based on newline char
plainbody = plainbody.splitlines()
for i in plainbody:
  # Look at each key for a match.
  for k in keys:
    if keysre[k].match(i):
      print i
sys.exit(0)

First thing to notice about the code is the relative brevity-37 lines in total. As usual, the first few lines simply get us set up: she-bang line and relevant imports, including the Python-supplied email module.

#!/usr/bin/env python
import email
import re
import sys
from email.Parser import Parser

There have been a few times in this column that I've mentioned the importance of regular expressions (RE). Python has good support for RE from the re module:

# The keywords we're looking for
keys = ['Company', 'Product', 'Model', 'Number', 'Price', 'Available']
# Compile each keyword into a regular expression
keysre = {}
for i in keys:
  keysre[i] = re.compile(i)

What is happening here is that we define a list of the keywords we're going to be looking for in the message body. Python regular expressions need to be compiled into an object, which is why we define the keysre dictionary. Of course, we could define these objects one at a time, but that's really inelegant and doesn't scale. In the loop, the dictionary is filled with keys that correspond to the words we're going to match, with a value of the compiled RE object.

# Read stdin into a single string
mystdin = sys.stdin.read()
# Create a parser object and parse the input
p = Parser()
ps = p.parsestr(mystdin)

The first part of this section is pretty simple: assign all of stdin to the variable mystdin. Part of the email library is the email parser object. This object allows an e-mail message, headers, MIME parts and all to be parsed, iterated over and picked apart. We're defining a new parser object and then loading the variable ps with a parsed version of the message that's arriving on stdin.

# Examine each message part for an appropriate plain body
for i in ps.walk():
  if i.get_content_subtype() != "plain":
    continue
  plainbody = i.as_string()

This section of the code hands us back the plain part of the message. MIME types are described in two parts, such as "text/html". We're only interested in the plain portion of the message if there are additional parts in the message. The conditional tests if the subpart is not plain. If it is not, we continue and go back to the top of the loop. If it is plain, we fall though and assign the entire subpart, as a string, to the variable plainbody.

# Break message into lines, based on newline char
plainbody = plainbody.splitlines()

The splitlines() string method returns a list, each element a line in the string, split by a separator-by default, the newline character. Now, we can examine each line in turn:

for i in plainbody:
  # Look at each key for a match.
  for k in keys:
    if keysre[k].match(i):
      print i

As we examine each line, an if statement tests for a match of our regular expressions by looping through the keysre dirctionary. If there's a match, we print it out. Naturally, we can take other action here besides printing it out, such as storing it internally, comparing it to some known value or even inserting it into a database. One thing you will likely want to do is to split the matching lines into key/value pairs. The string's split method does this very nicely. For example:

key, value = i.split(':')

The argument to split is the separator to split on. In our case, we know the lines are split by the colon character and that we're expecting back two values. The split method will happily split as many times as needed. In the case where you don't know how many values to expect, you may just want to assign to a list, like so:

values = i.split(':')

From there you can work out how many values were split and returned to you, and what to do with them.

Finally, we exit the program with a 'clean' exit code:

sys.exit(0)

Running the Program

If you don't happen to have any test e-mail sitting around, I've placed one on the MacTech ftp site, under this month's directory (ftp.mactech.com/src/mactech/volume25_2009/25.07.sit ). If you run your own mail server, you can actually just go grab a raw message from the mail spool-your own mail, mind you!

Since the instructions I gave in the first part allow postfix to send incoming mail through a pipe and to the application, we need a more convenient way to test. The command line makes this easy: just pipe it yourself. Don't forget to mark the program as executable:

chmod 770 epp.py

and then pipe away:

cat /path/to/mits_test_mail | ./epp.py

(or, substitute the ./ with the full path to the program, if needed). If you're using the test mail from the MacTech ftp site, you should see the output you expect: the values that we're matching on, with no headers, MIME clutter, etc. Take a look at the original test mail file to see just how much cruft is being left out.

Conclusion

This was a bit of a whirlwind tour of several concepts. I'd encourage you to bulk up an application like this by checking for error conditions and then taking appropriate action. Outside of that, though, it's pretty impressive at how few dedicated commands are needed to process a well-formed e-mail message. The rest are really just 'nuts and bolts' features of the language.

Media of the month: I'd like to think that everyone has some kind of music that they like. Something that reached them, or that reminds them of some period of time. Well, growing up in New York certainly left a musical stamp on me. I just finished "No Wave" by Marc Masters, and I just loved every second of it. I remember the NY scene around that time, but was certainly too young to fully appreciate it. I don't expect everyone to fully enjoy or 'get' No Wave. But sometimes, the best way to enjoy music is by reading about it. So think of the music that inspires you and find the reading material that points out its inspiration. Thanks to Bruce Gerson for inspiring the topic this month.

Next month, we'll expand on some of the concepts covered here and dig deeper into the well that Python has to offer.


Ed Marczak is the Executive Editor of MacTech Magazine. He has written for MacTech since 2004.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Vivaldi 1.0.118.19 - Lightweight browser...
Vivaldi browser. In 1994, two programmers started working on a web browser. Our idea was to make a really fast browser, capable of running on limited hardware, keeping in mind that users are... Read more
Stacks 2.6.11 - New way to create pages...
Stacks is a new way to create pages in RapidWeaver. It's a plugin designed to combine drag-and-drop simplicity with the power of fluid layout. Features: Fluid Layout: Stacks lets you build pages... Read more
xScope 4.1.3 - Onscreen graphic measurem...
xScope is powerful set of tools that are ideal for measuring, inspecting, and testing on-screen graphics and layouts. Its tools float above your desktop windows and can be accessed via a toolbar,... Read more
Cyberduck 4.7 - FTP and SFTP browser. (F...
Cyberduck is a robust FTP/FTP-TLS/SFTP browser for the Mac whose lack of visual clutter and cleverly intuitive features make it easy to use. Support for external editors and system technologies such... Read more
Labels & Addresses 1.7 - Powerful la...
Labels & Addresses is a home and office tool for printing all sorts of labels, envelopes, inventory labels, and price tags. Merge-printing capability makes the program a great tool for holiday... Read more
teleport 1.2.1 - Use one mouse/keyboard...
teleport is a simple utility to let you use one single mouse and keyboard to control several of your Macs. Simply reach the edge of your screen, and your mouse teleports to your other Mac! The... Read more
Apple iMovie 10.0.8 - Edit personal vide...
With an all-new design, Apple iMovie lets you enjoy your videos like never before. Browse your clips more easily, instantly share your favorite moments, and create beautiful HD movies and Hollywood-... Read more
Box Sync 4.0.6233 - Online synchronizati...
Box Sync gives you a hard-drive in the Cloud for online storage. Note: You must first sign up to use Box. What if the files you need are on your laptop -- but you're on the road with your iPhone? No... Read more
Fantastical 2.0.3 - Create calendar even...
Fantastical 2 is the Mac calendar you'll actually enjoy using. Creating an event with Fantastical is quick, easy, and fun: Open Fantastical with a single click or keystroke Type in your event... Read more
The Hit List 1.1.14 - Advanced reminder...
The Hit List manages the daily chaos of your modern life. It's easy to learn - it's as easy as making lists. And it's powerful enough to let you plan, then forget, then act when the time is right.... Read more

SoundHound + LiveLyrics is Making its De...
SoundHound Inc. has announced that SoundHound + LiveLyrics, will be one of the first third-party apps to hit the Apple Watch. With  SoundHound you'll be able to tap on your watch and have the app recognize the music you are listening to, then have... | Read more »
Adobe Joins the Apple Watch Lineup With...
A whole tidal wave of apps are headed for the Apple Watch, and Adobe has joined in with 3 new ways to enhance your creativity and collaborate with others. The watch apps pair with iPad/iPhone apps to give you total control over your Adobe projects... | Read more »
Z Steel Soldiers, Sequel to Kavcom'...
Kavcom has released Z Steel Soldiers, which continues the story of the comedic RTS originally created by the Bitmap Brothers. [Read more] | Read more »
Seene Lets You Create 3D Images With You...
Seene, by Obvious Engineering, is a 3D capture app that's meant to allow you to create visually stunning 3D images with a tap of your finger, and then share them as a 3D photo, video or gif. [Read more] | Read more »
Lost Within - Tips, Tricks, and Strategi...
Have you just downloaded Lost Within and are you in need of a guiding hand? While it’s not the toughest of games out there you might still want some helpful tips to get you started. [Read more] | Read more »
Entertain Your Pet With Your Watch With...
The Petcube Camera is a device that lets you use live video to check in on your pet, talk to them, and play with them using a laser pointer - all while you're away. And the Petcube app is coming to the Apple Watch, so you'll be able to hang out with... | Read more »
Now You Can Manage Your Line2 Calls With...
You'll be able to get your Line2 cloud phone service on the Apple Watch very soon. The watch app can send and receive messages using hands-free voice dictation, or by selecting from a list of provided responses. [Read more] | Read more »
R.B.I. Baseball 15 (Games)
R.B.I. Baseball 15 1.01 Device: iOS Universal Category: Games Price: $4.99, Version: 1.01 (iTunes) Description: The legendary Major League Baseball franchise returns to the diamond. Make History. ** ALL iPOD Touch, the iPad 2 and the... | Read more »
Here's How You Can Tell if an App W...
The Apple Watch is pretty much here, and that means a whole lot of compatible apps and games are going to be updated or released onto the App Store. That's okay though, beacause Apple has quietly updated their app description pages to make things... | Read more »
Forgotten Memories : Alternate Realities...
Forgotten Memories : Alternate Realities 1.0.1 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0.1 (iTunes) Description: + REDUCED PRICE ONLY THE LAUNCHING WEEK + "The most exciting horror game of 2015." - AppSpy... | Read more »

Price Scanner via MacPrices.net

Intel Compute Stick: A New Mini-Computing For...
The Intel Compute Stick, a new pocket-sized computer based on a quad-core Intel Atom processor running Windows 8.1 with Bing, is available now through Intel Authorized Dealers across much of the... Read more
Heal to Launch First One-Touch House Call Doc...
Santa Monica, California based Heal, a pioneer in on-demand personal health care services — will offer the first one-touch, on-demand house call doctor app for the Apple Watch. Heal’s Watch app,... Read more
Mac Notebooks: Avoiding MagSafe Power Adapter...
Apple Support says proper usage, care, and maintenance of Your Mac notebook’s MagSafe power adapter can substantially increase the the adapter’s service life. Of course, MagSafe itself is an Apple... Read more
12″ Retina MacBook In Shootout With Air And P...
BareFeats’ rob-ART morgan has posted another comparison of the 12″ MacBook with other Mac laptops, noting that the general goodness of all Mac laptops can make which one to purchase a tough decision... Read more
FileMaker Go for iPad and iPhone: Over 1.5 Mi...
FileMaker has announced that its FileMaker Go for iPad and iPhone app has surpassed 1.5 million downloads from the iTunes App Store. The milestone confirms the continued popularity of the FileMaker... Read more
Sale! 13-inch 2.7GHz Retina MacBook Pro for $...
 Best Buy has the new 2015 13″ 2.7GHz/128GB Retina MacBook Pro on sale for $1099 – $200 off MSRP. Choose free shipping or free local store pickup (if available). Price for online orders only, in-... Read more
Minimalist MacBook Confirms Death of Steve Jo...
ReadWrite’s Adriana Lee has posted a eulogy for the “Digital Hub” concept Steve Jobs first proposed back in 2001, declaring the new 12-inch MacBook with its single, over-subscribed USB-C port to be... Read more
13-inch 2.7GHz Retina MacBook Pro for $1234 w...
Adorama has the 13″ 2.7GHz/128GB Retina MacBook Pro in stock for $1234.99 ($65 off MSRP) including free shipping plus a free LG external DVD/CD optical drive. Adorama charges sales tax in NY & NJ... Read more
13-inch 2.5GHz MacBook Pro available for $999...
 Adorama has the 13-inch 2.5GHz MacBook Pro on sale for $999 including free shipping plus NY & NJ sales tax only. Their price is $100 off MSRP. Read more
Save up to $600 with Apple refurbished Mac Pr...
The Apple Store is offering Apple Certified Refurbished Mac Pros for up to $600 off the cost of new models. An Apple one-year warranty is included with each Mac Pro, and shipping is free. The... Read more

Jobs Board

*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* Watch SW Application Project Manager...
**Job Summary** The Apple Watch software team is looking for an Application Engineering Project Manager to work on new projects for Apple . The successful candidate Read more
Map Services Frameworks Manager - *Apple* M...
Job Description: Apple 's Maps Evaluation Organization is seeking a Senior Manager to lead the Map services assertions and performance frameworks group. Key Read more
Shuttle Driver - APPLE EAST, APPLE EAST (Unit...
HERE WE GROW AGAIN!! We are adding to our team! Red Lion Chevrolet of Apple Automotive is growing and we are looking for a PART TIME SHUTTLE DRIVER/ PORTER FOR SERVICE Read more
*Apple* Retail - Multiple Positions (US) - A...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.