TweetFollow Us on Twitter

Mac in the Shell: Data Manipulation with PHP

Volume Number: 24 (2008)
Issue Number: 04
Column Tag: Mac in the Shell

Mac in the Shell: Data Manipulation with PHP

PHP provides easy access to MySQL on OS X

by Edward Marczak

Introduction

Last month, we covered some of the basics of scripting with PHP. This month, we'll continue and talk about database access and string manipulation with PHP. Thanks to a pretty full install of PHP as the default in OS X, we don't have to worry about fetching, configuring, compiling and installing PHP - we can just get on with writing our script.

Working Environment

Last month, I mentioned that there are plenty of OS X-based editors that 'understand' PHP. This comes in the form of syntax coloring, appropriate code snippets, code completion and auto indentation. Before we continue, there are two more things that I should mention.

Coda, from Panic Software, is one of those editors - one that I should have mentioned last month. It's actually quite a bit more than a simple text editor: It's a full web development environment. I really mention it here because it has a way to access PHP documentation built right in (along with HTML, Javascript and more). Since PHP is such a popular web development language, as a web development tool, this made sense. At last check, though, access to this documentation was only available while you have an Internet connection.

I, on the other hand, am very much of an offline person. I want/need to accomplish things while on a plane, train, or wherever I may not have a good signal from some source, for whatever reason. Due to that, I've come to rely on the PHP Function Reference widget for Dashboard. Downloadable from here: http://code.google.com/p/phpfr/ (it has been "open sourced" since its introduction, and now lives as a Google Code project). Weighing in at close to 40MB, it's a little larger than your average widget, but that's the price you pay to carry around the entire PHP function set with descriptions. As a Dashboard widget, it lives out of sight until you need it. With a search function built-in, I tend to reach for this a lot. Highly recommended.

The Setup

I do a lot of data manipulation and migration work, and scripting languages are an essential part of the process. Perl, Ruby and Python are all good contenders for that position, but this particular column is about PHP! Our sample project this month will extract data from a MySQL database and give us a CSV file, ready for importing into another system. "But Ed," you say, "you can do that with MySQL alone!" While this is true, we aren't allowed to manipulate the data too drastically in that process.

Let's imagine that the data in our source system is stored with a single column for full name, but our target system wants separate first name and last name. Or that we need to go gather more data (like a zip+4 based on address) per record. There are hundreds of reasons you may need to filter the data as it goes from the database to a CSV file.

Connecting to the Database

One variable type that I left out of our discussion last month is resource. We looked at string, integer and other built-in types, but 'resource' really warrants its own discussion.

A variable of type "resource" holds a reference to some external resource. This may be a file on disk, a curl session to fetch a URL, a database connection or more. Again, it's just a reference, and you'll need to use functions that know how to access those resources via the resource variable that you pass to them.

First thing that our script needs to do is open a connection to the MySQL server. This happens with the mysql_connect command:

$connection = mysql_connect("server.example.com","user","password");

The mysql_connect command passes back a resource if successful, and FALSE if there's an error. Let's get into good habits right away: check the value of $connection for FALSE. Thankfully, we have access to the error that the server passes back to us in the form of two functions: mysql_errno() and mysql_error(). These functions return the error code from the previous MySQL operation. We should follow our connect, or any database operation with something similar to this:

if (! $connection) {
   print "Connection to database failed with ".mysql_errno()." - ".mysql_error()."\n";
   die();
}

Simply, if $connection is not TRUE (zero, or, FALSE), we print out a message containing the MySQL error information, and then we halt operation of the script and exit with die().

Also, looking at our initial connect string, what's wrong, or at least, what should send shivers up your spine? Yes, a password in a script. Unfortunately, that's just life in the big city, and you'll need to take precautions that allow only the right people to have access to the script. While you can put this information in external files and pull it in, or make your script get it from somewhere, the person that can read your file can also (typically) read how and where to get the password. Here be dragons, so, be conscious of this and plan for protection.

Once connected, we need to specify which database we're going to be working with. That's done with the mysql_select_db() function:

$select = mysql_select_db("customers");
...error checking here ...

All subsequent MySQL functions operate on the selected database. If you're working with more than one server, you can optionally pass in the connection resource that you obtained from mysql_connect():

$conn1=mysql_connect("server1","bob","s3kret");
$conn2=mysql_connect("server2","jane","h1dd3n");
$select1=mysql_select_db("sales",$conn1);
$select2=mysql_select_db("aggregates",$conn2);

...with the proper error checking, of course!

Once those preliminaries are complete, we can now query our database. While you can stuff a query into fewer lines, we're going to split it up in a more conventional manner. The mysql_query() function takes a string as an argument. Assigning the query to a variable makes for easier reading, easier variable substitution, and allows one to build dynamic queries. Let's start with a simple example:

$query="select uid,fname,lname from users";
$result=mysql_query($query);

$query is a simple string, and $result is of type resource. As with our earlier queries, if the execution fails, mysql_query() will return a Boolean FALSE, and our errors can be retrieved with mysql_errno() and mysql_error().

$result is simply a resource, and points to the result set. We now have to use it to fetch the actual data, database row by database row. If you purposefully limited your query to a single row, you could now go fetch it with one line. However, more often than not, you're expecting multiple rows - sometimes thousands. A perfect job for the while loop that we introduced last month:

while ($row=mysql_fetch_assoc($result)) { print "Scanning ".$row['uid'].": ".$row['fname']." ".$row['lname']."\n"; }

That's a lot packed into a small space, so, let's break it down. The data fetching happens with this line:

$row=mysql_fetch_assoc($result)

You could manually call that over and over and get a new row from the database each call. Of course, for large result sets, a loop is the only feasible method. The mysql_fetch_assoc() function returns an associative array. Unlike a "plain" array, you can access values with a key that is a string. In our case, each column from the database becomes an entry in the array. We asked for the fields, "uid", "fname" and "lname", which makes $row become:

$row['uid']
$row['fname']
$row['lname']

..filled with the values from the current row retrieved from the database.

Within the body of the loop, there's a single line: a print statement. It's fairly ugly, though, isn't it? It happens to be pretty standard fare in PHP. The concatenation operator (".") 'glues' two strings together. So, now it should be pretty easy to break down, as we're printing some static text (the bits in quotes) and some dynamic text, filled in by the values in the $row array. Sample output would look like this:

Scanning 1052: Mike Bollinger
Scanning 1053: Laura Wilkinson
Scanning 1055: Joanne Moss
...

Now that we have access to our data, let's get it into a format usable by almost all other apps: CSV.

Data Manipulation

If you simply wanted a CSV file of a MySQL database table, there are many simple ways to do so. However, when moving data from one system to another, we often need to massage the original data into a new format. We get that opportunity in our while loop, where we're accessing each row of the table and before writing it out. PHP has many string manipulation functions to help us do anything our hearts (and brains) desire.

The first thing we may want to do is to simply re-assign a cleaned-up version of a string before writing it out. Consider this:

switch ($row['state']) {
   case "NY":
   case "new york":
   case "noo yrk":
      $state="New York";
      break;
   case "CA":
   case "calif.":
   case "cal":
      $state="California";
      break;
   default:
      $state="Unknown";
}

Rather than writing six separate if statements, I chose to use a single switch statement. This is a) cleaner code and b) a nice introduction to the switch statement, which I did not cover last month. Generally, the switch statement will test a single expression for the results you choose to test for. In the preceding example, we're examining the variable "$row['state']". Each case statement within the switch block acts as an "if" statement, and all code through the following break statement is executed. The code in the default block matches when no other case statements match. While the switch statement may contain virtually any expression, the match in each case statement can only be a simple type: string, integer or floating-point. You cannot evaluate arrays or objects here. For these more complex cases, you still must rely on an if statement.

The simple line doing all of the work here is this: $state="...". We're just assigning a literal to our variable. We're even free to re-assign $row['state'] if it suits our needs and style.

One of the coolest ways to tear up a string is the explode() function. As always, an example is best:

$animal="bear cat dog elephant";
$the_animals=explode(" ",$animal);
After this runs, $the_animals will be an array containing:
$the_animals[0]="bear"
$the_animals[1]="cat"
$the_animals[2]="dog"
$the_animals[3]="elephant"

explode() can split based on any delimiter, not just a space. However, this is extremely useful for splitting up a full name. If you know that all entries in the database are in firstname-space-lastname format, then this will do what you want:

list($fname, $lname) = explode(" ",$fullname);

However, the reason you're probably involved is due to a more complex nature of the data. If most names are in two-name format, but some may contain a middle initial, too, then we have a few options. One would be to toss the middle initial, or include it in the first name. In that scenario we need to test how many elements were returned:

$fullname="William S. Fatire";
$names=explode(" ",$fullname);
if (count($names) > 2) {
   $fname=$names[0]." ".$names[1];
   $lname=$names[2];
} else {
   $fname=$names[0];
   $lname=$names[1];
}
print "The names are:\n";
print "Firstname: $fname\n";
print "Lastname: $lname\n";

The count() function is introduced here and simply returns the number of elements in an array.

Naturally, our target database may have a field for middle initial, in which case, we can retain it and assign it appropriately.

Let's now imagine that we want to create a default user id for our new users. We can base this id on the user's real name. PHP includes some nice string slicing functions. If we want to base the user ID on the user's first initial plus last name, that's simple:

$uid=$fname[0].$lname;

PHP can access string elements by character when you supply the zero-based offset in square brackets. So, in this example, we're just grabbing the first character of $fname.

If you had a more complex uid-creation scheme, it'd be easy to handle, too. If you needed a uid that contained the first three letters from the family name, and the first two from the first name, PHP includes a nice sub-string function. Generically, it looks like this:

substr ($string, $start [,$length ])

This will return a string that starts at '$start' in $string, and runs until the end of the string supplied, or optionally, for $length. Back to our first-three, first-two uid scheme:

$uid=substr($lname,0,3).substr($fname,0,2);

Nice, right? Of course, if you were really implementing this, you'd need to look for duplicates and also check for last names that may be shorter than 3 characters in total (like "Li").

All Together Now

I'd like to put all of this together to create a code snippet that takes data from a database, gives space to manipulate the data, and output a CSV file. Some new elements will also be introduced here.

Listing 1: db2csv.php

01 <?php
02 
03 mysql_connect("127.0.0.1", "dbuser", "dbpass") or
04     die("Could not connect: " . mysql_error());
05 mysql_select_db("mydb");
06 
07 $q="select * from user_profiles order by fid";
08 $r=mysql_query($q);
09 while ($row=mysql_fetch_assoc($r)) {
10    $pf[$row['fid']]=$row['title'];
11 }
12 
13 // Create the header
14 $curcsv="\"uid\",";
15 foreach ($pf as $key=>$val) {
16    $curcsv=$curcsv."\"".$val."\",";
17 }
18 $curcsv=substr($curcsv,0,strlen($curcsv)-1);
19 print "$curcsv\n";
20 
21 $q="select * from users where status=1";
22 $r=mysql_query($q);
23 while ($row=mysql_fetch_assoc($r)) {
24 // Build csv for current user
25    $curcsv="\"".$row['uid']."\",";
26    foreach ($pf as $i=>$val) {
27       $q2="select value from user_profiles where fid=$i and uid=".$row['uid'];
28       $r2=mysql_query($q2);
29       $row2=mysql_fetch_assoc($r2);
30       $curcsv=$curcsv."\"".$row2['value']."\",";
31    }
32    $curcsv=substr($curcsv,0,strlen($curcsv)-1);
33    print "$curcsv\n";
34 }
35 
36 ?>

Newly introduced here is the foreach loop. foreach gives the programmer (you) an easy way to iterate over arrays. Generically, foreach looks like this:

foreach ($array as $value)

The loop interates once for each element of the array, and $value is updated accordingly. A variation to this (seen on line 26 in listing 1) also includes the current key - very useful for associative arrays where the key is text or other representation of the index. That variant is simply:

foreach ($array as $key=>$value)

Breaking down listing 1, lines 1-5 should look familiar: the opening PHP tag, and then try to connect to the database. Line 7 defines the first query to the first table, and line 8 executes that query against the selected database. The while loop starting at line 9 is pretty standard fare, but what's going on in the loop body?

On line 10, we're creating an associative array ($pf) using a variable as the key ("index"). In this case, we're using the result of the database fetch $row['fid']. Pretty slick.

Lines 14 through 17 take case of one small but significant part of writing out a CSV file: making the first line a header that describes the remaining columns. In our case, we can do that with the contents of the array we just created. Another foreach loop neatly solves that. In the body of this loop, we're creating a variable that will hold the current line of the CSV file. Each field is wrapped in quotes, and then followed by a comma.

Line 18 removes the trailing comma from $curcsv, and line 19 prints out $curcsv. Now for the bulk of work.

Lines 21 through 33 handle the main load of this program. Line 21 and 22 set up a new query and execute it. Line 23 brings back our now familiar while loop, fetching one database row at a time. Line 25 starts $curcsv anew each iteration, wrapping the proper value ($row['uid'] in this case) in quotes followed by a comma.

Line 26 gets interesting: we use a new foreach loop to obtain more information about this particular uid for each of the values in $pf - by running a new query. We query and fetch again, and line 30 adds each field retrieved to $curcsv (wrapped and comma'd).

Finally, line 32 takes care of the trailing comma on $curcsv (because we blindly add it after each value), and line 33 prints the row.

Running this, or similar program, will simply dump all output to standard out - we're only using a print statement. You thought we were creating a CSV file? Well, we are! Everything in Unix is a file. Remember our shell redirectors? We can run this program as is and have the chance to visually inspect the output, and, when we're ready, simply redirect to a file:

php db2csv.php > users.csv

This is also a nice euphemism for, "I'm going to cover file manipulation next month!"

Conclusion

I'll continue to say: don't discount PHP as a general scripting language. It's fairly rapid development, has broad capabilities, and will typically be found across platforms (including versions for Windows - but you'll need to install it yourself). As mentioned, I'll cover some more aspects of scripting with PHP next month, including file manipulation.

Media of the month: Inside the Machine: An Illustrated Introduction to Microprocessors and Computer Architecture, by Jon Stokes. It's a very readable introduction to microprocessor architecture, and it's even current enough that it covers up through Intel's Core 2 Duo chips.

Until next time, enjoy your new scripting prowess!

References

Official PHP Documentation: http://www.php.net/docs.php


Ed Marczak is the Executive Editor for MacTech Magazine, and has been lucky enough to have ridden the computing and technology wave from early on. From teletype computing to MVS to Netware to modern OS X, his interest has always been piqued. He has also been fortunate enough to come into contact with some of the best minds in the business. Ed spends his non-compute time with his wife and two daughters.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Evernote 6.0.15 - Create searchable note...
Evernote allows you to easily capture information in any environment using whatever device or platform you find most convenient, and makes this information accessible and searchable at anytime, from... Read more
MacUpdate Desktop 6.0.7 - Search and ins...
MacUpdate Desktop 6 brings seamless 1-click installs and version updates to your Mac. With a free MacUpdate account and MacUpdate Desktop 6, Mac users can now install almost any Mac app on macupdate.... Read more
Freeway Pro 7.0.4 - Drag-and-drop Web de...
Freeway Pro lets you design and build sophisticated responsive websites code free. With its user-oriented drag-and-drop interface, Freeway Pro helps you piece together the website of your dreams. Add... Read more
VueScan 9.5.18 - Scanner software with a...
VueScan is a scanning program that works with most high-quality flatbed and film scanners to produce scans that have excellent color fidelity and color balance. VueScan is easy to use, and has... Read more
Firetask 3.7 - Innovative task managemen...
Firetask uniquely combines the advantages of classical priority-and-due-date-based task management with GTD. Stay focused and on top of your commitments - Firetask's "Today" view shows all relevant... Read more
VOX 2.5.3 - Music player that supports m...
VOX is a beautiful music player that supports many filetypes. The beauty is in its simplicity, yet behind the minimal exterior lies a powerful music player with a ton of features and support for all... Read more
Apple iMovie 10.0.9 - Edit personal vide...
With an all-new design, Apple iMovie lets you enjoy your videos like never before. Browse your clips more easily, instantly share your favorite moments, and create beautiful HD movies and Hollywood-... Read more
MarsEdit 3.7.1 - Quick and convenient bl...
MarsEdit is a blog editor for OS X that makes editing your blog like writing email, with spell-checking, drafts, multiple windows, and even AppleScript support. It works with with most blog services... Read more
ClamXav 2.8.1 - Free virus checker, base...
ClamXav is a free virus checker for OS X. It uses the tried, tested, and very popular ClamAV open source antivirus engine as a back end. I have been working on ClamXav for more than 10 years now, and... Read more
CrossOver 14.1.4 - Run Windows apps on y...
CrossOver can get your Windows productivity applications and PC games up and running on your Mac quickly and easily. CrossOver runs the Windows software that you need on Mac at home, in the office,... Read more

World of Tanks Blitz celebrates its firs...
Today marks the first anniversary of the launch of World of Tanks Blitz, the mobile version of the PC tank battler, World of Tanks. World of Tanks Blitz launched on iOS and Android on June 26th last year and to celebrate, Wargaming is giving all... | Read more »
Heroes and Castles 2 Has its Own Standal...
Heroes and Castles 2 is a third-person castle defense game from the same team behind Block Fortress and Bug Heroes. It's cool, it's fun, and now it has its very own free version. [Read more] | Read more »
Formula Cartoon All-Stars Lets You Race...
Ever want to pit your favorite characters from shows like Steven Universe, Adventure Time, and Regular Show against each other in a not quite death race? Well once upon a time you could, but Formula All Stars Touch N' Go doesn't exist anymore. Hope... | Read more »
Retype - Typography Photo Editor (Photo...
Retype - Typography Photo Editor 1.0 Device: iOS Universal Category: Photography Price: $2.99, Version: 1.0 (iTunes) Description: Retype is built out of passion for great typography and it's all about adding text to photo with style... | Read more »
Hungry Shark Evolution Celebrates Shark...
Shark Week is almost here, as is Independence Day, so naturally Hungry Shark Evolution is going to get in on the action. Yes, even the fireworks. [Read more] | Read more »
The New Trivia Crack Will Feature a Musi...
It's official: iHeartMedia (you may know them from iHeartRadio) will be in charge of providing music-related questions for Trivia Crack's upcoming sequel. Also Trivia Crack is getting a sequel. [Read more] | Read more »
Toca Life: City (Education)
Toca Life: City 1.0 Device: iOS Universal Category: Education Price: $2.99, Version: 1.0 (iTunes) Description: Welcome to Toca Life: City, a metropolis filled with everyday fun! Customize characters, explore exciting locations and... | Read more »
Ironkill's Big New Update is Big an...
The other popular robot fighting game on the App Store, Ironkill, has received a pretty substantial update today. We're talking new bots, new rewards, graphical tweaks; the works. [Read more] | Read more »
iOS Users, Say Hello to Dragon Quest VI:...
The App Store is no stranger to the Dragon Quest series, and has had its fair share of ports for quite some time. That tradition is staying pretty much exactly the same with the release of Dragon Quest VI: Realms of Revelation. [Read more] | Read more »
Now is the best time to try League of An...
League of Angels - Fire Raiders is the mobile sequel to the hit Facebook MMORPG League of Angels and is out now on iOS and Android. There are over a hundred beautifully hand-drawn heroes and angels to collect that each possess their own unique... | Read more »

Price Scanner via MacPrices.net

MacBook Airs on sale for up to $75 off MSRP
Save up to $75 on the purchase of a new 2015 13″ or 11″ 1.6GHz MacBook Air at the following resellers. Shipping is free with each model: 11" 128GB MSRP $899 11" 256GB... Read more
Apple’s Education discount saves up to $300 o...
Purchase a new Mac or iPad at The Apple Store for Education and take up to $300 off MSRP. All teachers, students, and staff of any educational institution qualify for the discount. Shipping is free,... Read more
Save up to $600 with Apple refurbished Mac Pr...
The Apple Store has Apple Certified Refurbished Mac Pros available for up to $600 off the cost of new models. An Apple one-year warranty is included with each Mac Pro, and shipping is free. The... Read more
Mac Pros on sale for up to $260 off MSRP
B&H Photo has Mac Pros on sale for up to $260 off MSRP. Shipping is free, and B&H charges sales tax in NY only: - 3.7GHz 4-core Mac Pro: $2799, $200 off MSRP - 3.5GHz 6-core Mac Pro: $3719.99... Read more
Save up to $400 on 2014 15-inch Retina MacBoo...
B&H Photo has previous-generation 2014 15″ Retina MacBook Pros on sale for up to $400 off original MSRP. Shipping is free, and B&H charges NY sales tax only: - 15″ 2.2GHz Retina MacBook Pro... Read more
15-inch Retina MacBook Pros on sale for up to...
B&H Photo has new 2015 15″ Retina MacBook Pros on sale for up to $125 off MSRP including free shipping plus NY sales tax only: - 15″ 2.2GHz Retina MacBook Pro: $1899.99 $100 off - 15″ 2.5GHz... Read more
College Student Deals: Additional $100 off Ma...
Take an additional $100 off all MacBooks and iMacs at Best Buy Online with their College Students Deals Savings, valid through July 11, 2015. Anyone with a valid .EDU email address can take advantage... Read more
Apple refurbished Time Capsules available for...
The Apple Store has certified refurbished Time Capsules available for $100 off MSRP. Apple’s one-year warranty is included with each Time Capsule, and shipping is free: - 2TB Time Capsule: $199, $100... Read more
Newsweek Launches iPhone App
The venerable weekly news magazine Newsweek, owned by IBT Media, has announced the launch of its first iPhone app. The new app is available through Apple’s App Store and will allow consumers to read... Read more
New Initiative Covering 80 Million Homes Will...
Today, Internet service providers, equipment suppliers and retail equipment manufacturers joined the Consumer Electronics Association (CEA) and the National Cable & Telecommunications Association... Read more

Jobs Board

Architect / Senior Software Engineer, *Apple...
**Job Summary** Apple Pay is already changing our pay-habit in a deeper and fundamental level. We are looking for a software engineer with a passion for large scale inte Read more
Project Manager, *Apple* Retail New Store O...
**Job Summary** An Apple Retail New Store Openings & Remodels Project Manager is responsible for successfully managing the openings, remodels, and small works of Read more
*Apple* Solutions Consultant - Retail Sales...
**Job Summary** As an Apple Solutions Consultant (ASC) you are the link between our customers and our products. Your role is to drive the Apple business in a retail Read more
Program Manger, WW *Apple* Direct Fulfillme...
**Job Summary** We are seeking a business analyst to work within our Worldwide Apple Direct Fulfillment Operations team. This role will work closely with related program Read more
Hardware Design Validation Engineer - *Apple...
**Job Summary** The Apple Watch team is looking for a Hardware Design Validation Engineer. This person will be part of the Apple Watch hardware team with Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.