Volume Number: 22 (2006)
Issue Number: 1
Column Tag: Programming

Build Your Own Supercomputer From Your Macs Laying Around

How to employ Tiger's Xgrid to build clusters and to contribute to grid/cluster projects.

by Mary Norbury-Glaser

Grid vs. Cluster

Even in the Linux world, where cluster and grid computing are well established, there is often confusion about how to distinguish a grid from a cluster. Let's look at characteristics of each.

Grid computing uses the available resources of many individual, loosely coupled computers used by many different people across organizations. Grid computing makes use of idle computer time and unused disk space on different systems, often desktop computers (the SETI@home project, for example). Administrative tasks like system scheduling and job management are "distributed" since the computers are typically located over a wide area of multiple domains. There is usually no single system image; Windows, Linux and Mac systems often contribute to a single grid. Grid computing can be used for both high throughput computing (Apple/Genentech BLAST, for example) and high performance computing (the San Diego Supercomputer Center, the National Center for Supercomputing Applications, the Argonne National Laboratory, and the Max Planck Institute for Gravitational Physics collaborated in the largest grid computing demonstration of simulations involving Einstein's General Relativity equations).

Cluster computing, in contrast, involves a group of tightly coupled computers that work in parallel to share processing as if they were one machine with multiple CPUs. Administrative job management and a scheduling system are centralized in a cluster. Cluster machines also share a single system image. In other words, the collection of systems in a cluster appear as a single entity to the user, the DBA, etc. Cluster computing can be used for high performance computing (see description in grid computing above), load balancing and high availability computing. Load balancing describes the situation when processing activity is efficiently distributed among cluster members so no individual computer gets overwhelmed. High availability computing is implemented in situations that require maximum uptime and where availability of services is critical.

Think of grid computing as involving computers from myriad networks around the globe while cluster computing is a set of machines usually in one location involved in parallel computing with the goal to get them to look like a single virtual machine.

Xgrid Introduced

At MacWorld 2004 in San Francisco, Apple's Advanced Computation Group quietly announced the Technology Preview Release of Xgrid version 1.0 as a free beta download. Xgrid created quite a lot of excitement in ripples throughout the conference attendees who were involved in or interested in compute intensive applications. The software download included the Xgrid app and a basic local alignment search tool: Apple/Genentech BLAST (an open source DNA and protein sequence matching application) that enabled distributed searches on an Xgrid cluster. It touted the use of zero configuration (Rendezvous at the time, now Bonjour) to discover available resources on the network.

XGrid is based on a NeXT application called Zilla developed in the late 80's by Richard Crandall, a then NeXT employee and now a Distinguished Scientist at Apple. Zilla was the first community supercomputing application (screen saver type distributed computation system).

With the publicity of the Virginia Tech's Terascale Computing Facility and their System X built on 1100 dual processor, 2.0 GHz Power Mac G5 computers, the potential of low-cost supercomputing became an exciting possibility. Using commodity based or commercial off-the-shelf (COTS) hardware and free software, distributed computing became well within the reach of any organization with existing hardware.

Xgrid and Tiger

In April 2005, Apple introduced Mac OS X version 10.4 codenamed "Tiger" and included Xgrid in both the client and server versions of the operating system. Xgrid was streamlined and arrived in Tiger a different animal:

Xgrid in Tiger no longer limited job submission to the GUI; the client-side Cocoa API was made available to developers who were encouraged to assimilate Xgrid into their applications instead of writing plugins. There are already several clients that have emerged from this addition to Xgrid: GridStuffer (a Cocoa client available at http://cmgm.stanford.edu/~cparnot/xgrid-stanford/html/goodies/GridStuffer-info.html) and PyXG (a Python interface to Xgrid, http://hammonds.scu.edu/~classes/pyxg.html).

A frustrating change for many users of the Panther technology release of Xgrid is the loss of the Xgrid controller on the Tiger client. Tiger client still includes the Xgrid agent but the GUI controller has been moved to Tiger server (see the next section for definitions of Xgrid terms). However, when we build our own cluster, we'll have some options. Patience, dear reader...

Definitions and Simple Explanations

Client: the Client submits jobs to the Controller. This can be any OS X 10.4 computer or OS X Server.

Controller: the Controller receives the jobs from the Client(s), splits the jobs into tasks and submits the tasks to the Agent(s). The Controller then receives the results back from the Agent(s) and delivers the information back to the Client(s). Apple has moved the Controller to OS X Server but has left the command line equivalent for managing the Controller and job submission on Client.

Agent: the Agent receives tasks from the Controller, runs the computations and sends the results back to the Controller. The Agent can be either a 10.3 or 10.4 computer. One task per CPU can be run.

Node: any single OS X (10.3 or 10.4) desktop computer or OS X 10.4 Server on the network.

There is only one Controller but there can be many Clients and many Agents.

The Controller, Client and Agent can be run on the same machine for purposing of testing. This is not, however, an optimal scenario for real world application.

Participate in a Grid...

Grid computing allows for an environment where idle CPU cycles and storage space of thousands of networked systems can work together on a particular processing-intensive problem. Current projects range from mathematics (prime number searches), science (climate prediction models), life sciences (cancer research), to cryptography (cracking data encryption schemes). By joining a grid project, your home or work computer can contribute to solving a global challenge.

The simplest use of Xgrid is to add your Mac to an existing distributed computing grid project. Deciding to participate in a grid is as simple as looking online for available Xgrid projects: http://distributedcomputing.info/projects.html lists active and upcoming projects under different categories with details on supported operating systems.

For example, Rosetta@home (http://boinc.bakerlab.org/rosetta/) is a project that is attempting to predict and design protein structures in an effort to help cure human diseases. To join this grid, go to their home page, create an account (you'll receive a return Account Key via email which you'll need to confirm at the web site and later insert into the application), make sure your computer meets the system requirements, and download the BOINC client software (Berkeley Open Infrastructure for Network Computing, http://boinc.berkeley.edu/) in either the GUI or command line flavor. The client software enables computers with different operating systems to "talk" to the project server.

Select the Projects tab and click the Attach to new project button. You'll be prompted for the project URL and Account Key that you received when you created your BOINC account.

Based on your chosen preferences (processor, disk, memory and network usage, etc.), the project will start and you can keep track of the progress through the tabs at the top of the BOINC app window.

Instructions for the command line version, along with command line options are available here: http://boinc.berkeley.edu/download.php.

...Or a Cluster

In this tutorial, we'll be looking at Xgrid@Stanford, a project run by Charles Parnot, a postdoctoral fellow in the Molecular and Cellular Physiology Department at Stanford University. His research involves 3D modeling of G protein-coupled receptors to study heart disease and heart rate control. The Stanford project began in March 2004 with 8 computers running at about 4 GHz. By September 2005, they had nearly 500 registered agents with an average of 200 machines online at any given time, the cluster running over 200 GHz. Xgrid@Stanford's home page (http://cmgm.stanford.edu/~cparnot/xgrid-stanford/index.html) describes the project and details FAQs, Goodies and Latest News about the project. Note that the Stanford group calls their project a "cluster" because they are using Xgrid exclusively and all the participating agents are Mac OS X computers. To join Xgrid@Stanford, we'll start by introducing ourselves to Xgrid (note the team has provided separate instructions for Panther and Tiger):

Set up Xgrid on Tiger:

1. Open System Preferences.

2. Open the Sharing Pref Pane.

3. Make sure your computer has a unique name!

4. Highlight Xgrid from the list under the Services tab and click the Configure button. Do not check the Xgrid checkbox!

5. Under Controller, select Use a specific controller: and type in:

b161-g4.stanford.edu

6. Under Agent accepts tasks: choose Always.

7. Under Authentication method: select None in the pop down list. You may get a security warning dialog box that you may ignore. Press OK on the Xgrid configuration sheet.

8. You will now be back at the Sharing Pref pane. Press the Start button.

9. Once the service is started, the Start button becomes a Stop button. If you need to change settings later, you will need to stop the agent, modify the settings and start it again.

10. Open the Energy Saver Pref pane and set Put the computer to sleep when it is inactive for: to Never.

Send an email to charles.parnot@gmail.com with your computer name, where you are from and where you heard about the project (reference MacTech, please!). The project team will let you know about system updates.

Download Dashboard Widget:

Download the Xgrid@Stanford Tachometer Widget from the Dashboard Widget section of Apple's website (http://www.apple.com/downloads/dashboard/status/xgridstanfordwidget.html) or from the developer's site (http://www.mekentosj.com/widgets/xgrid/).

The widget shows the total number of active or inactive agents or processors, the percentage of working agents and the current cluster speed.

Restore Screen Saver Functionality:

In the developer's preview of Xgrid, there was a nice tachometer screen saver feature that disappeared in Tiger. To get it back and view the Xgrid@Stanford widget as a screen saver, download either of these screen saver modules that allow Dashboard widgets to move about your screen during idle time: Amnesty (http://www.mesadynamics.com/amnesty_saver.htm) or Dashsaver (http://highearthorbit.com/software#DashSaver).

In this example, I've chosen to download Amnesty:

1. Drop the Amnesty Screen Saver.saver file into your Screen Savers directory (~/Library/Screen Savers).

2. Open the Desktop and Screen Saver Pref pane.

3. Select Amnesty Screen Saver from the list and click the Options button.

4. Select Xgrid@Stanford from the Widgets pop down list.

5. Set the time for when you want your screen saver to start.

Roll Your Own Cluster

You only need two computers to build a cluster with Xgrid: one agent/controller and one other agent. In this tutorial, we'll use three computers: one 12" Powerbook (named tiger12), one Mac mini (named mimimini), and one 15" Powerbook (our controller named norburym15).

Network the Macs Together:

If your Macs are not already networked together, you can easily do so using a 10/100Base-T Ethernet hub and a cat-5 Ethernet patch cable for each Mac. Connect each Mac to a hub port. Your Macs are also networked if you are sharing an Airport connection but you'll obviously suffer speed degradation in this scenario.

Verify that Bonjour is enabled (in /Applications/ Utilities/Directory Access.app) on all Macs.

Configure the Agents (all Macs):

Open the Sharing Pref pane, highlight Xgrid from the list under the Services tab and click the Configure button. Select these options: Use first available controller, Always, none for authentication.

Click the OK button. You'll get a security warning; ignore it for now.

Click the Start button to turn on Xgrid Sharing.

Configure the Controller:

Remember I mentioned previously that the GUI controller has been moved to Tiger server? We have two options for restoring this functionality in Tiger client: we can download XgridLite (http://edbaskerville.com/software/xgridlite/), a $15 shareware add-on to Tiger Client's System Preferences or we can use the Terminal to control starting and stopping the controller.

XgridLite has some nice, basic features: it can turn the controller on and off and you can set passwords for client and agent authentication.

But since this is MacTech after all, let's do it from the Terminal using the xgridctl daemon. Xgridctl syntax comes in this flavor:

xgridctl status target on|off|start|stop|restart

where the target can be either c, which indicates the controller, or a, to indicate the agent. On/off refers to launching the daemon at startup and start/stop/restart explicitly controls the daemon.

Edit the file /Library/Preferences/com.apple .xgrid.controller.plist by changing the Agent Authentication string and the ClientAuthentication string from Password to None. Don't forget to use sudo!

sudo /usr/sbin/xgridctl c on

modifies the launchd configuration to run the controller daemon at startup and this command will start the controller:

sudo  /usr/sbin/xgridctl c start

To use password authentication, you'll have to do a few more tweaks:

1. Edit the file /Library/Preferences/com.apple.xgrid .controller.plist by changing the Agent Authentication string and the Client Authentication string to use Password.

2. Go to the Sharing Pref pane for Xgrid and select the Password option for Authentication method: and enter the password you wish to use.

3. Reset the agent via the GUI (stop/start). This creates the file: /etc/xgrid/agent/controller-password

4. Copy the file:

sudo cp /etc/xgrid/agent/controller-password /etc/xgrid/controller/agent-password
sudo cp /etc/xgrid/agent/controller-password /etc/xgrid/controller/client-password

5. Start and stop both the agent (via the GUI) and the controller (via xgridctl as above).

The Xgrid Admin.app allows GUI management and monitoring of the controller and it's part of the Server Admin Tools. Server Admin Tools 10.4 can be downloaded and installed on a Tiger client machine (http://www.apple.com/ downloads/macosx/apple/serveradmintools104.html). Put this on your designated controller (mine is norburym15).

Launch Xgrid Admin.app and you will see a sheet asking to enter or choose a controller. You should see the name of the computer that you started the controller on using xgridctl (norburym15, in my case).

In The Xgrid Admin window Click the Connect button and you'll see the Overview:

In the Overview window Click the Connect button and you'll see your agents listed under the Agents tab:

There are no jobs listed under the Jobs tab yet so let's add one!

Add A Job Through the xgrid CLI Client:

Let's move over to my 12" Powerbook (as Client), open the Terminal and "talk" to the Controller (norburym15). First, we'll query to see what grids are available:

tiger12:~norburym$ xgrid -h norburym15.local -grid list

Our syntax includes the xgrid command, -h for hostname of the controller ("norburym15.local", since I'm using an Airport wireless network here at home), and the options -grid list to give us the list of available grids on our local network. The result is:

{gridList = (0); }

If we had two grids available, we'd see this result:

{gridList = (0,1); }

But we only have one grid and from the GUI above, we know it's called the default Xgrid. Let's take a look at information we can get from the CLI:

tiger12:~norburym$ xgrid -h norburym15.local -grid attributes -gid 0

We see this for our result:

{gridAttributes = {gridMegahertz = 0; isDefault = YES; name = Xgrid; }; }

The returned attributes include the current workload (gridMegahertz): 0 because there are no jobs running; whether the grid is the default one for the controller (isDefault): YES; and the name of the grid (name): Xgrid.

Let's run a job!

tiger12:~norburym$ xgrid -h norburym15.local -job run /bin/sh -c cal 2005

This command initiates a job that calls the default month's calendar. When we hit enter and take then run over to take a quick look at the Xgrid Admin GUI app on our controller (norburym15), we can see some information while the job is running. Here's the Overview tab with a job running:

Click the Agents tab:

And now click on the Jobs tab:

Finally, here is a screenshot of all the commands we've used to query the grid and initiate the job, along with the results from the job request:

Take a look at the xgrid man pages for more options and some very good examples (rare for man pages but then Apple wrote this one!).

Now...

Apple's Server Solutions site has a large section devoted to their proprietary tools for cluster computing (http://www.apple.com/xserve/cluster/). They provide ample information on their cluster technology solutions: G5, Xgrid, Xcode, Shark (performance optimization tool) and the Accelerate framework, as well as their cluster products: Apple Workgroup Cluster, Xserve G5 Cluster Node, Xserve Raid, OS X Server and Xsan. Their growing suite of integrated products provide robust and inexpensive solutions for customers who require solutions to their compute intensive application needs.

...And the Future

Steve Jobs' announcement in his WWDC 2005 keynote that Apple will deliver Macs with Intel microprocessors by Summer of 2006 ignited a considerable amount of discussion and speculation. The move from the PowerPC chip is controversial, no less for the apparent alliance with a company that has traditionally been aligned with "The Dark Side". This has, of course, sparked quite a bit of online discourse about running Windows on Intel-based Macs.

Apple is expected to use the Pentium M chip inside their next generation of desktop and portables because of its low power consumption and high performance. The Pentium M is geared to deliver more performance per clock cycle in order to consume less energy, which will ultimately make it run cooler than Pentium 4 chips. Of note, virtualization technology is built into Intel chips which will allow the machines to be partitioned to run different types of software like Windows or Linux at once, on top of Mac OS X. And hardware virtualization enables a system to run at near full-speed.

Apple has issued several statements saying they will not prevent Windows from running on Intel-based Macs but they will not allow the Mac OS to run on non-Apple machines. In fact, the Macs shipping as part of the Developers Transition Kit sport a security chip called the Trusted Platform Module that contains an encrypted serial number that verifies the OS is running on Apple hardware.

An interesting aspect of this is the potential development of cluster and grid virtualization. Traditional virtualization systems like VMWare emulate a PC down to the hardware devices. In other words, VMWare simultaneously creates multiple x86 virtual computers (guest virtual machines) and each virtual machine (VM) has it's own virtual CPU, disk, memory, etc. and all the virtual hardware is mapped to your computer's real hardware (the host machine). One limitation is significant emulation overhead and little to no opportunity for optimization. The VM (virtual machine) runs multiple kernels and full installations. Running virtualization on a cluster creates an environment with automatic load balancing via process migration: the master node provides system services while the compute nodes run the application. This scales up performance. Remember the difference between clusters and grids: with a cluster, there is a single point of administration while in a grid system, there are often different operating systems in different domains. While the cluster model provides a simpler, more well defined scenario, virtualization on grids will see advances in the future and Intel-based Macs could very well fill the growing need for low cost, commodity based distributed computing solutions.

Mary Norbury-Glaser is IT Director at the Barbara Davis Center for Childhood Diabetes, an affiliate center at the University of Colorado Health Sciences Center in Denver, Colorado. She has too-many-years-to-count experience in cross-platform systems implementation and administration in the education sector. You can reach her at norburym@mac.com.